Will it fit? Memory for serving and fine-tuning
Weights, KV cache, runtime and adapter states added up for a model card and a GPU, with a calculator.
What it covers9
Why it matters
Before you download a model or rent a GPU, one question decides everything: will it fit? Four lessons gave you the pieces. Weights cost parameters × bytes, from running open models. The KV cache grows with every token. Quantization shrinks the bytes. LoRA shrinks what training has to hold. This lesson adds them up.
You will compute what an 8-billion-parameter model needs to serve at 4 bits with an 8,192-token context (4.0 GB of weights, 1.07 GB of cache), what full fine-tuning of it needs (128 GB), and what a QLoRA run needs instead (5.7 GB before activations), with a calculator you can point at any model card. The numbers are those four lessons' formulas and nothing more, and the lesson says where they are rough.
What you'll be able to do
- Compute a model's weight memory from parameters and bits, and its KV cache from layers, KV heads, head size and tokens
- Add up what serving needs on a given GPU, and read the answer off a model card
- Compute what full fine-tuning needs (16 bytes per parameter) against a LoRA or QLoRA run
- Say which parts of the estimate are exact and which are rough
Show the answer
4 GB: 8 billion × 4 bits ÷ 8 bits per byte
Parameters × bits ÷ 8 gives bytes: 8 × 10⁹ × 4 ÷ 8 = 4 × 10⁹ bytes, 4 GB. At 16 bits the same model is 16 GB.
The idea: weights, cache and training state, added up
This lesson rests on Running open models locally, Inference and the KV cache and LoRA. Each gives one bill; this lesson adds them.
Weights: parameters × bytes
Every weight is stored once, so weight memory is the parameter count times the bytes per weight: 2 for bfloat16 (the 16-bit format most open models ship in, from Training at scale), 1 for 8-bit, 0.5 for 4-bit. An 8-billion-parameter model is 16 GB, 8 GB or 4 GB; a 70-billion one is 140 GB or 35 GB at 4 bits. Add about 1 GB for the runtime itself: the GPU's working context, buffers, the tokenizer and embeddings machinery. Those are the first two lines of any fitting estimate, and they are exact.
The same bytes set the speed. The inference lesson showed that generating one token reads every weight once, so a model that is half the bytes decodes about twice as fast on the same memory bandwidth; quantization is a speed tool as much as a memory one. The next cell prints the weight memory for the two sizes at three precisions.
Weights at three precisions
Parameters × bits ÷ 8, in GB, for an 8B and a 70B model.
def weights_gb(params_b, bits): return params_b * 1e9 * bits / 8 / 1e9 for params_b, bits in [(8, 16), (8, 8), (8, 4), (70, 4), (70, 16)]: print(f"{params_b}B at {bits}-bit: weights {weights_gb(params_b, bits):.1f} GB")
The 8B model drops from 16.0 GB to 4.0 GB between bfloat16 and 4-bit, which is the difference between needing a data-center GPU and fitting beside a browser on a laptop. The 70B model at 4 bits, 35.0 GB, still needs two consumer GPUs or one large one; at 16 bits, 140.0 GB, it needs several. The quantization lesson's quality table applies: 8-bit is close to lossless, 4-bit costs a little, and below that the model breaks.
print(8e9 * 4 / 8 / 1e9)
Show the answer
4.0
8 × 10⁹ parameters × 4 bits ÷ 8 bits per byte = 4 × 10⁹ bytes; divided by 10⁹, 4.0 GB.
Write a sentence or two, then compare it with a good answer in the lesson.
The cache grows with every token
Serving keeps the keys and values of every earlier token so each new token computes only its own: the KV cache. Its size is a product of five numbers:
In plain words: multiply the five sizes together, and double it for keys and values.
The result is for one conversation. Multiply it by the batch size when several sequences are served at once. The 2 is keys and values. For a 32-layer model with 8 key-value heads of size 128, in bfloat16, an 8,192-token context costs 1.07 GB per sequence. If the model had a key-value head for each of its 32 query heads instead, 4.29 GB: grouped-query attention, from the inference lesson, is what makes long contexts affordable. At a 131,072-token context the same model needs 17.18 GB of cache, more than its 4-bit weights four times over.
For scale, the course's tiny GPT (2 layers, 4 heads of size 8, 64 tokens, 32-bit numbers) needs 32,768 bytes of cache, 32.8 KB. The formula is the same; the numbers are what change. The next cell prints these, and the calculator after it lets you move every input.
Change any factor and the product moves with it. Eight KV heads instead of 32 divides the whole cache by 4.
Write a sentence or two, then compare it with a good answer in the lesson.
In code: the cache sizes, the calculator and the training sums
The cells below turn the sums into functions, then add up what training holds.
KV cache sizes
2 × layers × KV heads × head size × tokens × bytes, for a production-sized model at three settings and for the tiny GPT.
def kv_cache_gb(layers, kv_heads, head_dim, tokens, bytes_per_number=2, batch=1): return 2 * layers * kv_heads * head_dim * tokens * bytes_per_number * batch / 1e9 print("32 layers, 8 KV heads, head size 128, 8,192 tokens, bf16: " f"{kv_cache_gb(32, 8, 128, 8192):.2f} GB") print("the same with 32 KV heads (no grouping): " f"{kv_cache_gb(32, 32, 128, 8192):.2f} GB") print("8 KV heads at 131,072 tokens: " f"{kv_cache_gb(32, 8, 128, 131072):.2f} GB") tiny = 2 * 2 * 4 * 8 * 64 * 4 print(f"the tiny GPT (2 layers, 4 heads of 8, 64 tokens, fp32): {tiny} bytes = " f"{tiny / 1e3:.1f} KB")
1.07 GB for a comfortable 8,192-token context; four times that, 4.29 GB, without grouped-query attention; 17.18 GB for the full 131,072-token window the model card might advertise. The cache scales linearly with tokens and with batch size, so serving eight such sequences at the long context would need 137 GB of cache alone, which is why long-context serving is priced the way it is. The tiny GPT's 32.8 KB is the same formula with small numbers.
Show the answer
Shrinks to a quarter: the cache is proportional to the number of KV heads
KV heads is one factor in the product; 8 instead of 32 divides the cache by 4. The query heads do not enter the cache at all.
The KV cache in GB
Write kv_cache_gb(layers, kv_heads, head_dim, tokens, bytes_per_number=2, batch=1): 2 × layers × KV heads × head size × tokens × bytes per number × batch, divided by 10⁹.
def kv_cache_gb(layers, kv_heads, head_dim, tokens, bytes_per_number=2, batch=1): return 0.0 print(round(kv_cache_gb(32, 8, 128, 8192), 2))
Write a sentence or two, then compare it with a good answer in the lesson.
Plot the serving total against the context for 8 and for 32 KV heads. The dashed line is a 24 GB GPU. The weights are fixed at 4.0 GB, so the curves rise only because the cache grows.
import matplotlib.pyplot as plt def total_gb(kv_heads, tokens): weights = 4.0 cache = 2 * 32 * kv_heads * 128 * tokens * 2 / 1e9 return weights + cache + 1.0 tokens = [4096 * k for k in range(1, 33)] for heads in (8, 32): plt.plot(tokens, [total_gb(heads, t) for t in tokens], label=f"{heads} KV heads") plt.axhline(24, linestyle="--", label="24 GB GPU") plt.xlabel("context tokens") plt.ylabel("GB") plt.legend() print(round(total_gb(8, 131072), 2), round(total_gb(32, 131072), 2))
Fine-tuning: 16 bytes per parameter, or an adapter
Training holds more than the weights. For each parameter, full fine-tuning with mixed precision and Adam keeps the bfloat16 weight (2 bytes), its gradient (2), Adam's two running averages in 32-bit (8) and a 32-bit master copy of the weight (4): 16 bytes per parameter, the GPU lesson's rule of thumb, before activations. The 8B model needs 128 GB; the 70B model 1,120 GB. Neither fits one GPU.
LoRA changes what is trained. Each adapted matrix () is frozen and gets with trainable numbers. In the same 8B model, query and output are 4,096 × 4,096; key and value are 4,096 × 1,024 (8 key-value heads of 128); the three feed-forward matrices are 4,096 by 14,336. Adapting all seven at rank 16 costs 1,310,720 parameters per layer, 41,943,040 over 32 layers: 0.52 percent of the 8 billion, and 0.67 GB at 16 bytes each.
QLoRA puts the frozen base in 4-bit: 4.0 GB of weights, 0.67 GB of adapter states and 1 GB of runtime, 5.7 GB in all, which fits a consumer GPU. Training keeps no KV cache: each step reads whole sequences in one pass. It leaves out the activations saved for the backward pass, usually the next-largest item. They grow with batch and sequence length; gradient checkpointing trades compute for them.
Full fine-tuning against QLoRA
16 bytes per parameter for full fine-tuning; r(d_in + d_out) per adapted matrix for LoRA; the QLoRA total.
def gb(nbytes): return nbytes / 1e9 print(f"full fine-tuning at 16 bytes/parameter: 8B {gb(8e9 * 16):.0f} GB, " f"70B {gb(70e9 * 16):.0f} GB") d, ffn, kv, r, layers = 4096, 14336, 8 * 128, 16, 32 # (d_in, d_out) of q, k, v, o, gate, up, down: 8 KV heads of 128 make k and v narrow shapes = [(d, d), (d, kv), (d, kv), (d, d), (d, ffn), (d, ffn), (ffn, d)] per_layer = sum(r * (d_in + d_out) for d_in, d_out in shapes) total = per_layer * layers print(f"LoRA r={r}: {per_layer:,} parameters per layer, {total:,} in all " f"({100 * total / 8e9:.2f}% of 8B)") print(f"adapter weights + gradients + Adam at 16 bytes: {gb(total * 16):.2f} GB") qlora = gb(8e9 * 0.5) + gb(total * 16) + 1 print(f"QLoRA on 8B: 4-bit base {gb(8e9 * 0.5):.1f} GB + adapters " f"{gb(total * 16):.2f} GB + 1 GB runtime = {qlora:.1f} GB, before activations")
128 GB against 5.7 GB for the same model, which is the whole case for adapters. The adapter count is small because r = 16 is small next to 4,096 and 14,336: each matrix trades d_in × d_out numbers for r × (d_in + d_out), and 41,943,040 adapter parameters at 16 bytes are 0.67 GB. The 4-bit base is 4.0 GB. There is no cache to add, since training keeps none; add activations for your batch and sequence length on top before trusting a GPU that is close to the line.
print(16 * (4096 + 4096))
Show the answer
131072
r × (d_in + d_out) = 16 × 8,192 = 131,072 trainable numbers in place of the matrix's 16,777,216: 0.78 percent.
LoRA parameters for a model
Write lora_params(shapes, r): shapes is a list of (d_in, d_out) pairs, one per adapted matrix in one layer. Return the trainable parameters for one layer, sum(r * (d_in + d_out)).
def lora_params(shapes, r): return 0 shapes = [(4096, 4096), (4096, 1024), (4096, 1024), (4096, 4096), (4096, 14336), (4096, 14336), (14336, 4096)] print(lora_params(shapes, 16))
Show the answer
Each parameter also has a gradient (2), two Adam averages in 32-bit (8) and a 32-bit master copy (4)
2 + 2 + 8 + 4 = 16 bytes before activations. LoRA avoids all but the frozen weights' 2 (or 0.5 when quantized) for the base model.
Write a sentence or two, then compare it with a good answer in the lesson.
In production: read a model card with the formulas
A card says: 8 billion parameters, 32 layers, 8 key-value heads, head size 128, 131,072-token context. You have a 24 GB GPU. Weights at 4 bits, 4.0 GB; runtime, 1 GB; that leaves about 19 GB for the cache. The full context costs 17.18 GB for a single sequence, so it fits, with 1.8 GB to spare and no room for a second sequence. Had the model used 32 key-value heads, the same context would need 68.72 GB of cache and the card's window would be out of reach on this GPU at any precision.
The checklist: weights + cache + runtime, below the memory with some headroom, at the context and batch you will actually run; then speed, which is memory bandwidth divided by the bytes read per token, from the open-models lesson. What the formulas do not include: activations and attention workspace during a forward pass, memory fragmentation, and any second model (a draft model for speculative decoding, an embedding model) sharing the GPU. Treat the estimate as a floor with a margin of 10 to 20 percent. The next cell works the card.
A model card on a 24 GB GPU
The card's numbers through the formulas, at the full context, with 8 and with 32 KV heads.
def weights_gb(params_b, bits): return params_b * bits / 8 def kv_cache_gb(layers, kv_heads, head_dim, tokens, bytes_per_number=2, batch=1): return 2 * layers * kv_heads * head_dim * tokens * bytes_per_number * batch / 1e9 gpu, runtime = 24, 1.0 for kv_heads in (8, 32): w, kv = weights_gb(8, 4), kv_cache_gb(32, kv_heads, 128, 131072) total = w + kv + runtime verdict = (f"fits with {gpu - total:.2f} GB spare" if total <= gpu else "does not fit") print(f"{kv_heads} KV heads: weights {w:.1f} + cache {kv:.2f} " f"+ runtime {runtime:.0f} = {total:.2f} GB -> {verdict}")
With 8 key-value heads the full 131,072-token context fits with 1.82 GB to spare, a thin margin that activations would eat on a long prompt; a 65,536-token limit would be the safer setting. With 32 key-value heads the cache alone is 68.72 GB and the total 73.72 GB: no precision rescues it, because the cache does not depend on the weights' bits. That is the number to look for on a card: the KV heads, not only the parameter count.
Show the answer
Each new token reads every weight (and the cache) from GPU memory, which takes longer than the arithmetic
Decode is memory-bound: bytes read per token ÷ bandwidth sets the time, which is why fewer bytes (quantization, grouped-query attention) means more tokens per second.
Show the answer
8B at 4 bits, 32 layers, 8 KV heads, head 128, at 131,072 tokens: 4.0 + 17.18 + 1 = 22.18 GB
Only the first total, 22.18 GB, is under 24 GB. The others fail on cache (73.72 GB), on weights plus cache (34.18 GB) or on weights alone (36 GB).
Write a sentence or two, then compare it with a good answer in the lesson.
Practice what you just read
- 5 code examples you can edit and run
- a practice exercise that checks your code
- a 5-question quiz
- 9 review cards for spaced repetition
Try a quiz question
A 70B model at 4 bits needs about how much memory for its weights?
Before this lesson
Where this lesson leads.
575 lessons in thirteen phases. 176 are free without an account, and a free account opens the rest. See every lesson.
Phase 6 · about 35 hours of study
Modern LLM engineering
Post-training, fine-tuning, serving and safety
48 lessons 2 free
- Base models vs assistants FreeWhat post-training adds to a pretrained model.30 min
- Supervised fine-tuningTeach behavior from example conversations.40 min
- Instruction and preference dataThe datasets that shape assistant behavior.30 min
- RLHFReward models and reinforcement learning from human feedback.40 min
- DPO and direct preference methodsPreference training without a separate reward model.40 min
- Constitutional AITraining with AI feedback guided by written principles.35 min
- Training reasoning modelsReinforcement learning on verifiable problems.40 min
- Prompting vs RAG vs fine-tuningPick the cheapest approach that works.30 min
- LoRAFine-tune a big model by training small low-rank updates.40 min
- Fine-tune an open modelRun a real fine-tune and evaluate the result.45 min
- DistillationTrain a small model to imitate a large one.35 min
- Tuning an embedding model for retrievalWhen search fails on your own vocabulary: train an adapter with a contrastive loss on question and passage pairs, test on held-out passages, use batch size and hard negatives, and measure what tuning costs elsewhere.45 min
- QuantizationStore weights in fewer bits to run models cheaply.35 min
- Quantization in practice: zero points, calibration and formatsScale and zero point, per-channel scales, calibration sets, error on a layer's output, quantization-aware training and the file formats you meet at work.35 min
- Serving modelsBatching, throughput and latency in production.35 min
- One request through a serving stack: latency and costFollow one request through queue, prefill, a prefix cache, decode and the stream, and price a token from a GPU's hourly cost.35 min
- Speculative decodingDraft with a small model, verify with a big one.35 min
- Running open models locallyDownload, quantize and run a model on your own machine.30 min
- Will it fit? Memory for serving and fine-tuningWeights, KV cache, runtime and adapter states added up for a model card and a GPU, with a calculator.30 min
- InterpretabilityLooking inside models to see what they compute.40 min
- Confident and wrongWhy a model answers with the same assurance whether it is right, wrong or asked about something that does not exist.30 min
- Why it can't spell or count lettersThe model sees token IDs, and an embedding learned from usage carries nothing about spelling.25 min
- Copying from the promptIn-context learning as a two-head copying circuit: the base tiny GPT cannot copy a repeated pattern, a model trained on repeats grows an induction head and can.30 min
- Why temperature changes the answerThe same prompt at 0, 0.7 and 1.5: deterministic loops, plausible variety, broken grammar, measured on a real model.25 min
- Why long prompts diluteAttention is a budget shared by every token, so a fact far from the question loses its share; measured, with the cliff beyond the trained window.25 min
- Why it never says "I don't know"A chat model answers every request because its fine-tuning data always did; preference training adds flattery; how to build abstention in.25 min
- Benchmarks and their limitsWhat leaderboards measure, and what they miss.38 min
- Alignment and safetyMaking capable models behave as intended.35 min
- Evaluating agentsMeasure multi-step, tool-using behavior.35 min
- Reading AI papersGet the idea out of a research paper in 30 minutes.30 min
- Capstone: a tuned, evaluated assistantFine-tune, evaluate and deploy a specialist assistant.45 min
- Weight-space programming: what it is, and what has been shown Free OptionalA language model reads and writes another network's weights with no training loop: the published experiments, their limits and the criticism, and one edit you make by hand.30 min
- Anatomy of a checkpoint OptionalA model file is a set of named tensors: count them, find the ones that write to the residual stream, and save one to bytes and back.35 min
- Reading weights without running the model OptionalSingular values, and an attention head's token-to-token circuit, read from the matrices alone and then checked against a measurement.35 min
- Writing weights by hand OptionalSet a small network's 55 weights so it catches every ball in a paddle game, the way a program would be written, and meet compiled transformers.40 min
- Edits are deltas: task vectors, patches and hashes OptionalSubtract a base from a fine-tune and the edit becomes an object: scale it, take it away, ship it as a patch, and prove by hash what was applied.35 min
- Merging models OptionalAverage two fine-tunes, add their task vectors, try SLERP, TIES and DARE, measure the interference, and compare with a model trained on both tasks.40 min
- Low-rank edits OptionalCut a fine-tune's change down to its largest directions with the SVD: how few a behavior needs, and how to pull an adapter out of a delta.35 min
- Directions: steer the run or edit the weights OptionalFind a behavior's direction in the residual stream, add it at run time, project it out of the weights, and see the effect change from input to input.40 min
- Editing one fact OptionalA rank-one change to one matrix rewrites what a product is made from; then reliability, generalization, locality, and what breaks after many edits.40 min
- The editor's harness OptionalWhat an LLM that edits weights is given and what it is denied: weights as text and their cost in tokens, a sandbox, a budget of forward passes, and rules the harness enforces.40 min
- An LLM reads a network's weights OptionalSix recorded sessions: an LLM is handed 1,315 weights as text and asked what the network was trained to do. Score each answer against the network, and study one it got wrong.40 min
- An LLM fixes a model's weights OptionalFind a tiny GPT's loop by hand, then replay recorded sessions in which an LLM hunts for it with ten forward passes, and measure every fix on held-out questions.45 min
- An LLM writes a network's weights OptionalFive recorded sessions: an LLM writes every weight of a paddle network from the game's source and one sentence. Score each against its specification, test its design claims, and edit one by hand.40 min
- Did it work? Testing an edit OptionalReview a patch for a tiny GPT that can't stop talking: held-out questions, a paired comparison, a same-size control, and what the edit cost.40 min
- The published edit, on a real model OptionalRead the edit published for an open model of 870 million weights, apply and verify one in the same form, see what 16-bit and 4-bit storage do to a fix, and run the real thing from a notebook.45 min
- Safety, licenses and provenance OptionalBefore a custom model is released: trace its lineage by hash, work out what its parents' licenses pass on, see what an edit can do to a safeguard, and write its card.40 min
- Capstone: ship a custom model OptionalSet release gates, then diagnose, edit, measure and document a tiny GPT with a fault no lesson has fixed, and write the record a stranger could check.45 min
Never trained an AI? Teach a small one to write, right in your browser. It takes about 20 seconds and needs no account. Teach it to write