Quantization in practice: zero points, calibration and formats
Scale and zero point, per-channel scales, calibration sets, error on a layer's output, quantization-aware training and the file formats you meet at work.
What it covers11
Why it matters
Quantization rounds weights with one scale. Real quantization kits do more. They store a scale and a zero point, pick the range from calibration data, give each output row its own scale, and measure the error where it matters: on what the layer produces.
You will meet these choices as settings and file names, such as a 4-bit NormalFloat config or a file ending in Q4_K_M. This lesson shows what each one does to numbers you can print, so you can choose between them and check the result on your own evaluation before you ship.
What you'll be able to do
- Quantize a range with a scale and a zero point, by hand and in NumPy, and rebuild the numbers
- Say when a zero point and a clipped range beat a symmetric scale, and when clipping hurts
- Measure quantization error on a layer's output, with a calibration set kept apart from the test set
- Read a quantized model's file name and tool settings, and rerun your own evaluation at each precision
Show the answer
One scale for each output row
Rows differ in size, so a row with small weights wastes marks if it shares a scale with a row of large ones. A scale per row fits each row. The first cell below measures it.
The idea: a range, a scale and a zero point
This lesson rests on Quantization: store a weight as a whole number, and rebuild it by multiplying by a scale.
The earlier recipe used one number, the scale, with marks placed evenly on both sides of zero. Real kits keep two numbers, a scale and a zero point. Take a range from to and a count of bits. Spread marks over the range, numbered from 0 up:
In plain words: the scale is the gap between two marks. The zero point is the number of the mark that stands for 0.0, so the real number 0 is always stored exactly. is the number you get back after storing.
import numpy as np w = np.array([0.7, -0.33, 0.04]) scale = np.abs(w).max() / 7 print(round(scale, 2), np.round(w / scale).astype(int))
Show the answer
0.1 [ 7 -3 0]
The scale is 0.7 / 7 = 0.1. Then 0.7 / 0.1 = 7, -0.33 / 0.1 = -3.3 rounds to -3, and 0.04 / 0.1 = 0.4 rounds to 0.
Why a zero point
A symmetric scale puts marks evenly on both sides of zero. That wastes half the marks when the numbers are all positive, or nearly so. The outputs of a ReLU, and most outputs of the GELU used in the tiny GPT, sit above zero. Spreading the marks over the real range, from the smallest number to the largest, puts every mark to use. The step between marks is then about half as large, and so is the error.
Here is a range from -2 to 4 stored in 4 bits, which makes 16 marks. The scale is 6 / 15 = 0.4. The zero point is round(2 / 0.4) = 5, so mark 5 is 0.0. The number 1.5 lies between two marks. It lands on mark 9, which rebuilds as 1.6.
The largest possible error is half a gap, 0.2. Zero sits exactly on a mark.
scale = (4 - -2) / 255 zero = round(2 / scale) q = round(1.5 / scale) + zero print(round(scale, 4), zero, q)
Show the answer
0.0235 85 149
The scale is 6 / 255 = 0.0235. The zero point is round(2 / 0.0235) = 85. The number 1.5 / 0.0235 = 63.75 rounds to 64, and adding the zero point gives mark 149.
Outliers set the range, so clip them
If one huge number sets the top of the range, every other number shares few marks. You can clip: choose a range that leaves out the rarest values, such as the 0.1th to the 99.9th percentile. A clipped value is stored at the end of the range, so it carries an error. The rest get a smaller gap. Clipping is a trade: a little error on a few numbers for a smaller error on the many. It pays at few bits and can lose at many, as the second cell shows.
Write a sentence or two, then compare it with a good answer in the lesson.
In code: rows, calibration and the layer's output
The layer here is blocks.0.mlp.2.weight of the tiny GPT. It is a 32 by 128 matrix that reads a key, a list of 128 hidden activations, and writes 32 numbers to the stream (see Editing one fact). The setup above loads it, and gets the keys it reads on 64 catalog descriptions. Those 64 are the calibration set: the sample you use to choose ranges. It also gets the keys from 64 other descriptions, which are the test set.
Two habits matter more than any one formula. Measure the error on the layer's output, because that is what the model uses. And measure it on data you did not calibrate on.
One scale for the matrix, or one per row
Per tensor against per channel, measured on the output
Round the weights to 8, 6, 4 and 3 bits with one scale for the whole matrix, then with one scale for each output row (a channel). For each: the error in the weights, and the error in the layer's output on the test keys. Errors are relative: the size of the error divided by the size of the true value.
Setup (runs before every cell) · 26 lines
The lesson runs this once, before its first cell, and every cell, drill and example starts with what it defines. A cell that changes a setup object changes it for the cells after it, until the page reloads.
import numpy as np import wsp state, info = wsp.load_state("tiny-gpt") model, tok = wsp.build(state, info) W = state["blocks.0.mlp.2.weight"].astype(np.float64) # 32 x 128: reads a key def keys(texts): return np.concatenate([wsp.mlp_keys(model, tok, t, block=0) for t in texts]) K, K2 = keys(wsp.catalog(0, 64)), keys(wsp.catalog(64, 128)) # calibrate, then test def sym(x, bits, axis=None): levels = 2 ** (bits - 1) - 1 scale = np.abs(x).max(axis=axis, keepdims=True) / levels return np.round(x / scale).clip(-levels, levels) * scale def asym(x, bits, lo, hi): levels = 2 ** bits - 1 scale = (hi - lo) / levels zero = round(-lo / scale) q = np.clip(np.round(x / scale) + zero, 0, levels) return (q - zero) * scale def out_error(Wq, keys): return np.linalg.norm(keys @ (W - Wq).T) / np.linalg.norm(keys @ W.T)
print("weights", W.shape, "| calibration keys", K.shape, "| test keys", K2.shape) print(f"{'bits':>4} {'tensor: weights':>16} {'output':>8}", f"{'channel: weights':>17} {'output':>8}") for bits in (8, 6, 4, 3): row = [bits] for axis in (None, 1): Wq = sym(W, bits, axis) row += [np.linalg.norm(W - Wq) / np.linalg.norm(W), out_error(Wq, K2)] print(f"{row[0]:>4} {row[1]:>16.4f} {row[2]:>8.4f} {row[3]:>17.4f} {row[4]:>8.4f}")
At 4 bits the weights are off by 14% with one scale and 11% with one per row. The output errors are smaller, 10% and 8%, because the errors partly cancel when a row's weights are added up.
Read the table by column. A scale per row beats a scale for the whole matrix at every bit count, since each row is fitted to its own largest weight. And the output error is smaller than the weight error: a layer adds up many weights, and rounding errors that point both ways cancel. That is why you measure the output. Two roundings with the same weight error can differ in output error.
Show the answer
The layer adds up many weights, and rounding errors that point in opposite directions partly cancel
Each output is a sum over 128 products. Some rounding errors raise it and some lower it, so the sum's error is smaller than the errors in its parts.
Now plot the error against the bits for both choices.
Uses the setup above, 26 lines.
import matplotlib.pyplot as plt bits = [2, 3, 4, 5, 6, 7, 8] for label, axis in [("one scale per matrix", None), ("one scale per row", 1)]: errors = [out_error(sym(W, b, axis), K2) for b in bits] plt.plot(bits, errors, marker="o", label=label) plt.yscale("log") plt.xlabel("bits per weight") plt.ylabel("relative error on the layer's output") plt.title("Rounding one layer") plt.legend() plt.show()
Quantize with a scale and a zero point
Write affine(x, bits, lo, hi): return (q, scale, zero). x is a NumPy array, scale is (hi - lo) / (2 ** bits - 1), zero is round(-lo / scale), and q is np.round(x / scale) + zero, clipped to the range 0 to 2 ** bits - 1.
Uses the setup above, 26 lines.
import numpy as np def affine(x, bits, lo, hi): return x, 1.0, 0 q, scale, zero = affine(np.array([-2.0, 0.0, 1.5, 4.0]), 8, -2.0, 4.0) print(q, round(scale, 4), zero)
Calibration: choose the range from data
For weights the range is known: it is the weights. For the activations, the numbers that flow through the layer, the range depends on the input. So you run a calibration set through the model and record the range, then keep that range fixed. Here the keys have a smallest value of -0.17 and a largest of 3.96, so they are nearly all above zero. The next cell stores the keys at 8, 6 and 4 bits three ways: symmetric, the full range with a zero point, and a range clipped to the 0.1th to 99.9th percentile of the calibration keys.
Three ranges for the activations
Quantize the test keys with a range chosen from the calibration keys. For each way and bit count, the error in the layer's output.
Uses the setup above, 26 lines.
print(f"keys: min {K.min():.2f}, max {K.max():.2f}, below zero {(K < 0).mean():.0%}") lo, hi = np.percentile(K, [0.1, 99.9]) print(f"{'bits':>4} {'symmetric':>10} {'min to max':>11} {'clipped':>8}") for bits in (8, 6, 4): variants = [sym(K2, bits), asym(K2, bits, K.min(), K.max()), asym(K2, bits, lo, hi)] true = np.linalg.norm(K2 @ W.T) errs = [np.linalg.norm((Kq - K2) @ W.T) / true for Kq in variants] print(f"{bits:>4} {errs[0]:>10.4f} {errs[1]:>11.4f} {errs[2]:>8.4f}")
Symmetric is worst at every size: a mark on the negative side is almost never used. Using the full range with a zero point cuts the 4-bit error from 0.1771 to 0.1071. Clipping helps further at 4 bits (0.0784) and hurts at 8 bits (0.0146 against 0.0056): at 8 bits the gap is already tiny, so the clipped tail costs more than it saves.
Show the answer
At 8 bits the gaps are already small, so the error from clipping the tail outweighs the gain; at 4 bits the gaps are large, so shrinking them pays
Clipping adds an error on the few values beyond the cut and reduces the gap for all the rest. The gap matters most when there are few marks.
Weights that matter more: scale before you round
The output error depends on how large each input is. A weight that multiplies a large key counts for more than one that multiplies a small key. This is the idea behind methods such as AWQ and GPTQ. The simple version: multiply each column of the weights by a number that grows with that key's typical size, divide the matching key by the same number, and round the scaled weights. The product is unchanged before rounding. After it, the heavily used columns have been spread over more marks. The cell tries scales that are the key's mean size raised to a power : is plain rounding.
Scale columns by the key's size, then round
Round the weights to 4 bits per row after multiplying column j by mean_size[j] ** alpha, then undo the scale. For each alpha: the output error on the calibration keys and on the test keys.
Uses the setup above, 26 lines.
mean_size = np.abs(K).mean(axis=0) + 1e-6 print(f"{'alpha':>5} {'calibration':>12} {'test':>8}") for alpha in (0, 0.25, 0.5, 0.75, 1.0): s = mean_size ** alpha Wq = sym(W * s, 4, axis=1) / s print(f"{alpha:>5} {out_error(Wq, K):>12.4f} {out_error(Wq, K2):>8.4f}")
The best alpha on the calibration keys, 0.5, also helps on the test keys, a smaller gain than the one for per-row scales. Too much scaling hurts: alpha 1.0 is worse than none.
Measure error where it matters
Write output_error(W, Wq, X): the size of X @ (W - Wq).T divided by the size of X @ W.T, where size is np.linalg.norm. X holds one input per row.
Uses the setup above, 26 lines.
import numpy as np def output_error(W, Wq, X): return 0.0 W = np.array([[1.0, 2.0], [3.0, 4.0]]) Wq = np.array([[1.0, 2.0], [3.0, 3.0]]) X = np.array([[1.0, 0.0], [0.0, 1.0]]) print(round(output_error(W, Wq, X), 4))
Write a sentence or two, then compare it with a good answer in the lesson.
Training with rounding in the loop
Everything so far rounds after training, which is post-training quantization. If you can train again, you can let the model adapt to the rounding. In quantization-aware training the forward pass uses the rounded weights, so the loss sees the damage. The gradient has a problem: rounding is flat almost everywhere, so its slope is zero. The fix is the straight-through estimator: compute the gradient at the rounded weights and apply it to the float weights as if rounding were not there.
The cell below trains a six-weight linear model to a 2-bit grid, where each weight can only be -0.4, 0 or 0.4. It compares rounding the best float weights after training with training that sees the rounding.
2 bits after training, against 2 bits in the loop
The data has correlated inputs, so one weight can make up for another. Print the error of the float model, of the float model rounded after training, and of a model trained with the straight-through estimator.
Uses the setup above, 26 lines.
rng = np.random.default_rng(0) Z = rng.normal(size=(400, 6)) X = Z @ rng.normal(size=(6, 6)) * 0.6 + Z # correlated inputs w_true = np.array([0.9, -0.5, 0.3, 0.7, -0.2, 0.4]) y = X @ w_true STEP = 0.4 # 2 bits: -1, 0 or 1 steps def fake(w): return np.clip(np.round(w / STEP), -1, 1) * STEP def mse(w): return float(np.mean((X @ w - y) ** 2)) w_float = np.linalg.lstsq(X, y, rcond=None)[0] print("float weights:", round(mse(w_float), 4)) print("rounded after training:", round(mse(fake(w_float)), 4)) w = w_float.copy() for _ in range(300): grad = 2 * X.T @ (X @ fake(w) - y) / len(X) # gradient at the rounded weights w -= 0.02 * grad # straight-through: apply to w print("trained with rounding:", round(mse(fake(w)), 4), "| weights", fake(w))
Rounding the trained weights costs an error of 1.5088. Training with the rounding in the loop gets 0.9266 on the same grid. The weights are all on the grid, and the model has adapted to them.
Write a sentence or two, then compare it with a good answer in the lesson.
In production: formats, tools and your own evaluation
At work you rarely write the rounding yourself. You pick a format and a tool, and you read the names. A file called model-Q4_K_M.gguf carries its scheme in the name. The code below reads such a name the way llama.cpp's naming describes it: the number after Q is the bits, K means the "K-quant" family where each block has its own scale and minimum, and the last letter picks a small, medium or large mix of precisions across the tensors. It also works out the real cost per weight of a block: 4-bit weights, 32 per block, plus one 16-bit scale per block.
Read a format name, and cost a block
Split three file-name tags into bits, family and size, then compute the bits per weight for 4-bit blocks of 32 with one 16-bit scale.
Uses the setup above, 26 lines.
import re def read_tag(tag): m = re.fullmatch(r"Q(\d)_(K_)?([A-Z0-9]+)", tag) return {"bits": int(m.group(1)), "k_quant": bool(m.group(2)), "variant": m.group(3)} for tag in ("Q8_0", "Q4_0", "Q4_K_M"): print(tag, read_tag(tag)) block, bits, scale_bits = 32, 4, 16 print("bits per weight:", (block * bits + scale_bits) / block)
A tag only names a scheme. The honest cost per weight includes the scales: 4-bit blocks of 32 with a 16-bit scale cost 4.5 bits, not 4.
Rerun your own evaluation at each precision
The last step is the one that counts. A quantized model is a different model, so you score it on your own evaluation, at the precision you will ship. The cell scores the healthy fine-tune from Edits are deltas: task vectors, patches and hashes at 16 bits and at 8, 6, 5, 4 and 3 bits, on the 72 held-out questions and on ordinary text.
The same evaluation at six precisions
Quantize every weight matrix of tiny-gpt-uses and score it on the held-out questions and on the catalog text.
Uses the setup above, 26 lines.
uses, info2 = wsp.load_state("tiny-gpt-uses") tok2 = wsp.build(uses, info2)[1] docs = wsp.catalog() versions = [("bfloat16", wsp.to_bfloat16(uses))] versions += [(f"{b} bits", wsp.quantize(uses, b)) for b in (8, 6, 5, 4, 3)] print(f"{'precision':>10} {'held-out right':>15} {'catalog loss':>13}") for label, st in versions: m, _ = wsp.build(st, info2) right, loss = wsp.score(m, tok2, 'uses'), wsp.text_loss(m, tok2, docs) print(f"{label:>10} {right:>12}/72 {loss:>13.2f}")
The model is fine down to 6 bits, loses a little at 4 (71 of 72, loss 1.25) and falls apart at 3 (49 of 72, loss 1.83). Where it breaks is a fact about this model and this test, so run yours.
Show the answer
Run your own task evaluation at 4 bits, since a task can break while average text loss barely changes
Quantization damage is uneven. A model can keep its average loss and lose a skill your product needs.
Where it shows up in your week
You will pick a format by what the hardware can run, then check it. The checks that catch real problems are the same each time: a calibration set that looks like your real prompts, a test set kept apart, the error measured on outputs and then on your own task, and the release record naming the precision. What goes wrong is rarely the rounding. It is a calibration set from the wrong kind of text, a format whose scales were stored in too few bits, or a model shipped at a precision that was never scored.
Write a sentence or two, then compare it with a good answer in the lesson.
Goes deeper in
- Will it fit? Memory for serving and fine-tuning: memory for weights, caches and training
- One request through a serving stack: latency and cost: what a quantized model costs per token to serve
- The published edit, on a real model: an edit that survives or fails rounding
Practice what you just read
- 7 code examples you can edit and run
- a practice exercise that checks your code
- a 5-question quiz
- 9 review cards for spaced repetition
Try a quiz question
A tensor holds numbers between -2 and 4. Which stored pair rebuilds 0.0 exactly?
Before this lesson
Where this lesson leads.
575 lessons in thirteen phases. 176 are free without an account, and a free account opens the rest. See every lesson.
Phase 6 · about 35 hours of study
Modern LLM engineering
Post-training, fine-tuning, serving and safety
48 lessons 2 free
- Base models vs assistants FreeWhat post-training adds to a pretrained model.30 min
- Supervised fine-tuningTeach behavior from example conversations.40 min
- Instruction and preference dataThe datasets that shape assistant behavior.30 min
- RLHFReward models and reinforcement learning from human feedback.40 min
- DPO and direct preference methodsPreference training without a separate reward model.40 min
- Constitutional AITraining with AI feedback guided by written principles.35 min
- Training reasoning modelsReinforcement learning on verifiable problems.40 min
- Prompting vs RAG vs fine-tuningPick the cheapest approach that works.30 min
- LoRAFine-tune a big model by training small low-rank updates.40 min
- Fine-tune an open modelRun a real fine-tune and evaluate the result.45 min
- DistillationTrain a small model to imitate a large one.35 min
- Tuning an embedding model for retrievalWhen search fails on your own vocabulary: train an adapter with a contrastive loss on question and passage pairs, test on held-out passages, use batch size and hard negatives, and measure what tuning costs elsewhere.45 min
- QuantizationStore weights in fewer bits to run models cheaply.35 min
- Quantization in practice: zero points, calibration and formatsScale and zero point, per-channel scales, calibration sets, error on a layer's output, quantization-aware training and the file formats you meet at work.35 min
- Serving modelsBatching, throughput and latency in production.35 min
- One request through a serving stack: latency and costFollow one request through queue, prefill, a prefix cache, decode and the stream, and price a token from a GPU's hourly cost.35 min
- Speculative decodingDraft with a small model, verify with a big one.35 min
- Running open models locallyDownload, quantize and run a model on your own machine.30 min
- Will it fit? Memory for serving and fine-tuningWeights, KV cache, runtime and adapter states added up for a model card and a GPU, with a calculator.30 min
- InterpretabilityLooking inside models to see what they compute.40 min
- Confident and wrongWhy a model answers with the same assurance whether it is right, wrong or asked about something that does not exist.30 min
- Why it can't spell or count lettersThe model sees token IDs, and an embedding learned from usage carries nothing about spelling.25 min
- Copying from the promptIn-context learning as a two-head copying circuit: the base tiny GPT cannot copy a repeated pattern, a model trained on repeats grows an induction head and can.30 min
- Why temperature changes the answerThe same prompt at 0, 0.7 and 1.5: deterministic loops, plausible variety, broken grammar, measured on a real model.25 min
- Why long prompts diluteAttention is a budget shared by every token, so a fact far from the question loses its share; measured, with the cliff beyond the trained window.25 min
- Why it never says "I don't know"A chat model answers every request because its fine-tuning data always did; preference training adds flattery; how to build abstention in.25 min
- Benchmarks and their limitsWhat leaderboards measure, and what they miss.38 min
- Alignment and safetyMaking capable models behave as intended.35 min
- Evaluating agentsMeasure multi-step, tool-using behavior.35 min
- Reading AI papersGet the idea out of a research paper in 30 minutes.30 min
- Capstone: a tuned, evaluated assistantFine-tune, evaluate and deploy a specialist assistant.45 min
- Weight-space programming: what it is, and what has been shown Free OptionalA language model reads and writes another network's weights with no training loop: the published experiments, their limits and the criticism, and one edit you make by hand.30 min
- Anatomy of a checkpoint OptionalA model file is a set of named tensors: count them, find the ones that write to the residual stream, and save one to bytes and back.35 min
- Reading weights without running the model OptionalSingular values, and an attention head's token-to-token circuit, read from the matrices alone and then checked against a measurement.35 min
- Writing weights by hand OptionalSet a small network's 55 weights so it catches every ball in a paddle game, the way a program would be written, and meet compiled transformers.40 min
- Edits are deltas: task vectors, patches and hashes OptionalSubtract a base from a fine-tune and the edit becomes an object: scale it, take it away, ship it as a patch, and prove by hash what was applied.35 min
- Merging models OptionalAverage two fine-tunes, add their task vectors, try SLERP, TIES and DARE, measure the interference, and compare with a model trained on both tasks.40 min
- Low-rank edits OptionalCut a fine-tune's change down to its largest directions with the SVD: how few a behavior needs, and how to pull an adapter out of a delta.35 min
- Directions: steer the run or edit the weights OptionalFind a behavior's direction in the residual stream, add it at run time, project it out of the weights, and see the effect change from input to input.40 min
- Editing one fact OptionalA rank-one change to one matrix rewrites what a product is made from; then reliability, generalization, locality, and what breaks after many edits.40 min
- The editor's harness OptionalWhat an LLM that edits weights is given and what it is denied: weights as text and their cost in tokens, a sandbox, a budget of forward passes, and rules the harness enforces.40 min
- An LLM reads a network's weights OptionalSix recorded sessions: an LLM is handed 1,315 weights as text and asked what the network was trained to do. Score each answer against the network, and study one it got wrong.40 min
- An LLM fixes a model's weights OptionalFind a tiny GPT's loop by hand, then replay recorded sessions in which an LLM hunts for it with ten forward passes, and measure every fix on held-out questions.45 min
- An LLM writes a network's weights OptionalFive recorded sessions: an LLM writes every weight of a paddle network from the game's source and one sentence. Score each against its specification, test its design claims, and edit one by hand.40 min
- Did it work? Testing an edit OptionalReview a patch for a tiny GPT that can't stop talking: held-out questions, a paired comparison, a same-size control, and what the edit cost.40 min
- The published edit, on a real model OptionalRead the edit published for an open model of 870 million weights, apply and verify one in the same form, see what 16-bit and 4-bit storage do to a fix, and run the real thing from a notebook.45 min
- Safety, licenses and provenance OptionalBefore a custom model is released: trace its lineage by hash, work out what its parents' licenses pass on, see what an edit can do to a safeguard, and write its card.40 min
- Capstone: ship a custom model OptionalSet release gates, then diagnose, edit, measure and document a tiny GPT with a fault no lesson has fixed, and write the record a stranger could check.45 min
Never trained an AI? Teach a small one to write, right in your browser. It takes about 20 seconds and needs no account. Teach it to write