Index Ai Academy
Phase 6 · Inference

Quantization in practice: zero points, calibration and formats

Scale and zero point, per-channel scales, calibration sets, error on a layer's output, quantization-aware training and the file formats you meet at work.

About 35 minutes · A free account opens the whole lesson.
Already have an account? Sign in to open this lesson

What it covers11

Why it matters

Quantization rounds weights with one scale. Real quantization kits do more. They store a scale and a zero point, pick the range from calibration data, give each output row its own scale, and measure the error where it matters: on what the layer produces.

You will meet these choices as settings and file names, such as a 4-bit NormalFloat config or a file ending in Q4_K_M. This lesson shows what each one does to numbers you can print, so you can choose between them and check the result on your own evaluation before you ship.

What you'll be able to do

  • Quantize a range with a scale and a zero point, by hand and in NumPy, and rebuild the numbers
  • Say when a zero point and a clipped range beat a symmetric scale, and when clipping hurts
  • Measure quantization error on a layer's output, with a calibration set kept apart from the test set
  • Read a quantized model's file name and tool settings, and rerun your own evaluation at each precision
Quick check
A weight matrix is rounded to 4 bits. A guess: which choice gives the smaller error on what the layer produces?
  1. One scale for the whole matrix
  2. Neither: error depends only on the number of bits
  3. One scale for each output row
  4. They are always equal, because rounding is rounding
Show the answer

One scale for each output row

Rows differ in size, so a row with small weights wastes marks if it shares a scale with a row of large ones. A scale per row fits each row. The first cell below measures it.

The idea: a range, a scale and a zero point

This lesson rests on Quantization: store a weight as a whole number, and rebuild it by multiplying by a scale.

The earlier recipe used one number, the scale, with marks placed evenly on both sides of zero. Real kits keep two numbers, a scale and a zero point. Take a range from lolo to hihi and a count of bits. Spread 2bits2^{bits} marks over the range, numbered from 0 up:

scale=hi−lo2bits−1,zero=round ⁣(−loscale),q=round ⁣(xscale)+zero,x^=(q−zero)×scale\text{scale} = \frac{hi - lo}{2^{bits} - 1}, \quad \text{zero} = \text{round}\!\left(\frac{-lo}{\text{scale}}\right), \quad q = \text{round}\!\left(\frac{x}{\text{scale}}\right) + \text{zero}, \quad \hat{x} = (q - \text{zero}) \times \text{scale}

In plain words: the scale is the gap between two marks. The zero point is the number of the mark that stands for 0.0, so the real number 0 is always stored exactly. x^\hat{x} is the number you get back after storing.

Quick check
From the lesson on quantization: at 4 bits there are 7 levels on each side of zero. The largest weight in a group is 0.7. What does this print: the scale, and the whole numbers for the weights 0.7, -0.33 and 0.04?
import numpy as np

w = np.array([0.7, -0.33, 0.04])
scale = np.abs(w).max() / 7
print(round(scale, 2), np.round(w / scale).astype(int))
  1. 0.05 [14 -7 1]
  2. 0.1 [ 7 -4 0]
  3. 0.7 [ 1 0 0]
  4. 0.1 [ 7 -3 0]
Show the answer

0.1 [ 7 -3 0]

The scale is 0.7 / 7 = 0.1. Then 0.7 / 0.1 = 7, -0.33 / 0.1 = -3.3 rounds to -3, and 0.04 / 0.1 = 0.4 rounds to 0.

Why a zero point

A symmetric scale puts marks evenly on both sides of zero. That wastes half the marks when the numbers are all positive, or nearly so. The outputs of a ReLU, and most outputs of the GELU used in the tiny GPT, sit above zero. Spreading the marks over the real range, from the smallest number to the largest, puts every mark to use. The step between marks is then about half as large, and so is the error.

Here is a range from -2 to 4 stored in 4 bits, which makes 16 marks. The scale is 6 / 15 = 0.4. The zero point is round(2 / 0.4) = 5, so mark 5 is 0.0. The number 1.5 lies between two marks. It lands on mark 9, which rebuilds as 1.6.

Try it now

The largest possible error is half a gap, 0.2. Zero sits exactly on a mark.

Quick check
The range is -2 to 4 at 8 bits (256 marks). What does this print: the scale, the zero point, and the mark that stores 1.5?
scale = (4 - -2) / 255
zero = round(2 / scale)
q = round(1.5 / scale) + zero
print(round(scale, 4), zero, q)
  1. 0.4 5 9
  2. 0.0235 128 192
  3. 0.0235 85 149
  4. 0.0235 85 64
Show the answer

0.0235 85 149

The scale is 6 / 255 = 0.0235. The zero point is round(2 / 0.0235) = 85. The number 1.5 / 0.0235 = 63.75 rounds to 64, and adding the zero point gives mark 149.

Outliers set the range, so clip them

If one huge number sets the top of the range, every other number shares few marks. You can clip: choose a range that leaves out the rarest values, such as the 0.1th to the 99.9th percentile. A clipped value is stored at the end of the range, so it carries an error. The rest get a smaller gap. Clipping is a trade: a little error on a few numbers for a smaller error on the many. It pays at few bits and can lose at many, as the second cell shows.

In your own words
In your own words: why does a zero point halve the error for numbers that are all positive?

Write a sentence or two, then compare it with a good answer in the lesson.

In code: rows, calibration and the layer's output

The layer here is blocks.0.mlp.2.weight of the tiny GPT. It is a 32 by 128 matrix that reads a key, a list of 128 hidden activations, and writes 32 numbers to the stream (see Editing one fact). The setup above loads it, and gets the keys it reads on 64 catalog descriptions. Those 64 are the calibration set: the sample you use to choose ranges. It also gets the keys from 64 other descriptions, which are the test set.

Two habits matter more than any one formula. Measure the error on the layer's output, because that is what the model uses. And measure it on data you did not calibrate on.

One scale for the matrix, or one per row

Worked example

Per tensor against per channel, measured on the output

Round the weights to 8, 6, 4 and 3 bits with one scale for the whole matrix, then with one scale for each output row (a channel). For each: the error in the weights, and the error in the layer's output on the test keys. Errors are relative: the size of the error divided by the size of the true value.

Setup (runs before every cell) · 26 lines

The lesson runs this once, before its first cell, and every cell, drill and example starts with what it defines. A cell that changes a setup object changes it for the cells after it, until the page reloads.

import numpy as np
import wsp

state, info = wsp.load_state("tiny-gpt")
model, tok = wsp.build(state, info)
W = state["blocks.0.mlp.2.weight"].astype(np.float64)   # 32 x 128: reads a key

def keys(texts):
    return np.concatenate([wsp.mlp_keys(model, tok, t, block=0) for t in texts])

K, K2 = keys(wsp.catalog(0, 64)), keys(wsp.catalog(64, 128))   # calibrate, then test

def sym(x, bits, axis=None):
    levels = 2 ** (bits - 1) - 1
    scale = np.abs(x).max(axis=axis, keepdims=True) / levels
    return np.round(x / scale).clip(-levels, levels) * scale

def asym(x, bits, lo, hi):
    levels = 2 ** bits - 1
    scale = (hi - lo) / levels
    zero = round(-lo / scale)
    q = np.clip(np.round(x / scale) + zero, 0, levels)
    return (q - zero) * scale

def out_error(Wq, keys):
    return np.linalg.norm(keys @ (W - Wq).T) / np.linalg.norm(keys @ W.T)
print("weights", W.shape, "| calibration keys", K.shape, "| test keys", K2.shape)
print(f"{'bits':>4} {'tensor: weights':>16} {'output':>8}",
      f"{'channel: weights':>17} {'output':>8}")
for bits in (8, 6, 4, 3):
    row = [bits]
    for axis in (None, 1):
        Wq = sym(W, bits, axis)
        row += [np.linalg.norm(W - Wq) / np.linalg.norm(W), out_error(Wq, K2)]
    print(f"{row[0]:>4} {row[1]:>16.4f} {row[2]:>8.4f} {row[3]:>17.4f} {row[4]:>8.4f}")
In the lesson you can change this and run it.

At 4 bits the weights are off by 14% with one scale and 11% with one per row. The output errors are smaller, 10% and 8%, because the errors partly cancel when a row's weights are added up.

Read the table by column. A scale per row beats a scale for the whole matrix at every bit count, since each row is fitted to its own largest weight. And the output error is smaller than the weight error: a layer adds up many weights, and rounding errors that point both ways cancel. That is why you measure the output. Two roundings with the same weight error can differ in output error.

Quick check
In the table, the 4-bit weights are off by 0.1423 with one scale and the layer's output by 0.0999. Why is the output error smaller than the weight error?
  1. The layer adds up many weights, and rounding errors that point in opposite directions partly cancel
  2. The output is rounded to more bits than the weights are, so its error is smaller by design
  3. Output error is always one third of weight error
  4. The test keys were chosen to hide the error
Show the answer

The layer adds up many weights, and rounding errors that point in opposite directions partly cancel

Each output is a sum over 128 products. Some rounding errors raise it and some lower it, so the sum's error is smaller than the errors in its parts.

Now plot the error against the bits for both choices.

Uses the setup above, 26 lines.

import matplotlib.pyplot as plt

bits = [2, 3, 4, 5, 6, 7, 8]
for label, axis in [("one scale per matrix", None), ("one scale per row", 1)]:
    errors = [out_error(sym(W, b, axis), K2) for b in bits]
    plt.plot(bits, errors, marker="o", label=label)
plt.yscale("log")
plt.xlabel("bits per weight")
plt.ylabel("relative error on the layer's output")
plt.title("Rounding one layer")
plt.legend()
plt.show()
Drill

Quantize with a scale and a zero point

Write affine(x, bits, lo, hi): return (q, scale, zero). x is a NumPy array, scale is (hi - lo) / (2 ** bits - 1), zero is round(-lo / scale), and q is np.round(x / scale) + zero, clipped to the range 0 to 2 ** bits - 1.

Uses the setup above, 26 lines.

import numpy as np

def affine(x, bits, lo, hi):
    return x, 1.0, 0

q, scale, zero = affine(np.array([-2.0, 0.0, 1.5, 4.0]), 8, -2.0, 4.0)
print(q, round(scale, 4), zero)
In the lesson, your code is checked as you write it.

Calibration: choose the range from data

For weights the range is known: it is the weights. For the activations, the numbers that flow through the layer, the range depends on the input. So you run a calibration set through the model and record the range, then keep that range fixed. Here the keys have a smallest value of -0.17 and a largest of 3.96, so they are nearly all above zero. The next cell stores the keys at 8, 6 and 4 bits three ways: symmetric, the full range with a zero point, and a range clipped to the 0.1th to 99.9th percentile of the calibration keys.

Worked example

Three ranges for the activations

Quantize the test keys with a range chosen from the calibration keys. For each way and bit count, the error in the layer's output.

Uses the setup above, 26 lines.

print(f"keys: min {K.min():.2f}, max {K.max():.2f}, below zero {(K < 0).mean():.0%}")
lo, hi = np.percentile(K, [0.1, 99.9])
print(f"{'bits':>4} {'symmetric':>10} {'min to max':>11} {'clipped':>8}")
for bits in (8, 6, 4):
    variants = [sym(K2, bits), asym(K2, bits, K.min(), K.max()), asym(K2, bits, lo, hi)]
    true = np.linalg.norm(K2 @ W.T)
    errs = [np.linalg.norm((Kq - K2) @ W.T) / true for Kq in variants]
    print(f"{bits:>4} {errs[0]:>10.4f} {errs[1]:>11.4f} {errs[2]:>8.4f}")
In the lesson you can change this and run it.

Symmetric is worst at every size: a mark on the negative side is almost never used. Using the full range with a zero point cuts the 4-bit error from 0.1771 to 0.1071. Clipping helps further at 4 bits (0.0784) and hurts at 8 bits (0.0146 against 0.0056): at 8 bits the gap is already tiny, so the clipped tail costs more than it saves.

Quick check
At 8 bits the clipped range had a larger error than the full range, but at 4 bits it had a smaller one. Why?
  1. Clipping only works on weights, not activations
  2. The clipped range uses fewer marks at 8 bits
  3. At 8 bits the gaps are already small, so the error from clipping the tail outweighs the gain; at 4 bits the gaps are large, so shrinking them pays
  4. The percentile was computed wrongly at 8 bits, because the tail holds too few values to rank at that width, so the cut lands in the wrong place
Show the answer

At 8 bits the gaps are already small, so the error from clipping the tail outweighs the gain; at 4 bits the gaps are large, so shrinking them pays

Clipping adds an error on the few values beyond the cut and reduces the gap for all the rest. The gap matters most when there are few marks.

Weights that matter more: scale before you round

The output error depends on how large each input is. A weight that multiplies a large key counts for more than one that multiplies a small key. This is the idea behind methods such as AWQ and GPTQ. The simple version: multiply each column of the weights by a number sjs_j that grows with that key's typical size, divide the matching key by the same number, and round the scaled weights. The product is unchanged before rounding. After it, the heavily used columns have been spread over more marks. The cell tries scales that are the key's mean size raised to a power α\alpha: α=0\alpha = 0 is plain rounding.

Worked example

Scale columns by the key's size, then round

Round the weights to 4 bits per row after multiplying column j by mean_size[j] ** alpha, then undo the scale. For each alpha: the output error on the calibration keys and on the test keys.

Uses the setup above, 26 lines.

mean_size = np.abs(K).mean(axis=0) + 1e-6
print(f"{'alpha':>5} {'calibration':>12} {'test':>8}")
for alpha in (0, 0.25, 0.5, 0.75, 1.0):
    s = mean_size ** alpha
    Wq = sym(W * s, 4, axis=1) / s
    print(f"{alpha:>5} {out_error(Wq, K):>12.4f} {out_error(Wq, K2):>8.4f}")
In the lesson you can change this and run it.

The best alpha on the calibration keys, 0.5, also helps on the test keys, a smaller gain than the one for per-row scales. Too much scaling hurts: alpha 1.0 is worse than none.

Drill

Measure error where it matters

Write output_error(W, Wq, X): the size of X @ (W - Wq).T divided by the size of X @ W.T, where size is np.linalg.norm. X holds one input per row.

Uses the setup above, 26 lines.

import numpy as np

def output_error(W, Wq, X):
    return 0.0

W = np.array([[1.0, 2.0], [3.0, 4.0]])
Wq = np.array([[1.0, 2.0], [3.0, 3.0]])
X = np.array([[1.0, 0.0], [0.0, 1.0]])
print(round(output_error(W, Wq, X), 4))
In the lesson, your code is checked as you write it.
In your own words
In your own words: why do you calibrate on one set of prompts and test on another?

Write a sentence or two, then compare it with a good answer in the lesson.

Training with rounding in the loop

Everything so far rounds after training, which is post-training quantization. If you can train again, you can let the model adapt to the rounding. In quantization-aware training the forward pass uses the rounded weights, so the loss sees the damage. The gradient has a problem: rounding is flat almost everywhere, so its slope is zero. The fix is the straight-through estimator: compute the gradient at the rounded weights and apply it to the float weights as if rounding were not there.

The cell below trains a six-weight linear model to a 2-bit grid, where each weight can only be -0.4, 0 or 0.4. It compares rounding the best float weights after training with training that sees the rounding.

Worked example

2 bits after training, against 2 bits in the loop

The data has correlated inputs, so one weight can make up for another. Print the error of the float model, of the float model rounded after training, and of a model trained with the straight-through estimator.

Uses the setup above, 26 lines.

rng = np.random.default_rng(0)
Z = rng.normal(size=(400, 6))
X = Z @ rng.normal(size=(6, 6)) * 0.6 + Z                    # correlated inputs
w_true = np.array([0.9, -0.5, 0.3, 0.7, -0.2, 0.4])
y = X @ w_true

STEP = 0.4                                                  # 2 bits: -1, 0 or 1 steps
def fake(w):
    return np.clip(np.round(w / STEP), -1, 1) * STEP
def mse(w):
    return float(np.mean((X @ w - y) ** 2))

w_float = np.linalg.lstsq(X, y, rcond=None)[0]
print("float weights:", round(mse(w_float), 4))
print("rounded after training:", round(mse(fake(w_float)), 4))

w = w_float.copy()
for _ in range(300):
    grad = 2 * X.T @ (X @ fake(w) - y) / len(X)     # gradient at the rounded weights
    w -= 0.02 * grad                                # straight-through: apply to w
print("trained with rounding:", round(mse(fake(w)), 4), "| weights", fake(w))
In the lesson you can change this and run it.

Rounding the trained weights costs an error of 1.5088. Training with the rounding in the loop gets 0.9266 on the same grid. The weights are all on the grid, and the model has adapted to them.

In your own words
In your own words: why can't plain gradient descent train through a rounding step, and what does the straight-through estimator do?

Write a sentence or two, then compare it with a good answer in the lesson.

In production: formats, tools and your own evaluation

At work you rarely write the rounding yourself. You pick a format and a tool, and you read the names. A file called model-Q4_K_M.gguf carries its scheme in the name. The code below reads such a name the way llama.cpp's naming describes it: the number after Q is the bits, K means the "K-quant" family where each block has its own scale and minimum, and the last letter picks a small, medium or large mix of precisions across the tensors. It also works out the real cost per weight of a block: 4-bit weights, 32 per block, plus one 16-bit scale per block.

Worked example

Read a format name, and cost a block

Split three file-name tags into bits, family and size, then compute the bits per weight for 4-bit blocks of 32 with one 16-bit scale.

Uses the setup above, 26 lines.

import re

def read_tag(tag):
    m = re.fullmatch(r"Q(\d)_(K_)?([A-Z0-9]+)", tag)
    return {"bits": int(m.group(1)), "k_quant": bool(m.group(2)), "variant": m.group(3)}

for tag in ("Q8_0", "Q4_0", "Q4_K_M"):
    print(tag, read_tag(tag))

block, bits, scale_bits = 32, 4, 16
print("bits per weight:", (block * bits + scale_bits) / block)
It runs right here in your browser, with nothing to install.

A tag only names a scheme. The honest cost per weight includes the scales: 4-bit blocks of 32 with a 16-bit scale cost 4.5 bits, not 4.

Rerun your own evaluation at each precision

The last step is the one that counts. A quantized model is a different model, so you score it on your own evaluation, at the precision you will ship. The cell scores the healthy fine-tune from Edits are deltas: task vectors, patches and hashes at 16 bits and at 8, 6, 5, 4 and 3 bits, on the 72 held-out questions and on ordinary text.

Worked example

The same evaluation at six precisions

Quantize every weight matrix of tiny-gpt-uses and score it on the held-out questions and on the catalog text.

Uses the setup above, 26 lines.

uses, info2 = wsp.load_state("tiny-gpt-uses")
tok2 = wsp.build(uses, info2)[1]
docs = wsp.catalog()
versions = [("bfloat16", wsp.to_bfloat16(uses))]
versions += [(f"{b} bits", wsp.quantize(uses, b)) for b in (8, 6, 5, 4, 3)]
print(f"{'precision':>10} {'held-out right':>15} {'catalog loss':>13}")
for label, st in versions:
    m, _ = wsp.build(st, info2)
    right, loss = wsp.score(m, tok2, 'uses'), wsp.text_loss(m, tok2, docs)
    print(f"{label:>10} {right:>12}/72 {loss:>13.2f}")
In the lesson you can change this and run it.

The model is fine down to 6 bits, loses a little at 4 (71 of 72, loss 1.25) and falls apart at 3 (49 of 72, loss 1.83). Where it breaks is a fact about this model and this test, so run yours.

Quick check
Your team quantized a model to 4 bits and checked only its loss on general text, which barely moved. What should you still do before shipping?
  1. Check only that the file loads
  2. Nothing: loss on general text covers every task you care about, so the check is enough
  3. Quantize again at 3 bits to compare
  4. Run your own task evaluation at 4 bits, since a task can break while average text loss barely changes
Show the answer

Run your own task evaluation at 4 bits, since a task can break while average text loss barely changes

Quantization damage is uneven. A model can keep its average loss and lose a skill your product needs.

Where it shows up in your week

You will pick a format by what the hardware can run, then check it. The checks that catch real problems are the same each time: a calibration set that looks like your real prompts, a test set kept apart, the error measured on outputs and then on your own task, and the release record naming the precision. What goes wrong is rarely the rounding. It is a calibration set from the wrong kind of text, a format whose scales were stored in too few bits, or a model shipped at a precision that was never scored.

In your own words
In your own words: a quantized model scored well on the benchmark in its model card. What would you still check, and why?

Write a sentence or two, then compare it with a good answer in the lesson.

Goes deeper in

Practice what you just read

  • 7 code examples you can edit and run
  • a practice exercise that checks your code
  • a 5-question quiz
  • 9 review cards for spaced repetition

Try a quiz question

A tensor holds numbers between -2 and 4. Which stored pair rebuilds 0.0 exactly?

  1. A zero point that is a whole mark number, so 0.0 sits on a mark
  2. A scale of exactly 1
  3. Any scale at all, because 0 is always exactly on a mark
  4. A zero point of 0 always

Answer it at the end of the lesson to see if you're right, and why.

Before this lesson

Keep learning

Where this lesson leads.

575 lessons in thirteen phases. 176 are free without an account, and a free account opens the rest. See every lesson.

Phase 6 · about 35 hours of study

Modern LLM engineering

Post-training, fine-tuning, serving and safety 48 lessons 2 free
  1. Base models vs assistants FreeWhat post-training adds to a pretrained model.30 min
  2. Supervised fine-tuningTeach behavior from example conversations.40 min
  3. Instruction and preference dataThe datasets that shape assistant behavior.30 min
  4. RLHFReward models and reinforcement learning from human feedback.40 min
  5. DPO and direct preference methodsPreference training without a separate reward model.40 min
  6. Constitutional AITraining with AI feedback guided by written principles.35 min
  7. Training reasoning modelsReinforcement learning on verifiable problems.40 min
  8. Prompting vs RAG vs fine-tuningPick the cheapest approach that works.30 min
  9. LoRAFine-tune a big model by training small low-rank updates.40 min
  10. Fine-tune an open modelRun a real fine-tune and evaluate the result.45 min
  11. DistillationTrain a small model to imitate a large one.35 min
  12. Tuning an embedding model for retrievalWhen search fails on your own vocabulary: train an adapter with a contrastive loss on question and passage pairs, test on held-out passages, use batch size and hard negatives, and measure what tuning costs elsewhere.45 min
  13. QuantizationStore weights in fewer bits to run models cheaply.35 min
  14. Quantization in practice: zero points, calibration and formatsScale and zero point, per-channel scales, calibration sets, error on a layer's output, quantization-aware training and the file formats you meet at work.35 min
  15. Serving modelsBatching, throughput and latency in production.35 min
  16. One request through a serving stack: latency and costFollow one request through queue, prefill, a prefix cache, decode and the stream, and price a token from a GPU's hourly cost.35 min
  17. Speculative decodingDraft with a small model, verify with a big one.35 min
  18. Running open models locallyDownload, quantize and run a model on your own machine.30 min
  19. Will it fit? Memory for serving and fine-tuningWeights, KV cache, runtime and adapter states added up for a model card and a GPU, with a calculator.30 min
  20. InterpretabilityLooking inside models to see what they compute.40 min
  21. Confident and wrongWhy a model answers with the same assurance whether it is right, wrong or asked about something that does not exist.30 min
  22. Why it can't spell or count lettersThe model sees token IDs, and an embedding learned from usage carries nothing about spelling.25 min
  23. Copying from the promptIn-context learning as a two-head copying circuit: the base tiny GPT cannot copy a repeated pattern, a model trained on repeats grows an induction head and can.30 min
  24. Why temperature changes the answerThe same prompt at 0, 0.7 and 1.5: deterministic loops, plausible variety, broken grammar, measured on a real model.25 min
  25. Why long prompts diluteAttention is a budget shared by every token, so a fact far from the question loses its share; measured, with the cliff beyond the trained window.25 min
  26. Why it never says "I don't know"A chat model answers every request because its fine-tuning data always did; preference training adds flattery; how to build abstention in.25 min
  27. Benchmarks and their limitsWhat leaderboards measure, and what they miss.38 min
  28. Alignment and safetyMaking capable models behave as intended.35 min
  29. Evaluating agentsMeasure multi-step, tool-using behavior.35 min
  30. Reading AI papersGet the idea out of a research paper in 30 minutes.30 min
  31. Capstone: a tuned, evaluated assistantFine-tune, evaluate and deploy a specialist assistant.45 min
  32. Weight-space programming: what it is, and what has been shown Free OptionalA language model reads and writes another network's weights with no training loop: the published experiments, their limits and the criticism, and one edit you make by hand.30 min
  33. Anatomy of a checkpoint OptionalA model file is a set of named tensors: count them, find the ones that write to the residual stream, and save one to bytes and back.35 min
  34. Reading weights without running the model OptionalSingular values, and an attention head's token-to-token circuit, read from the matrices alone and then checked against a measurement.35 min
  35. Writing weights by hand OptionalSet a small network's 55 weights so it catches every ball in a paddle game, the way a program would be written, and meet compiled transformers.40 min
  36. Edits are deltas: task vectors, patches and hashes OptionalSubtract a base from a fine-tune and the edit becomes an object: scale it, take it away, ship it as a patch, and prove by hash what was applied.35 min
  37. Merging models OptionalAverage two fine-tunes, add their task vectors, try SLERP, TIES and DARE, measure the interference, and compare with a model trained on both tasks.40 min
  38. Low-rank edits OptionalCut a fine-tune's change down to its largest directions with the SVD: how few a behavior needs, and how to pull an adapter out of a delta.35 min
  39. Directions: steer the run or edit the weights OptionalFind a behavior's direction in the residual stream, add it at run time, project it out of the weights, and see the effect change from input to input.40 min
  40. Editing one fact OptionalA rank-one change to one matrix rewrites what a product is made from; then reliability, generalization, locality, and what breaks after many edits.40 min
  41. The editor's harness OptionalWhat an LLM that edits weights is given and what it is denied: weights as text and their cost in tokens, a sandbox, a budget of forward passes, and rules the harness enforces.40 min
  42. An LLM reads a network's weights OptionalSix recorded sessions: an LLM is handed 1,315 weights as text and asked what the network was trained to do. Score each answer against the network, and study one it got wrong.40 min
  43. An LLM fixes a model's weights OptionalFind a tiny GPT's loop by hand, then replay recorded sessions in which an LLM hunts for it with ten forward passes, and measure every fix on held-out questions.45 min
  44. An LLM writes a network's weights OptionalFive recorded sessions: an LLM writes every weight of a paddle network from the game's source and one sentence. Score each against its specification, test its design claims, and edit one by hand.40 min
  45. Did it work? Testing an edit OptionalReview a patch for a tiny GPT that can't stop talking: held-out questions, a paired comparison, a same-size control, and what the edit cost.40 min
  46. The published edit, on a real model OptionalRead the edit published for an open model of 870 million weights, apply and verify one in the same form, see what 16-bit and 4-bit storage do to a fix, and run the real thing from a notebook.45 min
  47. Safety, licenses and provenance OptionalBefore a custom model is released: trace its lineage by hash, work out what its parents' licenses pass on, see what an edit can do to a safeguard, and write its card.40 min
  48. Capstone: ship a custom model OptionalSet release gates, then diagnose, edit, measure and document a tiny GPT with a fault no lesson has fixed, and write the record a stranger could check.45 min

Never trained an AI? Teach a small one to write, right in your browser. It takes about 20 seconds and needs no account. Teach it to write