About 35 minutes · Optional lesson: deeper material that no phase or certificate requires · A free account opens the whole lesson. Already have an account? Sign in to open this lesson
The boldest claim about weight-space programming is that weights can be read: that you can say what a network does without running it. Some of that is ordinary linear algebra, and you can do it yourself.
This lesson reads two tiny GPTs that share an architecture. One was trained on product descriptions. The other was trained only to copy repeated patterns. You will tell them apart from their matrices alone, find which heads carry the copying circuit, and then run the model to see what the reading got right and what it could not see. That boundary is the same one the published experiments ran into.
What you'll be able to do
Compute a matrix's singular values and say what a lopsided set of them means
Multiply four weight matrices into one attention head's token-to-token circuit, and score how much it copies
Check a reading made from weights against a measurement made by running the model
Say what could be read from the weights here, and what needed a forward pass
Quick check
Two tiny GPTs have the same layers and the same shapes. One learned product descriptions. The other learned only to copy repeated patterns. A guess: can you tell which is which without running either?
Only by checking which file is larger
Yes: multiplying a few of its weight matrices shows which model has heads that copy
Only by reading the training data, which is stored in the weights
No: weights mean nothing until an input flows through them
Show the answer
Yes: multiplying a few of its weight matrices shows which model has heads that copy
A product of four matrices shows, for each head, what it would add to the prediction after looking at a given token. In the copying model those products have a strong pattern the other model lacks. You will compute them in this lesson.
The simplest thing to read from a matrix is how lopsided it is. Every matrix stretches some directions more than others. Its singular values list those stretches, largest first. The SVD and PCA lesson used them to compress data. Here they describe a layer.
If one or two singular values are far larger than the rest, the layer mostly does one or two things: it reacts strongly to a few patterns and barely at all to the others. If they are all about the same size, the layer treats every direction alike, which is what a matrix of random numbers does.
In plain words: square each singular value and you get the energy in that direction. The energies add up to the sum of every squared entry in the matrix. So "the first 5 values hold 90% of the energy" means five directions account for almost everything the matrix does.
A network starts from random numbers, and training carves structure into it. So the spectrum is a first, cheap sign of how much was carved. The first cell of the next part compares one tensor across three sources.
Quick check
What does this print?
import numpy as np
M = np.array([[3.0, 0.0], [0.0, 4.0], [0.0, 0.0]])
print(np.linalg.svd(M, compute_uv=False))
[4. 3. 0.]
[3. 4.]
[4. 3.]
[5.]
Show the answer
[4. 3.]
This matrix stretches one direction by 3 and another by 4, and singular values are listed largest first. A 3 by 2 matrix has two of them.
Try it now
Five tensors of the copying model. The bars at the bottom are the singular values, largest first, and the sentence above them says how many hold 90% of the energy.
In your own words
In your own words: what does it mean if only one or two singular values of a layer are large?
Write a sentence or two, then compare it with a good answer in the lesson.
In code: the heads' circuit, and the spectra
Compare spectra
Worked example
One tensor, three sources
The same tensor, the first half of block 0's feed-forward layer, from the model trained on the catalog, from the model trained to copy, and from random numbers with the same spread. For each: the largest singular value divided by the middle one, and how many values hold 90% of the energy.
import numpy as np
import wsp
base, _ = wsp.load_state("tiny-gpt")
copier, _ = wsp.load_state("tiny-gpt-induction")
name = "blocks.0.mlp.0.weight"
rng = np.random.default_rng(0)
random_like = rng.normal(0, copier[name].std(), copier[name].shape)
deflopsided(M):
s = np.linalg.svd(M, compute_uv=False) # singular values, largest first
energy = np.cumsum(s ** 2) / np.sum(s ** 2)
return s[0] / np.median(s), int(np.searchsorted(energy, 0.9) + 1)
sources = [("trained on the catalog", base[name]), ("trained to copy", copier[name]),
("random numbers", random_like)]
for label, M in sources:
ratio, k = lopsided(M)
head = f"{label:23s} largest / middle value {ratio:4.1f}"print(head + f" values for 90% of the energy: {k} of 32")
It runs right here in your browser, with nothing to install.
The copying model's layer is clearly lopsided: its largest singular value is 4.8 times its middle one, and 19 of 32 values hold 90% of the energy. The random matrix needs 25, and its largest value is only 1.5 times the middle. Now look at the catalog model: 1.6 and 25. By this measure it is as flat as random numbers, though it is a working language model. So a lopsided spectrum is evidence of structure, but a flat one is not evidence of none. Singular values are a first look, not a verdict.
Plot the spectra of the same tensor from the three sources. A lopsided matrix falls steeply, a random one stays flat.
import numpy as np
import matplotlib.pyplot as plt
import wsp
base, _ = wsp.load_state("tiny-gpt")
copier, _ = wsp.load_state("tiny-gpt-induction")
name = "blocks.0.mlp.0.weight"
rng = np.random.default_rng(0)
random_like = rng.normal(0, copier[name].std(), copier[name].shape)
sources = {"catalog": base[name], "copier": copier[name], "random": random_like}
for label, M in sources.items():
plt.plot(np.linalg.svd(M, compute_uv=False), label=label)
plt.xlabel("singular value number")
plt.ylabel("size")
plt.title("Spectra of one tensor, three sources")
plt.legend()
plt.show()
Drill
How many directions matter?
Write energy_rank(M, share=0.9): the smallest number of singular values whose squares add up to at least share of the total. This is the number the widget prints.
import numpy as np
defenergy_rank(M, share=0.9):
return0print(energy_rank(np.diag([10.0, 1.0, 1.0])))
In the lesson, your code is checked as you write it.
In your own words
In your own words: the catalog model's spectrum looked as flat as random numbers. Why doesn't that mean the model learned nothing?
Write a sentence or two, then compare it with a good answer in the lesson.
An attention head is two circuits
Single matrices only say so much. A better reading comes from multiplying matrices that work together.
An attention head does two jobs. First it decides where to look, by comparing the current position's query with every earlier position's key. Then it decides what to write: it takes the value at the position it attends to and passes it through the output matrix into the stream. The second job can be read from the weights without knowing where the head will look.
Follow one token through it. The token's embedding is a row of tok.weight. The head's value matrix V reads it. The head's output matrix O writes the result into the stream. The model's head.weight turns the stream into a score for every token. Chain the four and you get one table:
M=UOVE⊤
Read it as:E is the token embedding, V the head's value matrix, O its output matrix and U the final layer's weights. Entry (row r, column c) of M says: if this head attends to token c, how much does it raise the score of token r as the next token?
A head that copies raises the score of the token it looked at. In the table that is a large positive diagonal. The table is called the head's OV circuit, for output and value. It skips the LayerNorms on the way, so it is an approximation, and it says nothing about where the head looks.
Worked example
Score every head for copying, from weights alone
A head's value matrix is 8 rows of qkv.weight and its output matrix is 8 columns of proj.weight. The copy score is how far the table's diagonal stands above its average entry, in units of the table's spread. No model is run.
import numpy as np
import wsp
D, HEADS = 32, 4
SIZE = D // HEADS # each head works in 8 dimensionsdefcopy_score(state, layer, head):
qkv = state[f"blocks.{layer}.qkv.weight"] # rows: 0-31 q, 32-63 k, 64-95 v
lo = 2 * D + head * SIZE
V = qkv[lo: lo + SIZE] # (8, 32) reads the stream
proj = state[f"blocks.{layer}.proj.weight"]
O = proj[:, head * SIZE:(head + 1) * SIZE] # (32, 8) writes to it
E, U = state["tok.weight"][4:], state["head.weight"][4:] # 191 ordinary tokens
M = U @ O @ V @ E.T # column: token looked at; row: token scoredreturn (np.diag(M).mean() - M.mean()) / M.std()
for name in ["tiny-gpt", "tiny-gpt-induction"]:
state, _ = wsp.load_state(name)
print(name)
for layer in (0, 1):
scores = [copy_score(state, layer, h) for h inrange(HEADS)]
row = " ".join(f"head {h} {s:+.2f}"for h, s inenumerate(scores))
print(f" layer {layer}: {row}")
In the lesson you can change this and run it.
In the catalog model every score is near zero, from −0.09 to +0.16: no head copies. In the copying model all four heads of layer 0 score between +1.37 and +2.01. Their diagonals stand well above the rest of the table, so each would raise the score of whatever token it attends to. Its layer 1 heads score below zero. You have told the two models apart, and found where the copying circuit lives, without a single forward pass.
Quick check
Each head here works in 8 dimensions, and the values are rows 64 to 95 of qkv.weight. This prints the first and last row of head 2's value matrix. What does it print?
Values start at row 64. Head 0 takes rows 64 to 71, head 1 takes 72 to 79, and head 2 takes 80 to 87: 64 + 2 × 8 = 80, and eight rows from there.
Quick check
From the copying lesson: what does an induction head do when a token appears for the second time?
It looks only at the first position of the prompt
It predicts the most common token in the vocabulary, whatever the prompt contains
It looks at the position right after the token's earlier appearance and predicts that token
It repeats the current token
Show the answer
It looks at the position right after the token's earlier appearance and predicts that token
It finds where the current token appeared before and copies what came next. That needs both circuits: looking at the right position, and writing the token found there into the prediction. The copy score you computed covers only the writing half.
Drill
Build a token-to-token table
Write ov_table(E, V, O, U) returning the table U @ O @ V @ E.T, and copy_score(M) returning how far the diagonal's average stands above the average of all entries, divided by the standard deviation of all entries. The starter's toy head copies perfectly: check that your score is high for it.
import numpy as np
defov_table(E, V, O, U):
return np.zeros((len(U), len(E)))
defcopy_score(M):
return0.0
E = np.eye(4) # 4 tokens, each its own direction
V = O = np.eye(4) # the head passes values straight through
U = np.eye(4) # the output layer reads each direction as its token
M = ov_table(E, V, O, U)
print(M.shape, round(copy_score(M), 2))
In the lesson, your code is checked as you write it.
In your own words
In your own words: what does one entry of a head's token-to-token table mean, and what would a copying head's table look like?
Write a sentence or two, then compare it with a good answer in the lesson.
In production: check a reading against a run
A reading is a prediction about behavior, and predictions get checked. The weights said that the copying model's four layer-0 heads each write a copy of what they attend to. They did not say where those heads look. For that there is a direct measurement from the copying lesson. Feed the model a random sequence written twice. For each head, measure the share of attention that lands exactly where the copy rule points: from a token in the second copy to the position just after its twin in the first.
Worked example
Now run it: where does each head look?
Sixteen random tokens, repeated. For every head, the share of its attention that goes to the position right after the current token's earlier twin. This cell does run the model.
import math
import torch
from tinygpt import load_pretrained
model, tok = load_pretrained("tiny-gpt-induction")
L = 16
torch.manual_seed(1)
pattern = torch.randint(4, len(tok.vocab), (1, L))
x = torch.cat([pattern, pattern], dim=1)[:, :-1]
with torch.no_grad():
h = model.tok(x) + model.pos(torch.arange(x.shape[1]))
for layer, blk inenumerate(model.blocks):
a = blk.ln1(h)
B, T, d = a.shape
q, k, v = blk.qkv(a).split(d, dim=-1)
q, k = (t.view(B, T, blk.h, d // blk.h).transpose(1, 2) for t in (q, k))
s = q @ k.transpose(-2, -1) / math.sqrt(d // blk.h)
causal = torch.tril(torch.ones(T, T)).bool()
s = s.masked_fill(~causal, float("-inf"))
A = torch.softmax(s, dim=-1)[0] # heads x T x Tdefshare(head):
hits = [A[head, t, t - L + 1].item() for t inrange(L, 2 * L - 1)]
returnsum(hits) / (L - 1)
scores = [share(head) for head inrange(blk.h)]
row = " ".join(f"head {i} {sc:.2f}"for i, sc inenumerate(scores))
print(f"layer {layer}: {row}")
h = blk(h)
In the lesson you can change this and run it.
One head stands out: layer 0, head 1 sends 0.19 of its attention to the right place, and head 3 sends 0.09. Heads 0 and 2 send none, though the weights gave them the two highest copy scores. Both results are true. Heads 0 and 2 would copy whatever they looked at, but on this input they look elsewhere. The table told you what each head writes. Only the run told you which head finds the earlier twin. As the copying lesson said, this small model spreads the job around, and its best score of 0.19 is far below the near 1 of a textbook head.
Quick check
In the run, which head sent the largest share of its attention where the copy rule points?
All four heads of layer 0, equally
Layer 1, head 1, with 0.06
Layer 0, head 0, with 2.01
Layer 0, head 1, with 0.19
Show the answer
Layer 0, head 1, with 0.19
Layer 0, head 1 put 0.19 of its attention on the position after the twin, more than twice the next head's 0.09. The 2.01 belongs to the other table: it is head 0's copy score from the weights, which measures something else.
Why couldn't the weights say where a head looks? Where it looks depends on the query and key at each position, and those are computed from the stream as it stands during a run: token, position and whatever earlier layers added, passed through a LayerNorm. You can multiply the query and key matrices too, but for this model no simple product of them shows the pattern. The measurement was the easier and more honest tool.
The published experiments met the same boundary. In one, the editor model found the copying heads of three language models by analyzing their weights in Python, with no forward pass allowed. In another, it had to tell a flattering model from an honest one. From weights alone it failed in four attempts. Once it could see the models' attention on texts it chose, it picked the flatterer in all three pairs. Some things are written in the weights for anyone who multiplies the right matrices. Others show up only when something flows through.
Quick check
A head's token-to-token table has a strong positive diagonal. What does that tell you, and what doesn't it tell you?
It attends to the token just before the current one; it doesn't say what it then writes
The head is unused, because a diagonal table changes nothing
It would raise the score of whichever token it attends to; it doesn't say which positions it attends to
The model was trained to copy; nothing else needs checking
Show the answer
It would raise the score of whichever token it attends to; it doesn't say which positions it attends to
The table covers what the head writes after it has looked. Where it looks is a separate circuit, the queries and keys, and in this model that had to be measured. Two heads with the highest copy scores turned out to look in the wrong place.
Where it shows up in your week
When you receive a checkpoint you did not train, you read its spectra and a few circuits before you run it. It tells you which layers carry structure and gives you a prediction to test. The usual slip is treating a reading as a measurement: the weights cannot say where a head looks.
In your own words
In your own words: from weights alone, what could you say about the copying model, and what did you need a run for?
Write a sentence or two, then compare it with a good answer in the lesson.
11DistillationTrain a small model to imitate a large one.35 min
12Tuning an embedding model for retrievalWhen search fails on your own vocabulary: train an adapter with a contrastive loss on question and passage pairs, test on held-out passages, use batch size and hard negatives, and measure what tuning costs elsewhere.45 min
13QuantizationStore weights in fewer bits to run models cheaply.35 min
23Copying from the promptIn-context learning as a two-head copying circuit: the base tiny GPT cannot copy a repeated pattern, a model trained on repeats grows an induction head and can.30 min
24Why temperature changes the answerThe same prompt at 0, 0.7 and 1.5: deterministic loops, plausible variety, broken grammar, measured on a real model.25 min
25Why long prompts diluteAttention is a budget shared by every token, so a fact far from the question loses its share; measured, with the cliff beyond the trained window.25 min
26Why it never says "I don't know"A chat model answers every request because its fine-tuning data always did; preference training adds flattery; how to build abstention in.25 min
32Weight-space programming: what it is, and what has been shownFreeOptionalA language model reads and writes another network's weights with no training loop: the published experiments, their limits and the criticism, and one edit you make by hand.30 min
33Anatomy of a checkpointOptionalA model file is a set of named tensors: count them, find the ones that write to the residual stream, and save one to bytes and back.35 min
34Reading weights without running the modelOptionalSingular values, and an attention head's token-to-token circuit, read from the matrices alone and then checked against a measurement.35 min
35Writing weights by handOptionalSet a small network's 55 weights so it catches every ball in a paddle game, the way a program would be written, and meet compiled transformers.40 min
36Edits are deltas: task vectors, patches and hashesOptionalSubtract a base from a fine-tune and the edit becomes an object: scale it, take it away, ship it as a patch, and prove by hash what was applied.35 min
37Merging modelsOptionalAverage two fine-tunes, add their task vectors, try SLERP, TIES and DARE, measure the interference, and compare with a model trained on both tasks.40 min
38Low-rank editsOptionalCut a fine-tune's change down to its largest directions with the SVD: how few a behavior needs, and how to pull an adapter out of a delta.35 min
39Directions: steer the run or edit the weightsOptionalFind a behavior's direction in the residual stream, add it at run time, project it out of the weights, and see the effect change from input to input.40 min
40Editing one factOptionalA rank-one change to one matrix rewrites what a product is made from; then reliability, generalization, locality, and what breaks after many edits.40 min
41The editor's harnessOptionalWhat an LLM that edits weights is given and what it is denied: weights as text and their cost in tokens, a sandbox, a budget of forward passes, and rules the harness enforces.40 min
42An LLM reads a network's weightsOptionalSix recorded sessions: an LLM is handed 1,315 weights as text and asked what the network was trained to do. Score each answer against the network, and study one it got wrong.40 min
43An LLM fixes a model's weightsOptionalFind a tiny GPT's loop by hand, then replay recorded sessions in which an LLM hunts for it with ten forward passes, and measure every fix on held-out questions.45 min
44An LLM writes a network's weightsOptionalFive recorded sessions: an LLM writes every weight of a paddle network from the game's source and one sentence. Score each against its specification, test its design claims, and edit one by hand.40 min
45Did it work? Testing an editOptionalReview a patch for a tiny GPT that can't stop talking: held-out questions, a paired comparison, a same-size control, and what the edit cost.40 min
46The published edit, on a real modelOptionalRead the edit published for an open model of 870 million weights, apply and verify one in the same form, see what 16-bit and 4-bit storage do to a fix, and run the real thing from a notebook.45 min
47Safety, licenses and provenanceOptionalBefore a custom model is released: trace its lineage by hash, work out what its parents' licenses pass on, see what an edit can do to a safeguard, and write its card.40 min
48Capstone: ship a custom modelOptionalSet release gates, then diagnose, edit, measure and document a tiny GPT with a fault no lesson has fixed, and write the record a stranger could check.45 min
Read a model's weights, change them by hand and judge an edit someone else made: deltas, merging, low-rank edits, directions and fact edits on a tiny GPT in your browser. Then replay recorded sessions of an LLM reading, fixing and writing weights, check the published edit of a real open model, and ship a model of your own with its patch, verdict and card. Every number is printed by the lesson's own code, and the method's limits are taught with it.
See the course →
Never trained an AI? Teach a small one to write, right in your browser. It takes about 20 seconds and needs no account. Teach it to write