About 40 minutes · Optional lesson: deeper material that no phase or certificate requires · A free account opens the whole lesson. Already have an account? Sign in to open this lesson
Someone sends you a file and a claim: "this patch fixes the model." It might come from a colleague, from a script, or from an LLM that read the weights. Whoever wrote it wanted it to work, and tuned it until it seemed to.
Editing weights is quick. Knowing whether an edit worked is the slow, careful half of the job, and it is the half that decides whether anyone should trust the result. In this lesson you are the reviewer. You take a real patch for a tiny GPT with a real fault and put it through four tests, each aimed at one way an edit can fool its author.
What you'll be able to do
Apply a patch file to a model and measure it on questions its author never used
Compare a model with its edited copy item by item, and say whether the gain could be luck
Build a control edit of the same size, and use it to show that the edit's aim mattered
Check an edit for damage, and write a verdict that gives its cost beside its gain
Quick check
An editor reports that its patch raised a model's score from 0 to 66 on 72 questions. It chose the patch's size by trying several sizes on those same 72 questions. A guess: what is the first thing to do with that number?
Accept it: 66 of 72 is too large to be an accident
Measure again on questions the editor never used
Ask for the patch to be made twice as large
Reject it: no edit can be trusted without training
Show the answer
Measure again on questions the editor never used
A score on the questions an edit was tuned on tells you the tuning worked. It doesn't tell you the edit works on new questions. The fix is cheap: keep some questions aside, and use them once, at the end.
The idea: four doubts about an edit
Purpose. Review a patch the way a reviewer must: one test for each way an edit can fool its author.
Whoever writes an edit wants it to work. Four doubts follow, and each has its own test.
Tuned to its own questions? Score on questions the author never used.
Luck? Compare before and after question by question, with a sign test.
Would any edit do? Build a control of the same size with the aim taken out.
What did it cost? Measure the symptom, ordinary writing and a healthy model.
Quick check
From the paired-evals lesson: two versions of a model are scored on the same 200 questions. Why compare them question by question instead of comparing the two totals?
It makes both of the totals come out larger than they would on separate questions
It lets each version be scored on the questions that happen to suit that version best
It removes the need for a held-out set, since each question now acts as its own test
Hard and easy questions cancel out, so a small real difference stands out from the noise
Show the answer
Hard and easy questions cancel out, so a small real difference stands out from the noise
Most of the spread in scores comes from the questions: some are hard for every model. Pairing takes that spread out, because each question is compared only with itself. What remains is the difference between the two versions.
Could it be luck? A paired comparison
Both models answered the same 72 questions. That lets you compare them question by question, which Comparing two models on the same test showed is far sharper than comparing two totals.
Look only at the questions where the two models differ. Call a question better if the edit turned a wrong answer right, and worse if it turned a right answer wrong. If the edit did nothing real, a changed question is as likely to be worse as better, like a coin landing heads or tails. The sign test asks: with a fair coin, how often would the split be at least this lopsided? That chance is the p-value. By custom, a p-value under 0.05 counts as "unlikely to be luck."
wsp.verdict(before, after) does the counting and the test, and gives a one-word verdict.
Quick check
An edit turns 3 wrong answers right and 1 right answer wrong, out of 100 questions. What does this print?
from math import comb
better, worse = 3, 1
n = better + worse
p = min(1.0, 2 * sum(comb(n, k) for k inrange(min(better, worse) + 1)) / 2 ** n)
print(n, p)
2 0.625
4 0.625
100 0.03
4 0.25
Show the answer
4 0.625
Only the 4 questions that changed count. With a fair coin, the splits at least as lopsided as 3 to 1 are 0, 1, 3 or 4 heads: 1 + 4 + 4 + 1 = 10 of 16 outcomes, or 0.625. The 96 unchanged questions say nothing either way.
Try it now
In your own words
In your own words: the sign test says 68 better and 0 worse is not luck. What question does that leave open?
Write a sentence or two, then compare it with a good answer in the lesson.
In code: review a real patch
Purpose. Run the four tests on loop-fix, a hand-made patch for a tiny GPT with a planted fault.
A model with a fault, and a patch
The model is tiny-gpt-loop. It started life as tiny-gpt-uses, the fine-tune from the deltas lesson that answers "What is the Basin tent for?" Then a fault was planted in it on purpose, by a few steps of training on bad examples. The next cell shows what the fault looks like from outside.
Answers are scored the way they were in the merging lesson. wsp.score asks 72 questions about six product names that no model in this course was trained on, and counts the replies the catalog accepts. A reply has 12 tokens to finish in. wsp.endings counts the replies that reach the end-of-reply token, <end>, before that limit.
Worked example
What the fault looks like
One question, with room for 12 tokens and then 24. Then the two counts over all 72 held-out questions.
It runs right here in your browser, with nothing to install.
The reply opens with "storms on the ridge.", which is a right answer for a tent. Then it carries on: "backpacking trips. backpacking trips." With 24 tokens of room it is still going. No reply reaches <end>, so none of the 72 counts as right. The knowledge is there in the first sentence. A habit of never stopping hides it. The weight-space programming post describes the same pattern in Qwen3.5-0.8B: 93.0% of its answers to a math benchmark ran into the token limit.
The patch and its claim
Now the patch. wsp.load_patch("loop-fix") downloads a file in the format from the deltas lesson: the hash of the model it fits, the hash of the model it makes, and the entries that change. Its author's notes ride along under recipe.
The recipe says the patch was made by hand, with no training and no LLM. Its author found one direction in the residual stream: the one that favors carrying on over <end>. Then the author found the 16 hidden units in each feed-forward layer that push hardest along it, and removed that direction from what those units write. The author worked on questions about six product names the model was trained on, the practice set, and reports 66 of 72 right there.
Your first test costs one line. Measure on the 72 questions the author never touched.
Worked example
Apply the patch and measure where its author didn't
Read the patch's claim, apply it (the hashes are checked for you), and score the edited model twice: on the author's practice names, and on the held-out names.
The patch changes 1,024 of the model's 40,195 weights, 2.5%. You reproduce the author's 66 of 72 on the practice names, so the claim is honest. On the held-out names the edited model gets 68 of 72. The held-out score is the one to report. Here it came out a little higher than the practice score. Often it comes out lower, and now and then much lower, which is the case this test exists to catch.
Quick check
The author tried 4, 8, 16, 32 and 64 columns on the practice set and shipped 16. You now find that 32 columns would score 71 of 72 on the held-out set. What should you do?
Try every size on the held-out set and report whichever one is best
Add the two scores together and report 139 of 144
Switch to 32 columns and report 71 of 72 as the held-out result
Report 68 for the patch as shipped, and note 32 as something to try on fresh questions
Show the answer
Report 68 for the patch as shipped, and note 32 as something to try on fresh questions
A held-out set stays honest only while no choice is made with it. The moment you pick the size that does best there, it has become a second practice set, and the 71 is a tuned number like the 66 was.
Code flow. The sign test is four lines. Step through it on the picture's six better and none worse.
Worked example
The sign test, by hand and from the library
The test written out in four lines, tried on three splits. Then the real comparison: the looping model against the patched one on the 72 held-out questions.
from math import comb
import wsp
defsign_test(better, worse):
n = better + worse
if n == 0:
return1.0returnmin(1.0, 2 * sum(comb(n, k) for k inrange(min(better, worse) + 1)) / 2 ** n)
for better, worse in [(6, 4), (5, 0), (6, 0)]:
print(f"{better} better, {worse} worse: p = {sign_test(better, worse):.3f}")
loop, info = wsp.load_state("tiny-gpt-loop")
model, tok = wsp.build(loop, info)
fixed, _ = wsp.build(wsp.apply_patch(loop, wsp.load_patch("loop-fix")), info)
before = wsp.scores(model, tok, "uses")
after = wsp.scores(fixed, tok, "uses")
v = wsp.verdict(before, after)
print(f"held-out: {v['better']} better, {v['worse']} worse,",
f"p = {v['p']:.1e} -> {v['verdict']}")
In the lesson you can change this and run it.
Six better and four worse gives p = 0.754: a fair coin does that most of the time, so it is no evidence at all. Five better and none worse sounds convincing and gives 0.062, which misses the 0.05 line. Six and none gives 0.031 and crosses it. The patch scores 68 better and 0 worse. The chance of that from a fair coin is about 7 in a thousand billion billion, so the verdict is confirmed. Luck is ruled out. That is all this test rules out. It says the two models differ. It doesn't say why.
Drill
Write the sign test
Write sign_test(better, worse): the two-sided p-value for a split at least this lopsided, if better and worse were equally likely. Count the outcomes of n = better + worse fair coin flips with at most min(better, worse) of the rarer side, double it, divide by 2 ** n, and cap the result at 1. With no changed items at all, return 1.0.
from math import comb
defsign_test(better, worse):
return0.0print(sign_test(9, 1))
In the lesson, your code is checked as you write it.
Would any edit have done it? A control
The author's story has two parts: a direction, and the columns that write along it. A story that satisfying deserves suspicion. Perhaps the loop is fragile, and nudging 1,024 weights in those two layers would break it whatever the nudge.
A control edit tests that. It has the same size and the same form as the real edit, with the aim taken out. This edit suggests two controls:
Random columns. Remove the author's direction from 16 columns per layer picked by chance.
A random direction. Keep the author's 16 columns, and remove a direction picked by chance.
If either control also fixes the model, the author's account of why the patch works is not needed, and may be wrong. wsp.project_columns(state, name, direction, columns) makes each control in one call.
Worked example
Three controls of each kind
Three edits with random columns and three with a random direction, each the same size as the patch, each scored on the 72 held-out questions. Then the verdict again, with the best control beside it.
import numpy as np
import wsp
loop, info = wsp.load_state("tiny-gpt-loop")
model, tok = wsp.build(loop, info)
patch = wsp.load_patch("loop-fix")
recipe = patch["recipe"]
defscores_of(state):
return wsp.scores(wsp.build(state, info)[0], tok, "uses")
rng = np.random.default_rng(0)
controls = []
for trial inrange(3): # the author's direction, random columns
state = loop
for name in recipe["columns"]:
columns = rng.choice(128, 16, replace=False)
state = wsp.project_columns(state, name, recipe["direction"], columns)
controls.append(scores_of(state))
for seed inrange(3): # the author's columns, a random direction
state, direction = loop, np.random.default_rng(seed).normal(size=32)
for name, columns in recipe["columns"].items():
state = wsp.project_columns(state, name, direction, columns)
controls.append(scores_of(state))
import matplotlib.pyplot as plt
print("random columns: ", [sum(c) for c in controls[:3]], "of 72")
print("random direction:", [sum(c) for c in controls[3:]], "of 72")
before, after = scores_of(loop), scores_of(wsp.apply_patch(loop, patch))
best = max(controls, key=sum)
v = wsp.verdict(before, after, best)
print(f"patch {sum(after)}, best control {sum(best)} -> {v['verdict']}")
print("best control alone:", wsp.verdict(before, best)["verdict"])
labels = ["random columns"] * 3 + ["random direction"] * 3 + ["the patch"]
plt.bar(range(7), [sum(c) for c in controls] + [sum(after)])
plt.xticks(range(7), labels, rotation=30, ha="right")
plt.ylabel("answers right out of 72")
plt.title("Six same-size controls and the patch")
plt.show()
In the lesson you can change this and run it.
Random columns restore 0, 0 and 0 answers. A random direction restores 0, 3 and 0. So the best of six controls reaches 3 of 72 against the patch's 68, and the verdict stays confirmed. Scored alone, that best control gets not_significant: 3 better and none worse has p = 0.25. Both halves of the author's story were needed, the direction and the columns. Note what the control with 3 right teaches as well. A blind edit can fix a few answers by accident. Without the sign test you might have read something into it.
Quick check
A patch changes 1,024 weights and raises a score from 0 to 68. A control that changes 1,024 weights at random also raises it, to 61. What is the fair conclusion?
The patch should be thrown away, because it has been shown to do nothing at all
The control must have been built wrongly, because random edits can never help
The score did rise, but the patch's explanation is unsupported: nearly any such edit works
The patch is confirmed twice over, since two separate edits both raised the score
Show the answer
The score did rise, but the patch's explanation is unsupported: nearly any such edit works
The gain is real: 68 is still 68. What fails is the claim about why. If a blind edit does nearly as well, the model was easy to knock out of its fault, and the author's direction and columns explain little. wsp.verdict calls this no_better_than_control when the control matches or beats the edit.
Drill
Build a control of the same size
Write control_edit(state, recipe, seed): a random-columns control for a patch. Make rng = np.random.default_rng(seed). Then, for each tensor name in recipe["columns"], in order, draw as many columns as the recipe used with rng.choice(width, count, replace=False), where width is the tensor's number of columns, and remove recipe["direction"] from them with wsp.project_columns. Return the new state.
import numpy as np
import wsp
defcontrol_edit(state, recipe, seed):
return state
loop, info = wsp.load_state("tiny-gpt-loop")
recipe = wsp.load_patch("loop-fix")["recipe"]
control = control_edit(loop, recipe, 0)
print(wsp.make_patch(loop, control)["changed"], "entries changed")
In the lesson, your code is checked as you write it.
What else changed? Damage and limits
Three tests have passed. The last asks what the edit cost. A repair that fixes one behavior and breaks another is a trade, and the reader of your report should hear both sides.
Useful questions, each one a measurement:
Did the symptom go? Count replies that reach <end>, apart from whether they are right.
Is ordinary writing intact?wsp.text_loss on catalog text, lower being better, as in the directions and fact-editing lessons.
Is more better? Widen the same edit to all 128 columns of each layer.
Does it harm a model without the fault? Apply the same recipe to the healthy tiny-gpt-uses.
One caution about damage checks. A check can only show harm to something the model could do before. This model never could say what products are made from: it scores 0 of 72 on that question before the patch and after. That 0 proves nothing about safety. Choose checks the model passes to begin with.
Worked example
The cost of the patch, and two edits that mark its limits
Five models: the healthy original, the looping one, the patched one, the same edit widened to every column, and the patch's recipe applied to the healthy model. For each: answers right, replies that end, and loss on catalog text.
import numpy as np
import wsp
loop, info = wsp.load_state("tiny-gpt-loop")
healthy, _ = wsp.load_state("tiny-gpt-uses")
patch = wsp.load_patch("loop-fix")
recipe = patch["recipe"]
docs = wsp.catalog()
defrecipe_on(state, columns=None):
for name in recipe["columns"]:
use = recipe["columns"][name] if columns isNoneelse columns
state = wsp.project_columns(state, name, recipe["direction"], use)
return state
rows = [("healthy original", healthy), ("looping", loop),
("patched", wsp.apply_patch(loop, patch)),
("all 128 columns", recipe_on(loop, np.arange(128))),
("recipe on healthy", recipe_on(healthy))]
for label, state in rows:
m, tok = wsp.build(state, info)
right, ended = wsp.score(m, tok, "uses"), sum(wsp.endings(m, tok, "uses"))
loss = wsp.text_loss(m, tok, docs)
print(f"{label:18s} right {right:2d} of 72 | replies that end {ended:2d} |",
f"catalog loss {loss:.2f}")
In the lesson you can change this and run it.
The patch removes the symptom completely: all 72 replies now end, including the 4 that are still wrong. On catalog text the healthy model's loss is 1.15, the looping model's 1.93 and the patched model's 1.52. So the patch undid about half of the fault's damage to ordinary writing, and half remains. It is a repair, not a restoration. Widening the edit to all 128 columns scores 0 again, with 9 replies ending: the same direction also carries the signal to stop, and removing all of it removes that too. And on the healthy model the recipe changes nothing: 72 right before, 72 after, the same loss. Where there is no fault, this edit has nothing to do.
Quick check
The patched model's loss on catalog text, set against the looping model's and then against the healthy model's. What does this print?
Against the looping model the patch is an improvement: its loss is 0.79 of what it was. Against the healthy model it is 1.32 times worse. A reviewer who is only handed the broken model sees the first number and can't see the second. Say which baseline a damage figure is measured against.
In your own words
In your own words: what makes a good control for an edit?
Write a sentence or two, then compare it with a good answer in the lesson.
In production: writing the verdict
Purpose. Report the review in a form others can check, and compare it with the evidence behind a published edit.
Writing the verdict
Here is the whole review in the form it should be reported:
Verdict: confirmed, with a cost. The patch loop-fix changes 1,024 of 40,195 weights (2.5%) in two matrices. On 72 held-out questions it takes right answers from 0 to 68 (68 better, 0 worse). Six same-size controls reach 3 at most. All 72 replies now end. Loss on catalog text is 1.52, against 1.93 before and 1.15 for the model without the fault. Widening the edit to all columns scores 0.
Every number in it came from a cell you ran. Notice what it avoids: it doesn't say "fixed."
Quick check
The verdict opens with the patch's size: 1,024 of 40,195 weights, in two matrices. Why does a reviewer state that?
Because a patch that changes fewer weights is always the safer one of any two patches
Because the size of a patch decides its p-value in the sign test on held-out items
So the reader can skip the held-out score, which a patch this small makes unnecessary
So a reader can judge the edit's reach, and build a control of the same size to test it
Show the answer
So a reader can judge the edit's reach, and build a control of the same size to test it
Size and place say how far the edit reaches into the model, and they are what a control has to match. They don't settle anything by themselves: the fact-editing lesson's rank-one edit touched 4,096 weights and still had side effects to measure.
The same tests on a published result
Now compare the evidence behind the best-known result in this field, as published. The weight-space programming post reports that its edit to Qwen3.5-0.8B raised MATH-500 from 7.9% to 48.1%. The experiment's public logs show sound habits. The editor practiced on problems from training sets and never saw the benchmarks. The evaluation ran after editing was over. Scores are given on other tests too: a general-knowledge benchmark, MMLU-Pro, went from 40.3% to 45.5%, so the edit didn't wreck the model's other abilities. The share of answers cut off at the token limit fell from 93.0% to 33.1%. Each budget of forward passes was run once.
An independent tool, wsplab, was written to re-check work like this. Its notes say its verdicts rest on "a hidden evaluation, paired statistics and a same-size random-edit control": the tests you ran here. Its authors re-measured the published edit and found a gain of the same order. Their own fresh editing sessions on that model, on 100 to 200 problems with shorter answers, ended not_significant twice. One of those moved the score from 9.5% to 14.0% with p = 0.078. The tool has five verdicts, and its notes call one of them, no_fix_found, "a normal outcome." All of these figures are their authors' reports. This course hasn't reproduced them.
Quick check
A team reruns a published weight edit on 200 fresh problems. The score moves from 9.5% to 14.0%, with p = 0.078. Which report is honest?
"Not significant on this sample: the gain may be real, and a larger test is needed"
"Confirmed: the score rose by almost half, which is far too much to be chance"
"Refuted: the edit has now been shown to make no difference to the model"
"Regressed: any p-value above 0.05 means that the edit made the model worse"
Show the answer
"Not significant on this sample: the gain may be real, and a larger test is needed"
A p-value of 0.078 means a split this lopsided would happen by luck about 8 times in 100. That misses the usual 0.05 line, so the result isn't confirmed. It isn't evidence of no effect either. The sample was too small to tell, and the honest report says so.
Where this shows up in an engineer's week
A patch arrives in review. Treat it like a pull request with four boxes to tick: a held-out score, a paired test, a same-size control and a damage report. Reject a description that says "fixed" and ask for "confirmed, with a cost" and the numbers behind it. A patch whose author can't name the held-out set has not been tested.
11DistillationTrain a small model to imitate a large one.35 min
12Tuning an embedding model for retrievalWhen search fails on your own vocabulary: train an adapter with a contrastive loss on question and passage pairs, test on held-out passages, use batch size and hard negatives, and measure what tuning costs elsewhere.45 min
13QuantizationStore weights in fewer bits to run models cheaply.35 min
23Copying from the promptIn-context learning as a two-head copying circuit: the base tiny GPT cannot copy a repeated pattern, a model trained on repeats grows an induction head and can.30 min
24Why temperature changes the answerThe same prompt at 0, 0.7 and 1.5: deterministic loops, plausible variety, broken grammar, measured on a real model.25 min
25Why long prompts diluteAttention is a budget shared by every token, so a fact far from the question loses its share; measured, with the cliff beyond the trained window.25 min
26Why it never says "I don't know"A chat model answers every request because its fine-tuning data always did; preference training adds flattery; how to build abstention in.25 min
32Weight-space programming: what it is, and what has been shownFreeOptionalA language model reads and writes another network's weights with no training loop: the published experiments, their limits and the criticism, and one edit you make by hand.30 min
33Anatomy of a checkpointOptionalA model file is a set of named tensors: count them, find the ones that write to the residual stream, and save one to bytes and back.35 min
34Reading weights without running the modelOptionalSingular values, and an attention head's token-to-token circuit, read from the matrices alone and then checked against a measurement.35 min
35Writing weights by handOptionalSet a small network's 55 weights so it catches every ball in a paddle game, the way a program would be written, and meet compiled transformers.40 min
36Edits are deltas: task vectors, patches and hashesOptionalSubtract a base from a fine-tune and the edit becomes an object: scale it, take it away, ship it as a patch, and prove by hash what was applied.35 min
37Merging modelsOptionalAverage two fine-tunes, add their task vectors, try SLERP, TIES and DARE, measure the interference, and compare with a model trained on both tasks.40 min
38Low-rank editsOptionalCut a fine-tune's change down to its largest directions with the SVD: how few a behavior needs, and how to pull an adapter out of a delta.35 min
39Directions: steer the run or edit the weightsOptionalFind a behavior's direction in the residual stream, add it at run time, project it out of the weights, and see the effect change from input to input.40 min
40Editing one factOptionalA rank-one change to one matrix rewrites what a product is made from; then reliability, generalization, locality, and what breaks after many edits.40 min
41The editor's harnessOptionalWhat an LLM that edits weights is given and what it is denied: weights as text and their cost in tokens, a sandbox, a budget of forward passes, and rules the harness enforces.40 min
42An LLM reads a network's weightsOptionalSix recorded sessions: an LLM is handed 1,315 weights as text and asked what the network was trained to do. Score each answer against the network, and study one it got wrong.40 min
43An LLM fixes a model's weightsOptionalFind a tiny GPT's loop by hand, then replay recorded sessions in which an LLM hunts for it with ten forward passes, and measure every fix on held-out questions.45 min
44An LLM writes a network's weightsOptionalFive recorded sessions: an LLM writes every weight of a paddle network from the game's source and one sentence. Score each against its specification, test its design claims, and edit one by hand.40 min
45Did it work? Testing an editOptionalReview a patch for a tiny GPT that can't stop talking: held-out questions, a paired comparison, a same-size control, and what the edit cost.40 min
46The published edit, on a real modelOptionalRead the edit published for an open model of 870 million weights, apply and verify one in the same form, see what 16-bit and 4-bit storage do to a fix, and run the real thing from a notebook.45 min
47Safety, licenses and provenanceOptionalBefore a custom model is released: trace its lineage by hash, work out what its parents' licenses pass on, see what an edit can do to a safeguard, and write its card.40 min
48Capstone: ship a custom modelOptionalSet release gates, then diagnose, edit, measure and document a tiny GPT with a fault no lesson has fixed, and write the record a stranger could check.45 min
Read a model's weights, change them by hand and judge an edit someone else made: deltas, merging, low-rank edits, directions and fact edits on a tiny GPT in your browser. Then replay recorded sessions of an LLM reading, fixing and writing weights, check the published edit of a real open model, and ship a model of your own with its patch, verdict and card. Every number is printed by the lesson's own code, and the method's limits are taught with it.
See the course →
Never trained an AI? Teach a small one to write, right in your browser. It takes about 20 seconds and needs no account. Teach it to write