Find a tiny GPT's loop by hand, then replay recorded sessions in which an LLM hunts for it with ten forward passes, and measure every fix on held-out questions.
About 45 minutes · Optional lesson: deeper material that no phase or certificate requires · A free account opens the whole lesson. Already have an account? Sign in to open this lesson
The best-known result in the post is a repair. A small open model kept looping, and an LLM traced the loop to one direction written by two layers and removed it from 0.016% of the weights. The authors report that the model's math score went from 7.9% to 48.1%. Nobody retrained anything.
You have a model with the same fault in small: tiny-gpt-loop answers correctly and then can't stop. Here you find the fault yourself, with the tools from the lesson on directions. Then you replay recorded sessions in which an LLM was given the same model, ten forward passes and no hints, and you measure its work with the test you used on yours.
What you'll be able to do
Trace a looping model's fault to a direction in the residual stream and the hidden units that write it
Remove that direction from a few columns by hand, and measure the fix on practice questions
Replay a recorded session of an LLM working on the same model inside a budget of forward passes
Compare fixes by what they change, what they restore on held-out questions and what they cost
Quick check
A small language model answers every question correctly, then repeats its answer until it is cut off. A guess: how much of the model has to change to make it stop?
None of it: the fault is in the prompt, and the weights are fine
About half: stopping is a skill spread evenly over all of the layers
A small part: a few columns of two matrices, a few percent of the weights
All of it: a model with a fault this visible has to be trained again
Show the answer
A small part: a few columns of two matrices, a few percent of the weights
You will fix it by changing 1,024 of 40,195 weights, about 2.5%. The model already knows the answer and already knows how to stop. Something small is pushing against stopping, and it can be found and taken out.
The claim: a model that can't stop has one direction to remove
tiny-gpt-loop was fine-tuned to answer one kind of question: what a product is for. A reply counts as right when it ends and is an answer the catalog accepts.
Before you look, split the questions. The catalog has 30 product names. Six are held out: nobody works on them, and they are only used for the final measurement. You work on six others, the practice names. If you tune a fix on the same questions you later score it on, the score can't tell a real repair from a lucky fit. The post's editor had the same split: practice problems from training sets, and benchmarks it never saw.
Worked example
The symptom
Build the model, ask it one practice question, then score it on all 72 practice questions: 6 names times 12 kinds of product.
import wsp
loop, info = wsp.load_state("tiny-gpt-loop")
model, tok = wsp.build(loop, info)
facts = wsp.facts()
PRACTICE = [name for name in facts["names"] if name notin facts["held_out"]][:6]
print("practice names:", PRACTICE)
question = wsp.question("uses", "Summit", "rain jacket")
print(question)
print(repr(wsp.answer(model, tok, question)))
print("right:", wsp.score(model, tok, "uses", PRACTICE), "of 72")
print("replies that end:", sum(wsp.endings(model, tok, "uses", PRACTICE)), "of 72")
It runs right here in your browser, with nothing to install.
"long commutes." is an accepted answer for a rain jacket. The model gives it, then gives it again until the limit of 12 tokens cuts it off. No reply is counted right, because only one of the 72 ever ends. This is the pattern the post describes in Qwen: the ability is there, and something keeps it from finishing.
Where it should have stopped
A loop is a wrong choice made at one kind of place. After "long commutes ." the model should say <end>. It says "long" instead. So that is the position to study: the full stop that closes the first sentence of a reply.
At any position, a model holds a probability for every token it could say next. wsp.next_probs returns them. If the fault is narrow, <end> will be low there on every practice question.
Worked example
The next token, where the reply should end
For each practice question, build the text up to the end of the model's first sentence. Look at what it wants to say next on the first one, then count the top choice over all 72.
import numpy as np
import wsp
loop, info = wsp.load_state("tiny-gpt-loop")
model, tok = wsp.build(loop, info)
facts = wsp.facts()
PRACTICE = [name for name in facts["names"] if name notin facts["held_out"]][:6]
END = tok.token_id("<end>")
texts = []
for name in PRACTICE:
for product in facts["products"]:
question = wsp.question("uses", name, product["type"])
texts.append(question + " " + wsp.answer(model, tok, question).split(".")[0]
+ " .")
print(texts[0])
probs = wsp.next_probs(model, tok, texts[0])
for i in np.argsort(-probs)[:3]:
print(f" {tok.vocab[i]:8s} {probs[i]:.3f}")
print(f" <end> {probs[END]:.4f}")
nexts = [int(wsp.next_probs(model, tok, text).argmax()) for text in texts]
counts = {}
for t in nexts:
counts[tok.vocab[t]] = counts.get(tok.vocab[t], 0) + 1print(sorted(counts.items(), key=lambda item: -item[1]))
In the lesson you can change this and run it.
After the first sentence the model gives "long" 75% and <end> 0.2%. Over the 72 questions its top choice is never <end>: it is "cold" 29 times and "long" 27 times, both words that open an answer. So the model isn't stuck on one word. At the end of an answer it treats the position as the start of one.
Quick check
The model's top choice after its first sentence is one of six different words, depending on the question. What does that tell you about the fix?
The fault is in six separate places, one for each of the words, and needs six separate repairs
Nothing can be done in the weights, because the words differ from one question to the next
Lowering one word's score won't do: the push is toward answer words in general, and away from the end token
The model has forgotten the end token, so its row in the output layer has to be written again
Show the answer
Lowering one word's score won't do: the push is toward answer words in general, and away from the end token
If the model only overrated "long", you could find what boosts that word. Six different openers point to something shared: at this position, whatever separates carrying on from stopping is tipped the wrong way. That shared thing is what you look for next.
The direction that says "carry on"
You met this move in Directions: steer the run or edit the weights. The output layer scores a token by the dot product of that token's row with the final hidden vector. So the score of "long" minus the score of <end> is the dot product of the hidden vector with one fixed vector: row "long" minus row <end>. That vector is a direction in the residual stream. The further the stream points along it, the more the model prefers carrying on.
Two details. The final LayerNorm scales each of the 32 dimensions before the output layer reads them, so each row is multiplied by that scale. And there are six opening words, so you take the difference for whatever word the model chose on each question and average the 72 vectors.
You may read the output layer to find the direction. You may not edit it: the harness rules in The editor's harness forbid that, because it would hide the fault and leave its cause in place.
Worked example
Find the direction, and how far the stream points along it
Build the direction from the output layer's rows. Then measure the residual stream's component along it at the 72 places where the reply should end.
import numpy as np
import wsp
loop, info = wsp.load_state("tiny-gpt-loop")
model, tok = wsp.build(loop, info)
facts = wsp.facts()
PRACTICE = [name for name in facts["names"] if name notin facts["held_out"]][:6]
END = tok.token_id("<end>")
texts = []
for name in PRACTICE:
for product in facts["products"]:
question = wsp.question("uses", name, product["type"])
texts.append(question + " " + wsp.answer(model, tok, question).split(".")[0]
+ " .")
nexts = [int(wsp.next_probs(model, tok, text).argmax()) for text in texts]
rows = loop["head.weight"].astype(np.float64) * loop["ln_f.weight"].astype(np.float64)
direction = np.mean([rows[t] - rows[END] for t in nexts], axis=0)
direction = direction / np.linalg.norm(direction)
print("direction:", direction.shape, "length",
round(float(np.linalg.norm(direction)), 3))
along = [float(wsp.stream(model, tok, text) @ direction) for text in texts]
print("stream along the direction, where the reply should end:")
print(" mean", round(float(np.mean(along)), 2), "| positive at",
sum(a > 0for a in along), "of 72 | highest", round(max(along), 2))
In the lesson you can change this and run it.
The direction is one vector of 32 numbers, scaled to length 1. At the places where the reply should end, the stream points along it by 1.55 on average. Something upstream writes that component. In a transformer only a few tensors write to the stream at all: the embeddings, each block's attention output and each block's feed-forward output. You find the writer by measuring.
Quick check
A toy output layer with two rows, and one hidden vector. The first number printed is the score of "long" minus the score of the end token. The second uses the difference of the two rows as a direction. What does this print?
"long" scores 2 + 0.5 = 2.5 and the end token 0.5 + 1.5 = 2.0, a gap of 0.5. The direction is [1.5, -2], and its dot product with the hidden vector is 1.5 - 1 = 0.5. The two numbers are always equal: the gap between two scores is how far the hidden vector points along the difference of their rows.
In your own words
In your own words: why is the difference between two rows of the output layer a direction worth following back into the model?
Write a sentence or two, then compare it with a good answer in the lesson.
Who writes it
Start with the feed-forward layers, where a model keeps most of what it has learned. Each one has 128 hidden units. A unit's activation at a position says how strongly it fires there. Its column in the output matrix mlp.2.weight is the vector it adds to the stream when it fires.
So one unit's push along the direction, at one position, is a product of two numbers: its activation, and the dot product of its column with the direction. A unit that fires hard at the end of an answer and writes straight along the direction has a large positive push. A unit that is silent there has none, whatever its column holds.
wsp.mlp_keys returns the 128 activations at each position of a text. Average them over the 72 places, multiply by what each column writes along the direction, and rank the units.
Drill
Each hidden unit's push along a direction
Write push(acts, W, direction). acts holds one activation per hidden unit, W is the output matrix with one column per hidden unit, and direction is a vector in the stream. Return one number per unit: its activation times the component of its column along the direction. The starter tries it on three units writing into a stream of two dimensions.
In the lesson, your code is checked as you write it.
Worked example
The units that push hardest
For each feed-forward layer, average the hidden activations at the 72 places, compute every unit's push along the direction, and list the 16 strongest.
import numpy as np
import wsp
loop, info = wsp.load_state("tiny-gpt-loop")
model, tok = wsp.build(loop, info)
facts = wsp.facts()
PRACTICE = [name for name in facts["names"] if name notin facts["held_out"]][:6]
END = tok.token_id("<end>")
texts = []
for name in PRACTICE:
for product in facts["products"]:
question = wsp.question("uses", name, product["type"])
texts.append(question + " " + wsp.answer(model, tok, question).split(".")[0]
+ " .")
nexts = [int(wsp.next_probs(model, tok, text).argmax()) for text in texts]
rows = loop["head.weight"].astype(np.float64) * loop["ln_f.weight"].astype(np.float64)
direction = np.mean([rows[t] - rows[END] for t in nexts], axis=0)
direction = direction / np.linalg.norm(direction)
columns = {}
for block in (0, 1):
name = f"blocks.{block}.mlp.2.weight"
acts = np.mean([wsp.mlp_keys(model, tok, text, block=block)[-1] for text in texts],
axis=0)
push = acts * (direction @ loop[name].astype(np.float64))
top = np.argsort(-push)[:16]
columns[name] = sorted(int(c) for c in top)
print(name)
print(" 16 strongest units:", columns[name])
print(" their push:", round(float(push[top].sum()), 2), "| all positive pushes:",
round(float(push[push > 0].sum()), 2), "| every unit:",
round(float(push.sum()), 2) or0.0)
In the lesson you can change this and run it.
In each layer, 16 of the 128 units carry most of the push toward carrying on: 1.39 of 1.87 in block 0, and 3.75 of 4.43 in block 1. The last number matters too. Over all 128 units of block 0 the pushes cancel to zero, because other units write the opposite way, toward <end>. Keep that in mind when you choose how much to remove.
Quick check
One column of an output matrix, and a direction of length 1. The edit subtracts the column's component along the direction. What does this print: the edited column, and its component along the direction?
import numpy as np
column = np.array([3.0, 4.0])
d = np.array([1.0, 0.0])
edited = column - (column @ d) * d
print(edited, edited @ d)
[0. 0.] 0.0
[0. 4.] 0.0
[3. 0.] 3.0
[2. 4.] 2.0
Show the answer
[0. 4.] 0.0
The column's component along d is 3. Subtracting 3 × d removes it and leaves [0, 4]: everything the column wrote in other directions is untouched. The edited column's component along d is 0, so the unit can no longer push the stream that way.
Try it now
Step through. Each unit's push is its activation times its column's part along the direction.
Take the direction out of 16 columns
You know a direction and you know which columns write along it. The edit is the one from Directions: steer the run or edit the weights, made narrow. For each chosen column, subtract its component along the direction and leave the rest of the column alone. The unit still fires and still writes everything else it wrote. It can no longer push the stream toward carrying on.
wsp.project_columns(state, name, direction, columns) does this for one matrix. Two calls change 16 columns of 32 numbers in each of two matrices: 1,024 weights, with no training.
Worked example
The fix by hand, measured on practice questions
Apply the edit to both matrices and score the edited model on the 72 practice questions. Then compare your edit with the patch the practice API serves as loop-fix, which was made by the same recipe.
import numpy as np
import wsp
loop, info = wsp.load_state("tiny-gpt-loop")
model, tok = wsp.build(loop, info)
facts = wsp.facts()
PRACTICE = [name for name in facts["names"] if name notin facts["held_out"]][:6]
END = tok.token_id("<end>")
texts = []
for name in PRACTICE:
for product in facts["products"]:
question = wsp.question("uses", name, product["type"])
texts.append(question + " " + wsp.answer(model, tok, question).split(".")[0]
+ " .")
nexts = [int(wsp.next_probs(model, tok, text).argmax()) for text in texts]
rows = loop["head.weight"].astype(np.float64) * loop["ln_f.weight"].astype(np.float64)
direction = np.mean([rows[t] - rows[END] for t in nexts], axis=0)
direction = direction / np.linalg.norm(direction)
edited = loop
for block in (0, 1):
name = f"blocks.{block}.mlp.2.weight"
acts = np.mean([wsp.mlp_keys(model, tok, text, block=block)[-1] for text in texts],
axis=0)
push = acts * (direction @ loop[name].astype(np.float64))
edited = wsp.project_columns(edited, name, direction, np.argsort(-push)[:16])
fixed, _ = wsp.build(edited, info)
patch = wsp.make_patch(loop, edited)
print("weights changed:", patch["changed"], "in", list(patch["tensors"]))
print("practice questions right:", wsp.score(fixed, tok, "uses", PRACTICE),
"of 72 | replies that end:", sum(wsp.endings(fixed, tok, "uses", PRACTICE)))
served = wsp.apply_patch(loop, wsp.load_patch("loop-fix"))
print("largest difference from the served patch:",
max(float(abs(served[k] - edited[k]).max()) for k in loop))
In the lesson you can change this and run it.
From 0 right to 66 of 72, with 70 replies ending, by changing 1,024 weights. Your edit and the served patch differ by at most 0.000000015 in any weight: the same edit, up to the rounding of the direction when the patch was stored. That is the patch you reviewed, or will review, in the lesson on testing an edit. There it restores 68 of 72 held-out questions, and the same edit on all 128 columns restores none, because it also removes the units that push toward <end>.
In your own words
In your own words: you never compared the looping model with a healthy one. How did you find where to edit?
Write a sentence or two, then compare it with a good answer in the lesson.
The replay: the same model, given to an LLM
Now the recordings. An LLM gives a different answer each time, so each session was run once, on November 10, 2026, and saved: every script, its output, the forward passes and the final reply. Nothing is generated when you run a cell.
The editor was Claude Fable 5.2 inside Claude Code, in a fresh session. It could see the name of this course and nothing of its contents. It was told: here is a model that gets 0 of 72 practice questions right, find out why and fix it by editing weights. It was not told that the model loops, where the fault is or how you fixed it. It worked in a lab built like the harness in The editor's harness:
a NumPy kernel with the weights and the 72 practice questions, and at most 40 code runs;
a forward-pass tool with a budget of 10 passes, where one pass answers the practice questions or reads up to 144 texts the editor writes;
no training, no running the model outside that tool, and no edit to the embeddings, the final norm or the output layer.
The lab enforced the two budgets and the refused edits. The other rules were checked afterward by reading every script.
Worked example
Replay the first session, step by step
Load the recording. Print its limits, what it used and what the audit found. Then list every code run with the forward passes it spent and the start of the explanation the editor had to write at the top of each script.
import wsp
session = wsp.load_session("fix-loop-1")
print("recorded:", session["recorded"], "| editor:", session["editor"], "| date:",
session["date"])
passes = sum(len(step["passes"]) for step in session["steps"])
print("limits:", session["limits"], "| used:", len(session["steps"]), "runs and",
passes, "forward passes")
calls = session["audit"]["tool_calls"]
print("tool calls:", {kind: sum(call.startswith(kind) for call in calls)
for kind in ("Read", "Write", "Edit", "Run")},
"| outside the rules:", session["audit"]["outside_the_rules"])
print()
for step in session["steps"]:
spent = ", ".join(f"{p['kind']} {p['which']}"for p in step["passes"]) or"no pass"print(f"{step['run']:2d} [{spent}] {step['explanation'][:84]}")
In the lesson you can change this and run it.
Twenty-two runs and 7 of its 10 passes. Sixteen runs spent no pass at all: most of the work is arithmetic on the weights and on what earlier passes returned. Pass 1 showed the symptom, as your first cell did. Passes 2 and 3 read texts the editor wrote: answers up to their full stop, and catalog-style sentences for contrast. Runs 11 and 12 ended in the editor's own errors before anything was spent. The audit counts 47 tool calls: one read of the task, 24 scripts written or changed, 22 runs, and none outside the rules.
Quick check
In run 14 the editor saved an edit, spent one pass reading with it, and then removed it again. Its explanation says the pass was to "measure what block 1's feed-forward layer receives as input." Why would an editor do that?
It had saved the edit by mistake, and reading with it was the only way to clear it again
The lab's read doesn't return that input, so it turned some units into meters for one pass
It was testing a first version of its fix, and removed it when the score turned out low
It was getting around the budget, since a pass made with a temporary edit isn't counted
Show the answer
The lab's read doesn't return that input, so it turned some units into meters for one pass
The rules say the model runs only inside the tool. So the editor rewired 64 hidden units, for one pass, so that their activations carried the layer's input back out. The pass was counted like any other. It is a neat trick, and it shows what a budget buys: the editor chose to spend a pass on a measurement, because it could not compute that number itself.
What it found, and what it built
You removed something: a direction, from the columns that wrote it. The editor went another way. Its final reply says, under DIAGNOSIS:
No single corrupted spot: row and column norms, singular values, the <end> row of head.weight, per-head q/k/v/proj norms and the head value circuits all look ordinary. So this is a missing mechanism that I built, not a damaged weight that I repaired.
It never found the columns you found. It decided the model lacked a way to stop, and wrote one. It took the least used hidden unit of block 1, number 72, and rebuilt it as a "full stop, so stop" unit. The unit's input row makes it fire at a full stop. Its output column pushes the stream toward <end>. That is 32 weights in, 1 bias and 32 weights out: 65 weights, where your fix changed 1,024.
Both the row and the column came from closed-form formulas over measurements, the same kind of arithmetic as the rank-one edit in Editing one fact. The next cell applies the recorded patch.
Worked example
Apply the recorded patch
Load the patch the session made, see which entries it changes, and score the patched model on the practice questions the editor worked with.
import wsp
loop, info = wsp.load_state("tiny-gpt-loop")
facts = wsp.facts()
PRACTICE = [name for name in facts["names"] if name notin facts["held_out"]][:6]
session = wsp.load_session("fix-loop-1")
patch = wsp.load_patch(session["made"])
print(session["made"], "|", patch["changed"], "weights:",
{name: len(t["index"]) for name, t in patch["tensors"].items()})
print("unit:",
sorted({i // 32for i in patch["tensors"]["blocks.1.mlp.0.weight"]["index"]}),
"| column:",
sorted({i % 128for i in patch["tensors"]["blocks.1.mlp.2.weight"]["index"]}))
edited = wsp.apply_patch(loop, patch)
fixed, tok = wsp.build(edited, info)
print("practice questions right:", wsp.score(fixed, tok, "uses", PRACTICE), "of 72")
print("the editor's own report:", session["practice"])
In the lesson you can change this and run it.
The patch touches one row of the input matrix, one bias and one column of the output matrix, all belonging to unit 72. On the practice questions it scores 72 of 72, which is what the editor reported from its last paired pass. Your fix scored 66. On the numbers the editor could see, its edit is better than yours and about sixteen times smaller.
In your own words
In your own words: the editor never found the columns you found. What did it do instead?
Write a sentence or two, then compare it with a good answer in the lesson.
The grading: the final measurement, and what to trust
Neither you nor the editor has looked at the held-out names. Now is the time, once, for every candidate.
The score on held-out questions is half of the measurement. The task said the edited model would also be judged on its ordinary writing, the catalog's product descriptions. Two numbers cover that. The catalog loss is the model's average loss per token on 40 descriptions, where lower is better. And each description has three or four sentences, so after its first sentence a sound model carries on. The last column counts the descriptions where the model's top choice at that point is <end>.
The row for tiny-gpt-uses is the model before the fault was planted. Neither you nor the editor had it. It is here as the mark to aim for.
Worked example
Every fix, on the same test
Score the looping model, the fix by hand and each recorded fix on the 72 held-out questions. Then measure their ordinary writing.
import wsp
loop, info = wsp.load_state("tiny-gpt-loop")
healthy, _ = wsp.load_state("tiny-gpt-uses")
tok = wsp.build(loop, info)[1]
END = tok.token_id("<end>")
candidates = {"tiny-gpt-uses": healthy, "tiny-gpt-loop": loop,
"loop-fix": wsp.apply_patch(loop, wsp.load_patch("loop-fix"))}
for name in wsp.sessions("fix"):
made = wsp.load_session(name)["made"]
candidates[made] = wsp.apply_patch(loop, wsp.load_patch(made))
catalog = wsp.catalog()
firsts = [line.split(".")[0] + " ."for line in catalog]
print(f"{'model':15s} {'changed':>7s} {'held-out':>8s} {'end':>4s} "f"{'catalog loss':>12s} {'stops early':>11s}")
for name, state in candidates.items():
model, _ = wsp.build(state, info)
changed = wsp.make_patch(loop,
state)["changed"] if name.startswith("loop-fix") else0
early = sum(int(wsp.next_probs(model, tok, text).argmax()) == END
for text in firsts)
print(f"{name:15s} {changed:7d} {wsp.score(model, tok, 'uses'):8d} "f"{sum(wsp.endings(model, tok, 'uses')):4d} "f"{wsp.text_loss(model, tok, catalog):12.2f} {early:8d}/40")
In the lesson you can change this and run it.
On the task the recorded fixes win. Both answer all 72 held-out questions with 65 weights. Yours answers 68 with 1,024.
The last two columns tell another story. The planted fault had damaged ordinary writing too: the catalog loss went from 1.15 to 1.93. Your fix brings it back to 1.52. The first recorded fix leaves it at 1.89 and ends every description after its first sentence, 40 of 40. The second leaves the loss at 1.93, exactly where the fault put it: no new damage, and no repair.
Quick check
A fix answers every held-out question correctly. On ordinary catalog text, the fixed model now ends a description after its first sentence, 40 times of 40. What is the fairest summary?
It broke the rules, because an edit may not change how ordinary text is written
It is a complete repair, because held-out questions are the only fair test
It repaired the task that was tested and planted a new fault in what was not
It did nothing, because the catalog loss is nearly the same as before the fix
Show the answer
It repaired the task that was tested and planted a new fault in what was not
Both halves are true and both belong in the report. The model stops where it should and also where it shouldn't. Held-out questions measure the first half only. A score on the target task says nothing about damage elsewhere, which is why the task named ordinary writing as part of the judgment.
Where it shows up in your week
When someone hands you a fix with a single score, you ask for the second measurement and for the editor's own account of the damage. Read the session log, check the budget against what the lab enforced, and compare the patch with a control.
In your own words
In your own words: both recorded fixes score 72 of 72. What in the table would make you hesitate before shipping the first one?
Write a sentence or two, then compare it with a good answer in the lesson.
The editor knew
Read the end of the session's reply. Under DOUBTS, before anyone had measured anything on the catalog, the editor wrote this:
The unit fires at every full stop, not only in answers. In catalog text the model now puts about 0.95 on <end> after a full stop [...] text generated from catalog prompts would stop after the first sentence.
It had measured the side effect itself, on catalog-style texts it wrote from the vocabulary, with one of its passes. It also explained why it couldn't do better: at a full stop, it found nothing in the stream that tells an answer from a description, so a single unit can't fire on one and not the other. It sketched a larger fix, about 196 weights, and stopped with 3 passes unspent.
That is worth as much as the patch. A fix that comes with an accurate account of its own damage can be reviewed. One that comes with a score alone can't.
Drill
Measure the side effect yourself
Write stops_early(model, tok, lines): the number of texts in lines where, right after the first sentence, the model's most likely next token is <end>. Take the first sentence as everything up to the first full stop, and put the full stop back as its own token: line.split(".")[0] + " .". The starter runs it on the looping model with the LLM's patch.
In the lesson, your code is checked as you write it.
The second session
One run is an anecdote. That is a fair criticism of the post, which reports one run for each budget. So the session was recorded twice, with the same task and a fresh editor each time.
Session 2 reached the same diagnosis, "no tensor is damaged," and made the same kind of edit: one rebuilt unit, number 97 this time, 65 weights. It differed in how it spent its budget. It used all 10 passes, three of them to measure how the full stop of an answer differs from the full stop of a catalog sentence. Its unit fires on the first and mostly not on the second, which session 1 had judged one unit couldn't do. That is the row loop-fix-llm-2 in your table: the same 72 of 72, and no description cut short.
Neither session found what you found. The fault had been planted by changing two matrices a little in every column, too little for the norms and singular values that both editors checked to show anything. Your recipe never needed to see the change. It followed the symptom to the columns that carry it, and so your fix is the only one that makes ordinary writing better. Both editors added a way to stop and left the cause where it was.
Quick check
Both recorded sessions concluded that no tensor was damaged. Two matrices had in fact been changed. Why could two careful sessions miss that?
The editors were barred from reading those two matrices, so they could not look at the change
The change was small in every column, and norms and singular values don't show a change like that
The editors never ran the model, so they had no way to know that it had a fault to look for
The change was in the output layer, which the rules put off limits, so nobody could inspect it
Show the answer
The change was small in every column, and norms and singular values don't show a change like that
With nothing to compare against, a matrix that moved a little everywhere looks like any trained matrix. Summary statistics can only flag a spot that stands out. Your recipe didn't need the change to stand out. It started from the behavior and asked which units produce it, where they fire.
Enforced, audited, and taken on trust
How much of this session can you rely on? Sort the rules by how they were kept.
Enforced by the lab. Ten passes and forty runs: session 1 used 7 and 22. No edit to the embeddings, the final norm or the output layer: the function that saves an edit refuses them. A kernel with NumPy and the lab, and nothing else.
Audited afterward. The transcript lists every tool call the session made: one read of the task, its own scripts written or changed in its folder, and runs of them. Nothing else. Then a second session, which was given only the rules and the logged scripts, read all 22 of them for training and for code that runs the model itself. Its findings are stored with the recording.
Worked example
What the audit of the scripts found
Print three of the audit's findings, as they are stored with the recording.
import wsp
audit = wsp.load_session("fix-loop-1")["audit"]
for key in ("scripts_read_by", "training", "own_forward_pass"):
print(key.upper())
print(" " + audit[key])
In the lesson you can change this and run it.
No training, and no forward pass on any text. But the reader would not sign off on the second rule without a caveat. In four scripts the editor took vectors from the weights, a token's embedding or a column it was designing, and put them through a layer's own formula to see what the output layer would make of them. The task allowed "any arithmetic on the weights themselves" and forbade computing "a layer of it on any input." Those two phrases overlap here, and the audit says so. The editor had listed the same steps itself, as "judgement calls an auditor should see." A separate reading of session 2's 29 scripts ended the same way: no training, no forward pass on text, and the same caveat.
What counts as training?
One more edge. The row and the column of unit 72 came from least-squares formulas over recorded activations. Nothing was iterated and no practice answer was used as a target, so by the letter of the rules it isn't training. But a formula that fits a direction to data is a close relative of training, and you may draw the line elsewhere. The rule in the post's logs has the same edge: it bars "iterative fitting of weights to a loss," and its editor also built its edits from measured directions.
Rules about what an editor may compute are hard to write and harder to check. That is the practical argument for a harness that enforces what it can, logs everything, and lets a reader see the rest.
Quick check
From Did it work? Testing an edit: a patch restores the held-out score. Which further check tells you whether the reasoning behind the patch was needed?
A second run of the same patch on the same held-out questions as before
A control: an edit of the same size and form with its aim replaced by chance
A look at the editor's confidence, since a sure editor reasoned correctly
A count of how many weights the patch changes, since smaller is always better
Show the answer
A control: an edit of the same size and form with its aim replaced by chance
If a random edit of the same size works as well, the model was easy to knock out of its fault and the story about why the patch works is unsupported. The by-hand patch passes that test in the lesson on testing an edit: six controls restore 3 questions at most. The recorded patches haven't been through it here.
In your own words
In your own words: the by-hand fix restores 68 of 72 and the LLM's first fix 72 of 72. Which would you ship, and what would you write next to it?
Write a sentence or two, then compare it with a good answer in the lesson.
11DistillationTrain a small model to imitate a large one.35 min
12Tuning an embedding model for retrievalWhen search fails on your own vocabulary: train an adapter with a contrastive loss on question and passage pairs, test on held-out passages, use batch size and hard negatives, and measure what tuning costs elsewhere.45 min
13QuantizationStore weights in fewer bits to run models cheaply.35 min
23Copying from the promptIn-context learning as a two-head copying circuit: the base tiny GPT cannot copy a repeated pattern, a model trained on repeats grows an induction head and can.30 min
24Why temperature changes the answerThe same prompt at 0, 0.7 and 1.5: deterministic loops, plausible variety, broken grammar, measured on a real model.25 min
25Why long prompts diluteAttention is a budget shared by every token, so a fact far from the question loses its share; measured, with the cliff beyond the trained window.25 min
26Why it never says "I don't know"A chat model answers every request because its fine-tuning data always did; preference training adds flattery; how to build abstention in.25 min
32Weight-space programming: what it is, and what has been shownFreeOptionalA language model reads and writes another network's weights with no training loop: the published experiments, their limits and the criticism, and one edit you make by hand.30 min
33Anatomy of a checkpointOptionalA model file is a set of named tensors: count them, find the ones that write to the residual stream, and save one to bytes and back.35 min
34Reading weights without running the modelOptionalSingular values, and an attention head's token-to-token circuit, read from the matrices alone and then checked against a measurement.35 min
35Writing weights by handOptionalSet a small network's 55 weights so it catches every ball in a paddle game, the way a program would be written, and meet compiled transformers.40 min
36Edits are deltas: task vectors, patches and hashesOptionalSubtract a base from a fine-tune and the edit becomes an object: scale it, take it away, ship it as a patch, and prove by hash what was applied.35 min
37Merging modelsOptionalAverage two fine-tunes, add their task vectors, try SLERP, TIES and DARE, measure the interference, and compare with a model trained on both tasks.40 min
38Low-rank editsOptionalCut a fine-tune's change down to its largest directions with the SVD: how few a behavior needs, and how to pull an adapter out of a delta.35 min
39Directions: steer the run or edit the weightsOptionalFind a behavior's direction in the residual stream, add it at run time, project it out of the weights, and see the effect change from input to input.40 min
40Editing one factOptionalA rank-one change to one matrix rewrites what a product is made from; then reliability, generalization, locality, and what breaks after many edits.40 min
41The editor's harnessOptionalWhat an LLM that edits weights is given and what it is denied: weights as text and their cost in tokens, a sandbox, a budget of forward passes, and rules the harness enforces.40 min
42An LLM reads a network's weightsOptionalSix recorded sessions: an LLM is handed 1,315 weights as text and asked what the network was trained to do. Score each answer against the network, and study one it got wrong.40 min
43An LLM fixes a model's weightsOptionalFind a tiny GPT's loop by hand, then replay recorded sessions in which an LLM hunts for it with ten forward passes, and measure every fix on held-out questions.45 min
44An LLM writes a network's weightsOptionalFive recorded sessions: an LLM writes every weight of a paddle network from the game's source and one sentence. Score each against its specification, test its design claims, and edit one by hand.40 min
45Did it work? Testing an editOptionalReview a patch for a tiny GPT that can't stop talking: held-out questions, a paired comparison, a same-size control, and what the edit cost.40 min
46The published edit, on a real modelOptionalRead the edit published for an open model of 870 million weights, apply and verify one in the same form, see what 16-bit and 4-bit storage do to a fix, and run the real thing from a notebook.45 min
47Safety, licenses and provenanceOptionalBefore a custom model is released: trace its lineage by hash, work out what its parents' licenses pass on, see what an edit can do to a safeguard, and write its card.40 min
48Capstone: ship a custom modelOptionalSet release gates, then diagnose, edit, measure and document a tiny GPT with a fault no lesson has fixed, and write the record a stranger could check.45 min
Read a model's weights, change them by hand and judge an edit someone else made: deltas, merging, low-rank edits, directions and fact edits on a tiny GPT in your browser. Then replay recorded sessions of an LLM reading, fixing and writing weights, check the published edit of a real open model, and ship a model of your own with its patch, verdict and card. Every number is printed by the lesson's own code, and the method's limits are taught with it.
See the course →
Never trained an AI? Teach a small one to write, right in your browser. It takes about 20 seconds and needs no account. Teach it to write