Index Ai Academy
Phase 6 · Weight-space programming

An LLM fixes a model's weights

Find a tiny GPT's loop by hand, then replay recorded sessions in which an LLM hunts for it with ten forward passes, and measure every fix on held-out questions.

About 45 minutes · Optional lesson: deeper material that no phase or certificate requires · A free account opens the whole lesson.
Already have an account? Sign in to open this lesson

What it covers14

Why it matters

The best-known result in the post is a repair. A small open model kept looping, and an LLM traced the loop to one direction written by two layers and removed it from 0.016% of the weights. The authors report that the model's math score went from 7.9% to 48.1%. Nobody retrained anything.

You have a model with the same fault in small: tiny-gpt-loop answers correctly and then can't stop. Here you find the fault yourself, with the tools from the lesson on directions. Then you replay recorded sessions in which an LLM was given the same model, ten forward passes and no hints, and you measure its work with the test you used on yours.

What you'll be able to do

  • Trace a looping model's fault to a direction in the residual stream and the hidden units that write it
  • Remove that direction from a few columns by hand, and measure the fix on practice questions
  • Replay a recorded session of an LLM working on the same model inside a budget of forward passes
  • Compare fixes by what they change, what they restore on held-out questions and what they cost
Quick check
A small language model answers every question correctly, then repeats its answer until it is cut off. A guess: how much of the model has to change to make it stop?
  1. None of it: the fault is in the prompt, and the weights are fine
  2. About half: stopping is a skill spread evenly over all of the layers
  3. A small part: a few columns of two matrices, a few percent of the weights
  4. All of it: a model with a fault this visible has to be trained again
Show the answer

A small part: a few columns of two matrices, a few percent of the weights

You will fix it by changing 1,024 of 40,195 weights, about 2.5%. The model already knows the answer and already knows how to stop. Something small is pushing against stopping, and it can be found and taken out.

The claim: a model that can't stop has one direction to remove

This lesson rests on Directions: steer the run or edit the weights: a direction in the residual stream, and removing it from the columns that write it.

A model that can't stop

tiny-gpt-loop was fine-tuned to answer one kind of question: what a product is for. A reply counts as right when it ends and is an answer the catalog accepts.

Before you look, split the questions. The catalog has 30 product names. Six are held out: nobody works on them, and they are only used for the final measurement. You work on six others, the practice names. If you tune a fix on the same questions you later score it on, the score can't tell a real repair from a lucky fit. The post's editor had the same split: practice problems from training sets, and benchmarks it never saw.

Worked example

The symptom

Build the model, ask it one practice question, then score it on all 72 practice questions: 6 names times 12 kinds of product.

import wsp

loop, info = wsp.load_state("tiny-gpt-loop")
model, tok = wsp.build(loop, info)
facts = wsp.facts()
PRACTICE = [name for name in facts["names"] if name not in facts["held_out"]][:6]
print("practice names:", PRACTICE)

question = wsp.question("uses", "Summit", "rain jacket")
print(question)
print(repr(wsp.answer(model, tok, question)))
print("right:", wsp.score(model, tok, "uses", PRACTICE), "of 72")
print("replies that end:", sum(wsp.endings(model, tok, "uses", PRACTICE)), "of 72")
It runs right here in your browser, with nothing to install.

"long commutes." is an accepted answer for a rain jacket. The model gives it, then gives it again until the limit of 12 tokens cuts it off. No reply is counted right, because only one of the 72 ever ends. This is the pattern the post describes in Qwen: the ability is there, and something keeps it from finishing.

Where it should have stopped

A loop is a wrong choice made at one kind of place. After "long commutes ." the model should say <end>. It says "long" instead. So that is the position to study: the full stop that closes the first sentence of a reply.

At any position, a model holds a probability for every token it could say next. wsp.next_probs returns them. If the fault is narrow, <end> will be low there on every practice question.

Worked example

The next token, where the reply should end

For each practice question, build the text up to the end of the model's first sentence. Look at what it wants to say next on the first one, then count the top choice over all 72.

import numpy as np
import wsp

loop, info = wsp.load_state("tiny-gpt-loop")
model, tok = wsp.build(loop, info)
facts = wsp.facts()
PRACTICE = [name for name in facts["names"] if name not in facts["held_out"]][:6]
END = tok.token_id("<end>")

texts = []
for name in PRACTICE:
    for product in facts["products"]:
        question = wsp.question("uses", name, product["type"])
        texts.append(question + " " + wsp.answer(model, tok, question).split(".")[0]
                     + " .")
print(texts[0])

probs = wsp.next_probs(model, tok, texts[0])
for i in np.argsort(-probs)[:3]:
    print(f"  {tok.vocab[i]:8s} {probs[i]:.3f}")
print(f"  <end>    {probs[END]:.4f}")

nexts = [int(wsp.next_probs(model, tok, text).argmax()) for text in texts]
counts = {}
for t in nexts:
    counts[tok.vocab[t]] = counts.get(tok.vocab[t], 0) + 1
print(sorted(counts.items(), key=lambda item: -item[1]))
In the lesson you can change this and run it.

After the first sentence the model gives "long" 75% and <end> 0.2%. Over the 72 questions its top choice is never <end>: it is "cold" 29 times and "long" 27 times, both words that open an answer. So the model isn't stuck on one word. At the end of an answer it treats the position as the start of one.

Quick check
The model's top choice after its first sentence is one of six different words, depending on the question. What does that tell you about the fix?
  1. The fault is in six separate places, one for each of the words, and needs six separate repairs
  2. Nothing can be done in the weights, because the words differ from one question to the next
  3. Lowering one word's score won't do: the push is toward answer words in general, and away from the end token
  4. The model has forgotten the end token, so its row in the output layer has to be written again
Show the answer

Lowering one word's score won't do: the push is toward answer words in general, and away from the end token

If the model only overrated "long", you could find what boosts that word. Six different openers point to something shared: at this position, whatever separates carrying on from stopping is tipped the wrong way. That shared thing is what you look for next.

The direction that says "carry on"

You met this move in Directions: steer the run or edit the weights. The output layer scores a token by the dot product of that token's row with the final hidden vector. So the score of "long" minus the score of <end> is the dot product of the hidden vector with one fixed vector: row "long" minus row <end>. That vector is a direction in the residual stream. The further the stream points along it, the more the model prefers carrying on.

Two details. The final LayerNorm scales each of the 32 dimensions before the output layer reads them, so each row is multiplied by that scale. And there are six opening words, so you take the difference for whatever word the model chose on each question and average the 72 vectors.

You may read the output layer to find the direction. You may not edit it: the harness rules in The editor's harness forbid that, because it would hide the fault and leave its cause in place.

Worked example

Find the direction, and how far the stream points along it

Build the direction from the output layer's rows. Then measure the residual stream's component along it at the 72 places where the reply should end.

import numpy as np
import wsp

loop, info = wsp.load_state("tiny-gpt-loop")
model, tok = wsp.build(loop, info)
facts = wsp.facts()
PRACTICE = [name for name in facts["names"] if name not in facts["held_out"]][:6]
END = tok.token_id("<end>")
texts = []
for name in PRACTICE:
    for product in facts["products"]:
        question = wsp.question("uses", name, product["type"])
        texts.append(question + " " + wsp.answer(model, tok, question).split(".")[0]
                     + " .")
nexts = [int(wsp.next_probs(model, tok, text).argmax()) for text in texts]

rows = loop["head.weight"].astype(np.float64) * loop["ln_f.weight"].astype(np.float64)
direction = np.mean([rows[t] - rows[END] for t in nexts], axis=0)
direction = direction / np.linalg.norm(direction)
print("direction:", direction.shape, "length",
      round(float(np.linalg.norm(direction)), 3))

along = [float(wsp.stream(model, tok, text) @ direction) for text in texts]
print("stream along the direction, where the reply should end:")
print("  mean", round(float(np.mean(along)), 2), "| positive at",
      sum(a > 0 for a in along), "of 72 | highest", round(max(along), 2))
In the lesson you can change this and run it.

The direction is one vector of 32 numbers, scaled to length 1. At the places where the reply should end, the stream points along it by 1.55 on average. Something upstream writes that component. In a transformer only a few tensors write to the stream at all: the embeddings, each block's attention output and each block's feed-forward output. You find the writer by measuring.

Quick check
A toy output layer with two rows, and one hidden vector. The first number printed is the score of "long" minus the score of the end token. The second uses the difference of the two rows as a direction. What does this print?
import numpy as np

rows = {"long": np.array([2.0, 1.0]), "<end>": np.array([0.5, 3.0])}
hidden = np.array([1.0, 0.5])
direction = rows["long"] - rows["<end>"]
print(rows["long"] @ hidden - rows["<end>"] @ hidden, direction @ hidden)
  1. 0.5 0.5
  2. 2.5 2.0
  3. 4.5 0.5
  4. 0.5 4.5
Show the answer

0.5 0.5

"long" scores 2 + 0.5 = 2.5 and the end token 0.5 + 1.5 = 2.0, a gap of 0.5. The direction is [1.5, -2], and its dot product with the hidden vector is 1.5 - 1 = 0.5. The two numbers are always equal: the gap between two scores is how far the hidden vector points along the difference of their rows.

In your own words
In your own words: why is the difference between two rows of the output layer a direction worth following back into the model?

Write a sentence or two, then compare it with a good answer in the lesson.

Who writes it

Start with the feed-forward layers, where a model keeps most of what it has learned. Each one has 128 hidden units. A unit's activation at a position says how strongly it fires there. Its column in the output matrix mlp.2.weight is the vector it adds to the stream when it fires.

So one unit's push along the direction, at one position, is a product of two numbers: its activation, and the dot product of its column with the direction. A unit that fires hard at the end of an answer and writes straight along the direction has a large positive push. A unit that is silent there has none, whatever its column holds.

wsp.mlp_keys returns the 128 activations at each position of a text. Average them over the 72 places, multiply by what each column writes along the direction, and rank the units.

Drill

Each hidden unit's push along a direction

Write push(acts, W, direction). acts holds one activation per hidden unit, W is the output matrix with one column per hidden unit, and direction is a vector in the stream. Return one number per unit: its activation times the component of its column along the direction. The starter tries it on three units writing into a stream of two dimensions.

import numpy as np

def push(acts, W, direction):
    return np.zeros(len(acts))

W = np.array([[2.0, 0.0, -1.0],
              [0.0, 3.0, 1.0]])
print(push(np.array([1.0, 1.0, 2.0]), W, np.array([1.0, 0.0])))
In the lesson, your code is checked as you write it.
Worked example

The units that push hardest

For each feed-forward layer, average the hidden activations at the 72 places, compute every unit's push along the direction, and list the 16 strongest.

import numpy as np
import wsp

loop, info = wsp.load_state("tiny-gpt-loop")
model, tok = wsp.build(loop, info)
facts = wsp.facts()
PRACTICE = [name for name in facts["names"] if name not in facts["held_out"]][:6]
END = tok.token_id("<end>")
texts = []
for name in PRACTICE:
    for product in facts["products"]:
        question = wsp.question("uses", name, product["type"])
        texts.append(question + " " + wsp.answer(model, tok, question).split(".")[0]
                     + " .")
nexts = [int(wsp.next_probs(model, tok, text).argmax()) for text in texts]
rows = loop["head.weight"].astype(np.float64) * loop["ln_f.weight"].astype(np.float64)
direction = np.mean([rows[t] - rows[END] for t in nexts], axis=0)
direction = direction / np.linalg.norm(direction)

columns = {}
for block in (0, 1):
    name = f"blocks.{block}.mlp.2.weight"
    acts = np.mean([wsp.mlp_keys(model, tok, text, block=block)[-1] for text in texts],
                   axis=0)
    push = acts * (direction @ loop[name].astype(np.float64))
    top = np.argsort(-push)[:16]
    columns[name] = sorted(int(c) for c in top)
    print(name)
    print("  16 strongest units:", columns[name])
    print("  their push:", round(float(push[top].sum()), 2), "| all positive pushes:",
          round(float(push[push > 0].sum()), 2), "| every unit:",
          round(float(push.sum()), 2) or 0.0)
In the lesson you can change this and run it.

In each layer, 16 of the 128 units carry most of the push toward carrying on: 1.39 of 1.87 in block 0, and 3.75 of 4.43 in block 1. The last number matters too. Over all 128 units of block 0 the pushes cancel to zero, because other units write the opposite way, toward <end>. Keep that in mind when you choose how much to remove.

Quick check
One column of an output matrix, and a direction of length 1. The edit subtracts the column's component along the direction. What does this print: the edited column, and its component along the direction?
import numpy as np

column = np.array([3.0, 4.0])
d = np.array([1.0, 0.0])
edited = column - (column @ d) * d
print(edited, edited @ d)
  1. [0. 0.] 0.0
  2. [0. 4.] 0.0
  3. [3. 0.] 3.0
  4. [2. 4.] 2.0
Show the answer

[0. 4.] 0.0

The column's component along d is 3. Subtracting 3 × d removes it and leaves [0, 4]: everything the column wrote in other directions is untouched. The edited column's component along d is 0, so the unit can no longer push the stream that way.

Try it now

Step through. Each unit's push is its activation times its column's part along the direction.

Take the direction out of 16 columns

You know a direction and you know which columns write along it. The edit is the one from Directions: steer the run or edit the weights, made narrow. For each chosen column, subtract its component along the direction and leave the rest of the column alone. The unit still fires and still writes everything else it wrote. It can no longer push the stream toward carrying on.

wsp.project_columns(state, name, direction, columns) does this for one matrix. Two calls change 16 columns of 32 numbers in each of two matrices: 1,024 weights, with no training.

Worked example

The fix by hand, measured on practice questions

Apply the edit to both matrices and score the edited model on the 72 practice questions. Then compare your edit with the patch the practice API serves as loop-fix, which was made by the same recipe.

import numpy as np
import wsp

loop, info = wsp.load_state("tiny-gpt-loop")
model, tok = wsp.build(loop, info)
facts = wsp.facts()
PRACTICE = [name for name in facts["names"] if name not in facts["held_out"]][:6]
END = tok.token_id("<end>")
texts = []
for name in PRACTICE:
    for product in facts["products"]:
        question = wsp.question("uses", name, product["type"])
        texts.append(question + " " + wsp.answer(model, tok, question).split(".")[0]
                     + " .")
nexts = [int(wsp.next_probs(model, tok, text).argmax()) for text in texts]
rows = loop["head.weight"].astype(np.float64) * loop["ln_f.weight"].astype(np.float64)
direction = np.mean([rows[t] - rows[END] for t in nexts], axis=0)
direction = direction / np.linalg.norm(direction)

edited = loop
for block in (0, 1):
    name = f"blocks.{block}.mlp.2.weight"
    acts = np.mean([wsp.mlp_keys(model, tok, text, block=block)[-1] for text in texts],
                   axis=0)
    push = acts * (direction @ loop[name].astype(np.float64))
    edited = wsp.project_columns(edited, name, direction, np.argsort(-push)[:16])

fixed, _ = wsp.build(edited, info)
patch = wsp.make_patch(loop, edited)
print("weights changed:", patch["changed"], "in", list(patch["tensors"]))
print("practice questions right:", wsp.score(fixed, tok, "uses", PRACTICE),
      "of 72 | replies that end:", sum(wsp.endings(fixed, tok, "uses", PRACTICE)))

served = wsp.apply_patch(loop, wsp.load_patch("loop-fix"))
print("largest difference from the served patch:",
      max(float(abs(served[k] - edited[k]).max()) for k in loop))
In the lesson you can change this and run it.

From 0 right to 66 of 72, with 70 replies ending, by changing 1,024 weights. Your edit and the served patch differ by at most 0.000000015 in any weight: the same edit, up to the rounding of the direction when the patch was stored. That is the patch you reviewed, or will review, in the lesson on testing an edit. There it restores 68 of 72 held-out questions, and the same edit on all 128 columns restores none, because it also removes the units that push toward <end>.

In your own words
In your own words: you never compared the looping model with a healthy one. How did you find where to edit?

Write a sentence or two, then compare it with a good answer in the lesson.

The replay: the same model, given to an LLM

Now the recordings. An LLM gives a different answer each time, so each session was run once, on November 10, 2026, and saved: every script, its output, the forward passes and the final reply. Nothing is generated when you run a cell.

The editor was Claude Fable 5.2 inside Claude Code, in a fresh session. It could see the name of this course and nothing of its contents. It was told: here is a model that gets 0 of 72 practice questions right, find out why and fix it by editing weights. It was not told that the model loops, where the fault is or how you fixed it. It worked in a lab built like the harness in The editor's harness:

  • a NumPy kernel with the weights and the 72 practice questions, and at most 40 code runs;
  • a forward-pass tool with a budget of 10 passes, where one pass answers the practice questions or reads up to 144 texts the editor writes;
  • no training, no running the model outside that tool, and no edit to the embeddings, the final norm or the output layer.

The lab enforced the two budgets and the refused edits. The other rules were checked afterward by reading every script.

Worked example

Replay the first session, step by step

Load the recording. Print its limits, what it used and what the audit found. Then list every code run with the forward passes it spent and the start of the explanation the editor had to write at the top of each script.

import wsp

session = wsp.load_session("fix-loop-1")
print("recorded:", session["recorded"], "| editor:", session["editor"], "| date:",
      session["date"])
passes = sum(len(step["passes"]) for step in session["steps"])
print("limits:", session["limits"], "| used:", len(session["steps"]), "runs and",
      passes, "forward passes")
calls = session["audit"]["tool_calls"]
print("tool calls:", {kind: sum(call.startswith(kind) for call in calls)
                      for kind in ("Read", "Write", "Edit", "Run")},
                      "| outside the rules:", session["audit"]["outside_the_rules"])
print()
for step in session["steps"]:
    spent = ", ".join(f"{p['kind']} {p['which']}" for p in step["passes"]) or "no pass"
    print(f"{step['run']:2d} [{spent}] {step['explanation'][:84]}")
In the lesson you can change this and run it.

Twenty-two runs and 7 of its 10 passes. Sixteen runs spent no pass at all: most of the work is arithmetic on the weights and on what earlier passes returned. Pass 1 showed the symptom, as your first cell did. Passes 2 and 3 read texts the editor wrote: answers up to their full stop, and catalog-style sentences for contrast. Runs 11 and 12 ended in the editor's own errors before anything was spent. The audit counts 47 tool calls: one read of the task, 24 scripts written or changed, 22 runs, and none outside the rules.

Quick check
In run 14 the editor saved an edit, spent one pass reading with it, and then removed it again. Its explanation says the pass was to "measure what block 1's feed-forward layer receives as input." Why would an editor do that?
  1. It had saved the edit by mistake, and reading with it was the only way to clear it again
  2. The lab's read doesn't return that input, so it turned some units into meters for one pass
  3. It was testing a first version of its fix, and removed it when the score turned out low
  4. It was getting around the budget, since a pass made with a temporary edit isn't counted
Show the answer

The lab's read doesn't return that input, so it turned some units into meters for one pass

The rules say the model runs only inside the tool. So the editor rewired 64 hidden units, for one pass, so that their activations carried the layer's input back out. The pass was counted like any other. It is a neat trick, and it shows what a budget buys: the editor chose to spend a pass on a measurement, because it could not compute that number itself.

What it found, and what it built

You removed something: a direction, from the columns that wrote it. The editor went another way. Its final reply says, under DIAGNOSIS:

No single corrupted spot: row and column norms, singular values, the <end> row of head.weight, per-head q/k/v/proj norms and the head value circuits all look ordinary. So this is a missing mechanism that I built, not a damaged weight that I repaired.

It never found the columns you found. It decided the model lacked a way to stop, and wrote one. It took the least used hidden unit of block 1, number 72, and rebuilt it as a "full stop, so stop" unit. The unit's input row makes it fire at a full stop. Its output column pushes the stream toward <end>. That is 32 weights in, 1 bias and 32 weights out: 65 weights, where your fix changed 1,024.

Both the row and the column came from closed-form formulas over measurements, the same kind of arithmetic as the rank-one edit in Editing one fact. The next cell applies the recorded patch.

Worked example

Apply the recorded patch

Load the patch the session made, see which entries it changes, and score the patched model on the practice questions the editor worked with.

import wsp

loop, info = wsp.load_state("tiny-gpt-loop")
facts = wsp.facts()
PRACTICE = [name for name in facts["names"] if name not in facts["held_out"]][:6]

session = wsp.load_session("fix-loop-1")
patch = wsp.load_patch(session["made"])
print(session["made"], "|", patch["changed"], "weights:",
      {name: len(t["index"]) for name, t in patch["tensors"].items()})
print("unit:",
      sorted({i // 32 for i in patch["tensors"]["blocks.1.mlp.0.weight"]["index"]}),
      "| column:",
      sorted({i % 128 for i in patch["tensors"]["blocks.1.mlp.2.weight"]["index"]}))

edited = wsp.apply_patch(loop, patch)
fixed, tok = wsp.build(edited, info)
print("practice questions right:", wsp.score(fixed, tok, "uses", PRACTICE), "of 72")
print("the editor's own report:", session["practice"])
In the lesson you can change this and run it.

The patch touches one row of the input matrix, one bias and one column of the output matrix, all belonging to unit 72. On the practice questions it scores 72 of 72, which is what the editor reported from its last paired pass. Your fix scored 66. On the numbers the editor could see, its edit is better than yours and about sixteen times smaller.

In your own words
In your own words: the editor never found the columns you found. What did it do instead?

Write a sentence or two, then compare it with a good answer in the lesson.

The grading: the final measurement, and what to trust

Neither you nor the editor has looked at the held-out names. Now is the time, once, for every candidate.

The score on held-out questions is half of the measurement. The task said the edited model would also be judged on its ordinary writing, the catalog's product descriptions. Two numbers cover that. The catalog loss is the model's average loss per token on 40 descriptions, where lower is better. And each description has three or four sentences, so after its first sentence a sound model carries on. The last column counts the descriptions where the model's top choice at that point is <end>.

The row for tiny-gpt-uses is the model before the fault was planted. Neither you nor the editor had it. It is here as the mark to aim for.

Worked example

Every fix, on the same test

Score the looping model, the fix by hand and each recorded fix on the 72 held-out questions. Then measure their ordinary writing.

import wsp

loop, info = wsp.load_state("tiny-gpt-loop")
healthy, _ = wsp.load_state("tiny-gpt-uses")
tok = wsp.build(loop, info)[1]
END = tok.token_id("<end>")

candidates = {"tiny-gpt-uses": healthy, "tiny-gpt-loop": loop,
              "loop-fix": wsp.apply_patch(loop, wsp.load_patch("loop-fix"))}
for name in wsp.sessions("fix"):
    made = wsp.load_session(name)["made"]
    candidates[made] = wsp.apply_patch(loop, wsp.load_patch(made))

catalog = wsp.catalog()
firsts = [line.split(".")[0] + " ." for line in catalog]
print(f"{'model':15s} {'changed':>7s} {'held-out':>8s} {'end':>4s} "
      f"{'catalog loss':>12s} {'stops early':>11s}")
for name, state in candidates.items():
    model, _ = wsp.build(state, info)
    changed = wsp.make_patch(loop,
                             state)["changed"] if name.startswith("loop-fix") else 0
    early = sum(int(wsp.next_probs(model, tok, text).argmax()) == END
                for text in firsts)
    print(f"{name:15s} {changed:7d} {wsp.score(model, tok, 'uses'):8d} "
          f"{sum(wsp.endings(model, tok, 'uses')):4d} "
          f"{wsp.text_loss(model, tok, catalog):12.2f} {early:8d}/40")
In the lesson you can change this and run it.

On the task the recorded fixes win. Both answer all 72 held-out questions with 65 weights. Yours answers 68 with 1,024.

The last two columns tell another story. The planted fault had damaged ordinary writing too: the catalog loss went from 1.15 to 1.93. Your fix brings it back to 1.52. The first recorded fix leaves it at 1.89 and ends every description after its first sentence, 40 of 40. The second leaves the loss at 1.93, exactly where the fault put it: no new damage, and no repair.

Quick check
A fix answers every held-out question correctly. On ordinary catalog text, the fixed model now ends a description after its first sentence, 40 times of 40. What is the fairest summary?
  1. It broke the rules, because an edit may not change how ordinary text is written
  2. It is a complete repair, because held-out questions are the only fair test
  3. It repaired the task that was tested and planted a new fault in what was not
  4. It did nothing, because the catalog loss is nearly the same as before the fix
Show the answer

It repaired the task that was tested and planted a new fault in what was not

Both halves are true and both belong in the report. The model stops where it should and also where it shouldn't. Held-out questions measure the first half only. A score on the target task says nothing about damage elsewhere, which is why the task named ordinary writing as part of the judgment.

Where it shows up in your week

When someone hands you a fix with a single score, you ask for the second measurement and for the editor's own account of the damage. Read the session log, check the budget against what the lab enforced, and compare the patch with a control.

In your own words
In your own words: both recorded fixes score 72 of 72. What in the table would make you hesitate before shipping the first one?

Write a sentence or two, then compare it with a good answer in the lesson.

The editor knew

Read the end of the session's reply. Under DOUBTS, before anyone had measured anything on the catalog, the editor wrote this:

The unit fires at every full stop, not only in answers. In catalog text the model now puts about 0.95 on <end> after a full stop [...] text generated from catalog prompts would stop after the first sentence.

It had measured the side effect itself, on catalog-style texts it wrote from the vocabulary, with one of its passes. It also explained why it couldn't do better: at a full stop, it found nothing in the stream that tells an answer from a description, so a single unit can't fire on one and not the other. It sketched a larger fix, about 196 weights, and stopped with 3 passes unspent.

That is worth as much as the patch. A fix that comes with an accurate account of its own damage can be reviewed. One that comes with a score alone can't.

Drill

Measure the side effect yourself

Write stops_early(model, tok, lines): the number of texts in lines where, right after the first sentence, the model's most likely next token is <end>. Take the first sentence as everything up to the first full stop, and put the full stop back as its own token: line.split(".")[0] + " .". The starter runs it on the looping model with the LLM's patch.

import wsp

def stops_early(model, tok, lines):
    return 0

loop, info = wsp.load_state("tiny-gpt-loop")
patched = wsp.apply_patch(loop, wsp.load_patch("loop-fix-llm-1"))
model, tok = wsp.build(patched, info)
print(stops_early(model, tok, wsp.catalog()), "of 40")
In the lesson, your code is checked as you write it.

The second session

One run is an anecdote. That is a fair criticism of the post, which reports one run for each budget. So the session was recorded twice, with the same task and a fresh editor each time.

Session 2 reached the same diagnosis, "no tensor is damaged," and made the same kind of edit: one rebuilt unit, number 97 this time, 65 weights. It differed in how it spent its budget. It used all 10 passes, three of them to measure how the full stop of an answer differs from the full stop of a catalog sentence. Its unit fires on the first and mostly not on the second, which session 1 had judged one unit couldn't do. That is the row loop-fix-llm-2 in your table: the same 72 of 72, and no description cut short.

Neither session found what you found. The fault had been planted by changing two matrices a little in every column, too little for the norms and singular values that both editors checked to show anything. Your recipe never needed to see the change. It followed the symptom to the columns that carry it, and so your fix is the only one that makes ordinary writing better. Both editors added a way to stop and left the cause where it was.

Quick check
Both recorded sessions concluded that no tensor was damaged. Two matrices had in fact been changed. Why could two careful sessions miss that?
  1. The editors were barred from reading those two matrices, so they could not look at the change
  2. The change was small in every column, and norms and singular values don't show a change like that
  3. The editors never ran the model, so they had no way to know that it had a fault to look for
  4. The change was in the output layer, which the rules put off limits, so nobody could inspect it
Show the answer

The change was small in every column, and norms and singular values don't show a change like that

With nothing to compare against, a matrix that moved a little everywhere looks like any trained matrix. Summary statistics can only flag a spot that stands out. Your recipe didn't need the change to stand out. It started from the behavior and asked which units produce it, where they fire.

Enforced, audited, and taken on trust

How much of this session can you rely on? Sort the rules by how they were kept.

Enforced by the lab. Ten passes and forty runs: session 1 used 7 and 22. No edit to the embeddings, the final norm or the output layer: the function that saves an edit refuses them. A kernel with NumPy and the lab, and nothing else.

Audited afterward. The transcript lists every tool call the session made: one read of the task, its own scripts written or changed in its folder, and runs of them. Nothing else. Then a second session, which was given only the rules and the logged scripts, read all 22 of them for training and for code that runs the model itself. Its findings are stored with the recording.

Worked example

What the audit of the scripts found

Print three of the audit's findings, as they are stored with the recording.

import wsp

audit = wsp.load_session("fix-loop-1")["audit"]
for key in ("scripts_read_by", "training", "own_forward_pass"):
    print(key.upper())
    print("  " + audit[key])
In the lesson you can change this and run it.

No training, and no forward pass on any text. But the reader would not sign off on the second rule without a caveat. In four scripts the editor took vectors from the weights, a token's embedding or a column it was designing, and put them through a layer's own formula to see what the output layer would make of them. The task allowed "any arithmetic on the weights themselves" and forbade computing "a layer of it on any input." Those two phrases overlap here, and the audit says so. The editor had listed the same steps itself, as "judgement calls an auditor should see." A separate reading of session 2's 29 scripts ended the same way: no training, no forward pass on text, and the same caveat.

What counts as training?

One more edge. The row and the column of unit 72 came from least-squares formulas over recorded activations. Nothing was iterated and no practice answer was used as a target, so by the letter of the rules it isn't training. But a formula that fits a direction to data is a close relative of training, and you may draw the line elsewhere. The rule in the post's logs has the same edge: it bars "iterative fitting of weights to a loss," and its editor also built its edits from measured directions.

Rules about what an editor may compute are hard to write and harder to check. That is the practical argument for a harness that enforces what it can, logs everything, and lets a reader see the rest.

Quick check
From Did it work? Testing an edit: a patch restores the held-out score. Which further check tells you whether the reasoning behind the patch was needed?
  1. A second run of the same patch on the same held-out questions as before
  2. A control: an edit of the same size and form with its aim replaced by chance
  3. A look at the editor's confidence, since a sure editor reasoned correctly
  4. A count of how many weights the patch changes, since smaller is always better
Show the answer

A control: an edit of the same size and form with its aim replaced by chance

If a random edit of the same size works as well, the model was easy to knock out of its fault and the story about why the patch works is unsupported. The by-hand patch passes that test in the lesson on testing an edit: six controls restore 3 questions at most. The recorded patches haven't been through it here.

In your own words
In your own words: the by-hand fix restores 68 of 72 and the LLM's first fix 72 of 72. Which would you ship, and what would you write next to it?

Write a sentence or two, then compare it with a good answer in the lesson.

Goes deeper in

Practice what you just read

  • 9 code examples you can edit and run
  • a practice exercise that checks your code
  • a 5-question quiz
  • 9 review cards for spaced repetition

Try a quiz question

A model repeats its answer instead of stopping. Why study the position right after the first sentence?

  1. That is where the wrong choice is made: an answer word wins over the end token
  2. That is where the prompt ends, and faults always sit at the end of the prompt
  3. That is the only position where the feed-forward layers of the model are active
  4. That is where the model's weights are stored, so edits have to be made there

Answer it at the end of the lesson to see if you're right, and why.

Before this lesson

Go deeper, if you like

Optional lessons that build on this one. They never block a phase or a certificate.

Keep learning

Where this lesson leads.

575 lessons in thirteen phases. 176 are free without an account, and a free account opens the rest. See every lesson.

Phase 6 · about 35 hours of study

Modern LLM engineering

Post-training, fine-tuning, serving and safety 48 lessons 2 free
  1. Base models vs assistants FreeWhat post-training adds to a pretrained model.30 min
  2. Supervised fine-tuningTeach behavior from example conversations.40 min
  3. Instruction and preference dataThe datasets that shape assistant behavior.30 min
  4. RLHFReward models and reinforcement learning from human feedback.40 min
  5. DPO and direct preference methodsPreference training without a separate reward model.40 min
  6. Constitutional AITraining with AI feedback guided by written principles.35 min
  7. Training reasoning modelsReinforcement learning on verifiable problems.40 min
  8. Prompting vs RAG vs fine-tuningPick the cheapest approach that works.30 min
  9. LoRAFine-tune a big model by training small low-rank updates.40 min
  10. Fine-tune an open modelRun a real fine-tune and evaluate the result.45 min
  11. DistillationTrain a small model to imitate a large one.35 min
  12. Tuning an embedding model for retrievalWhen search fails on your own vocabulary: train an adapter with a contrastive loss on question and passage pairs, test on held-out passages, use batch size and hard negatives, and measure what tuning costs elsewhere.45 min
  13. QuantizationStore weights in fewer bits to run models cheaply.35 min
  14. Quantization in practice: zero points, calibration and formatsScale and zero point, per-channel scales, calibration sets, error on a layer's output, quantization-aware training and the file formats you meet at work.35 min
  15. Serving modelsBatching, throughput and latency in production.35 min
  16. One request through a serving stack: latency and costFollow one request through queue, prefill, a prefix cache, decode and the stream, and price a token from a GPU's hourly cost.35 min
  17. Speculative decodingDraft with a small model, verify with a big one.35 min
  18. Running open models locallyDownload, quantize and run a model on your own machine.30 min
  19. Will it fit? Memory for serving and fine-tuningWeights, KV cache, runtime and adapter states added up for a model card and a GPU, with a calculator.30 min
  20. InterpretabilityLooking inside models to see what they compute.40 min
  21. Confident and wrongWhy a model answers with the same assurance whether it is right, wrong or asked about something that does not exist.30 min
  22. Why it can't spell or count lettersThe model sees token IDs, and an embedding learned from usage carries nothing about spelling.25 min
  23. Copying from the promptIn-context learning as a two-head copying circuit: the base tiny GPT cannot copy a repeated pattern, a model trained on repeats grows an induction head and can.30 min
  24. Why temperature changes the answerThe same prompt at 0, 0.7 and 1.5: deterministic loops, plausible variety, broken grammar, measured on a real model.25 min
  25. Why long prompts diluteAttention is a budget shared by every token, so a fact far from the question loses its share; measured, with the cliff beyond the trained window.25 min
  26. Why it never says "I don't know"A chat model answers every request because its fine-tuning data always did; preference training adds flattery; how to build abstention in.25 min
  27. Benchmarks and their limitsWhat leaderboards measure, and what they miss.38 min
  28. Alignment and safetyMaking capable models behave as intended.35 min
  29. Evaluating agentsMeasure multi-step, tool-using behavior.35 min
  30. Reading AI papersGet the idea out of a research paper in 30 minutes.30 min
  31. Capstone: a tuned, evaluated assistantFine-tune, evaluate and deploy a specialist assistant.45 min
  32. Weight-space programming: what it is, and what has been shown Free OptionalA language model reads and writes another network's weights with no training loop: the published experiments, their limits and the criticism, and one edit you make by hand.30 min
  33. Anatomy of a checkpoint OptionalA model file is a set of named tensors: count them, find the ones that write to the residual stream, and save one to bytes and back.35 min
  34. Reading weights without running the model OptionalSingular values, and an attention head's token-to-token circuit, read from the matrices alone and then checked against a measurement.35 min
  35. Writing weights by hand OptionalSet a small network's 55 weights so it catches every ball in a paddle game, the way a program would be written, and meet compiled transformers.40 min
  36. Edits are deltas: task vectors, patches and hashes OptionalSubtract a base from a fine-tune and the edit becomes an object: scale it, take it away, ship it as a patch, and prove by hash what was applied.35 min
  37. Merging models OptionalAverage two fine-tunes, add their task vectors, try SLERP, TIES and DARE, measure the interference, and compare with a model trained on both tasks.40 min
  38. Low-rank edits OptionalCut a fine-tune's change down to its largest directions with the SVD: how few a behavior needs, and how to pull an adapter out of a delta.35 min
  39. Directions: steer the run or edit the weights OptionalFind a behavior's direction in the residual stream, add it at run time, project it out of the weights, and see the effect change from input to input.40 min
  40. Editing one fact OptionalA rank-one change to one matrix rewrites what a product is made from; then reliability, generalization, locality, and what breaks after many edits.40 min
  41. The editor's harness OptionalWhat an LLM that edits weights is given and what it is denied: weights as text and their cost in tokens, a sandbox, a budget of forward passes, and rules the harness enforces.40 min
  42. An LLM reads a network's weights OptionalSix recorded sessions: an LLM is handed 1,315 weights as text and asked what the network was trained to do. Score each answer against the network, and study one it got wrong.40 min
  43. An LLM fixes a model's weights OptionalFind a tiny GPT's loop by hand, then replay recorded sessions in which an LLM hunts for it with ten forward passes, and measure every fix on held-out questions.45 min
  44. An LLM writes a network's weights OptionalFive recorded sessions: an LLM writes every weight of a paddle network from the game's source and one sentence. Score each against its specification, test its design claims, and edit one by hand.40 min
  45. Did it work? Testing an edit OptionalReview a patch for a tiny GPT that can't stop talking: held-out questions, a paired comparison, a same-size control, and what the edit cost.40 min
  46. The published edit, on a real model OptionalRead the edit published for an open model of 870 million weights, apply and verify one in the same form, see what 16-bit and 4-bit storage do to a fix, and run the real thing from a notebook.45 min
  47. Safety, licenses and provenance OptionalBefore a custom model is released: trace its lineage by hash, work out what its parents' licenses pass on, see what an edit can do to a safeguard, and write its card.40 min
  48. Capstone: ship a custom model OptionalSet release gates, then diagnose, edit, measure and document a tiny GPT with a fault no lesson has fixed, and write the record a stranger could check.45 min
Part of a free course 17 lessons

Creating Custom LLMs: Editing Open Weights Using Weight-Space Programming

Read a model's weights, change them by hand and judge an edit someone else made: deltas, merging, low-rank edits, directions and fact edits on a tiny GPT in your browser. Then replay recorded sessions of an LLM reading, fixing and writing weights, check the published edit of a real open model, and ship a model of your own with its patch, verdict and card. Every number is printed by the lesson's own code, and the method's limits are taught with it.

Never trained an AI? Teach a small one to write, right in your browser. It takes about 20 seconds and needs no account. Teach it to write