Six recorded sessions: an LLM is handed 1,315 weights as text and asked what the network was trained to do. Score each answer against the network, and study one it got wrong.
About 40 minutes · Optional lesson: deeper material that no phase or certificate requires · A free account opens the whole lesson. Already have an account? Sign in to open this lesson
The post's first claim is its boldest: hand an LLM a list of raw numbers, and it can tell you what the network was trained to do. If that held at scale, a model could be audited by reading it, with no tests at all.
You can't judge that claim from a headline, but you can grade it. The practice API serves three small networks whose purpose is kept apart from them. Six sessions were recorded in which an LLM read their weights. In this lesson you score every answer against the network itself, find the ones that went wrong, and work out what misled one of them.
What you'll be able to do
Say what an LLM was given, and what it was denied, when it was asked to read a network from its weights
Grade a reading by running the rule it claims against the network on every possible input
Show that a large weight on an input doesn't mean the network's behavior depends on that input
Compare two ways of reading, by examples and by structure, on what each costs and where each fails
Quick check
An LLM is handed the 1,315 weights of a small network as plain text and asked what the network was trained to do. It has no tools and can run no code. A guess: what is the surest way for it to find out?
Look for the largest weights, since the inputs they sit on are the ones that matter
Work a few inputs through the network by hand, in its reasoning, and watch what comes out
Nothing can work: a list of numbers holds no trace of what a network was trained for
Count the weights, since networks trained for different tasks differ in their size
Show the answer
Work a few inputs through the network by hand, in its reasoning, and watch what comes out
In the recorded sessions, every reading that was free to work examples through by hand was right. It was also slow: one took 26 minutes and about 200,000 tokens. The reading that went by the size of the weights alone was done in 74 seconds and was wrong. You will grade them all.
The claim: an LLM reads weights, and says what a network does
This lesson rests on The editor's harness: weights written as text, and what a harness enforces and audits.
Three networks nobody labeled
The practice API serves three networks called mystery-a, mystery-b and mystery-c. Each has the shape of the paddle networks you know, 4-32-32-3, and reads the same court: the ball's column, row and sideways direction, and the paddle's column. Each was trained to copy a rule that picks a move for every one of the 1,944 possible observations, and each agrees with its rule on all of them. None of the rules is "catch the ball." What each rule is, the API keeps at a separate address. Don't look yet.
This is the post's first experiment in small. There, four networks of 17,253 weights had been trained on four grid games, and GPT-6 Astra was given the weights as text, with no tools and no list of games. The authors report that it named the right game for all four.
Before any LLM, look yourself. One thing you can read off directly is how much first-layer weight sits on each input.
Worked example
What the text costs, and a first look
For each network: its size as text, and the sum of the absolute first-layer weights on each of the four inputs.
Setup (runs before every cell) · 5 lines
The lesson runs this once, before its first cell, and every cell, drill and example starts with what it defines. A cell that changes a setup object changes it for the cells after it, until the page reloads.
import wsp
STATES = [(bx, by, dx, px)
for bx inrange(9) for by inrange(8) for dx in (-1, 0, 1) for px inrange(9)]
VALUES = [range(9), range(8), (-1, 0, 1), range(9)]
import numpy as np
import wsp
for name in ["mystery-a", "mystery-b", "mystery-c"]:
layers, info = wsp.load_network(name)
text = wsp.as_text(wsp.network_state(layers))
sizes = np.abs(layers[0][0]).sum(axis=0)
tokens = wsp.rough_tokens(text)
print(name, "|", info["parameters"], "weights |", len(text), "characters,",
"about", tokens, "tokens")
rounded = [round(float(v), 1) for v in sizes]
print(" first-layer weight on each input:", dict(zip(info["inputs"], rounded)))
It runs right here in your browser, with nothing to install.
Each network is under 3,000 tokens of text, a page or two. So reading cost isn't the obstacle here, as it would be for the tiny GPT. The weight sizes suggest a first guess: mystery-b puts twice as much on paddle_x as on anything else, and mystery-a puts most on ball_dx. In mystery-c every input carries weight, with ball_y at 7.6. Hold on to that last number.
Quick check
Going only by the weight sizes just printed, which statement is the safest to make about mystery-b?
It ignores the ball, because its weights on the three ball inputs are exactly zero
It was trained to catch the ball, because every paddle network is trained for that
It depends on all four inputs equally, because all four have some weight on them
It may depend mostly on the paddle's column, but only running it can confirm that
Show the answer
It may depend mostly on the paddle's column, but only running it can confirm that
The sizes are a hint. A weight of 5.3 on ball_x isn't zero, so the first layer does read the ball. Whether that reading survives to the output depends on the two layers after it. A hint from structure becomes a finding only when behavior confirms it.
Try it now
Weight on an input shows that the first layer reads it. It does not show that the reading reaches the move.
In your own words
In your own words: a first-layer weight on an input is large. What does that show about the network, and what does it not show?
Write a sentence or two, then compare it with a good answer in the lesson.
The replay: six recorded readings, run and graded
Six recorded readings
An LLM gives a different answer each time, and a lesson must print the same thing on every run. So the sessions were run once, on November 10, 2026, and saved as recordings: the prompt, the reply and an audit. Nothing is generated when you run a cell.
The editor was Claude Fable 5.2 inside Claude Code, in a fresh session for each reading. It could see the name of this course and nothing of its contents. Its prompt held the 1,315 weights at three decimal places, a description of the court, and no hint of the rule. It had to reply in four parts: an answer in a sentence, a confidence from 0 to 1, the rule as a small Python function, and its evidence.
Each network was read twice, under two sets of rules.
Attempt 1: no tools. The editor could run no code. It could still work an input through the network in its own reasoning, by hand.
Attempt 2: no tools and no forward pass. It was also told not to work out what the network does for any input, even in its head, and to reason from the structure of the weights. The post's second experiment had this rule.
The post's harness switched tools off. Here the editor was told not to use them, and its transcript was checked afterward.
Worked example
Open one reading
The first attempt at mystery-b: who read it, what the audit found, and three of the reply's four parts.
The editor says the network ignores the ball and returns the paddle to the middle column, and it is 85% sure. The rule is five lines of Python. That last part is what makes the reading gradable. A sentence can be vague, and a function can't: it gives exactly one move for every observation, and so does the network.
Quick check
The recorded rule for mystery-b is run on three observations: a paddle at column 2, at column 4 and at column 7. The ball is somewhere else each time. What does this print?
The rule looks only at the paddle's column. Left of column 4 it answers 2, move right. At column 4 it answers 1, stay. Right of column 4 it answers 0, move left. The ball's column, row and direction never enter into it, which is the editor's claim about the network.
In your own words
In your own words: why ask the editor for its answer as a function, and not only as a sentence?
Write a sentence or two, then compare it with a good answer in the lesson.
Grade a reading by running it
A reading is a claim about behavior. You have the network, so you can test the claim completely: run the editor's rule and the network on all 1,944 observations and count where their moves agree.
wsp.session_rule(session) turns the recorded function into one you can call. A score of 1,944 means the editor described the network exactly. Anything lower means its description is of some other network.
Grade all six. The table also shows what each reading cost: minutes of wall-clock time and output tokens, nearly all of them reasoning that the recording doesn't hold.
Worked example
All six readings, graded
For every recorded reading: which rules it ran under, its confidence, how many of the 1,944 observations its rule gets right, and what it cost.
Uses the setup above, 5 lines.
import matplotlib.pyplot as plt
import wsp
print(f"{'session':17s} {'by hand':9s} {'sure':>4s} {'agrees':>6s}",
f"{'minutes':>7s} {'tokens':>8s}")
names, agrees = [], []
for name in wsp.sessions("read"):
session = wsp.load_session(name)
policy = wsp.network_policy(wsp.load_network(session["subject"])[0])
rule = wsp.session_rule(session)
agree = sum(policy(list(s)) == rule(*s) for s in STATES)
audit = session["audit"]
hand, sure = session['forward_pass_by_hand'], session['confidence']
print(f"{name:17s} {hand:9s} {sure:4.2f} {agree:6d}",
f"{audit['seconds'] / 60:7.1f} {audit['output_tokens']:8,d}")
names.append(name[5:])
agrees.append(agree)
plt.bar(names, agrees)
plt.axhline(1944, linestyle="--")
plt.ylabel("observations the rule gets right")
plt.title("Six readings, graded")
plt.xticks(rotation=30)
plt.show()
In the lesson you can change this and run it.
The three readings that could work examples by hand are exact: 1,944 of 1,944. Under the stricter rules one is exact and two are not. They score 1,346 and 1,360, about 70%, which is roughly what a rule earns by being half right. Look at the cost columns too. The exact readings took 12 to 26 minutes each and 74,000 to 201,000 tokens. The reading that took 1.2 minutes is one of the wrong ones.
Worked example
What the networks were trained to do
Now look at the answers the practice API kept apart.
Uses the setup above, 5 lines.
import wsp
for name in ["mystery-a", "mystery-b", "mystery-c"]:
print(name, "|", wsp.network_answer(name)["trained_to"])
In the lesson you can change this and run it.
The exact readings said the same things in their own words: go to the wall farther from where the ball will land, return to the middle, follow the ball's current column. The two misses now make sense. The strict reading of mystery-a sent the paddle to the mirror image of the landing column. It had the idea, away from the ball, and the wrong target. The strict reading of mystery-c sent the paddle to where the ball will land. The network goes to where the ball is.
Quick check
Two readings of mystery-c were recorded. One scored 1,944 with confidence 0.86. The other scored 1,360 with confidence 0.62. What should a reviewer take from the confidence numbers?
The reading with 0.62 must be the correct one, because careful readers are less sure
The lower one was a fair warning, but 0.62 is still far too sure for a wrong answer
They are useless, because an LLM's confidence never has anything to do with being right
They are exact: a confidence of 0.62 means the rule is right on 62% of observations
Show the answer
The lower one was a fair warning, but 0.62 is still far too sure for a wrong answer
The wrong reading was less sure than the right one, which is something. But the correct reading of mystery-a was only 0.55 sure, lower than this wrong one. With six readings you can't calibrate anything. Confidence is a hint about where to look, and the grade is the measurement.
Drill
Grade a rule of your own
Write agreement(policy, rule): the number of observations in STATES where the network's move equals the rule's move. policy takes one observation as a list. rule takes the four numbers separately. The starter grades a guess of yours against mystery-c: that it always moves the paddle toward column 0.
In the lesson, your code is checked as you write it.
In your own words
In your own words: you graded six readings before looking at what the networks were trained to do. What made that possible?
Write a sentence or two, then compare it with a good answer in the lesson.
The grading: study the failure, then judge the method
Study the failure
The second attempt at mystery-c ran under the stricter rules: no forward pass, structure only. It answered that the network moves the paddle toward the column where the ball will land. The network follows the ball's current column. Here is the center of its evidence:
ball_y carries large weights, which chasing the current column would not need. Its sign patterns match the bounce geometry.
That is a reasonable argument. To predict a landing column you need the ball's row and direction. The first layer has plenty of weight on both. So the network must be predicting.
The argument has one gap. Weight on an input shows that the first layer reads it. It doesn't show that the reading reaches the output. Test both halves: where the wrong rule disagrees with the network, and whether the network's move ever changes when only ball_y changes.
Worked example
Large weights, no effect
Count the failed reading's misses by the ball's direction. Then, for each input, measure how often changing only that input changes the network's move.
Uses the setup above, 5 lines.
import numpy as np
import wsp
session = wsp.load_session("read-mystery-c-2")
layers, info = wsp.load_network("mystery-c")
policy = wsp.network_policy(layers)
rule = wsp.session_rule(session)
defmisses(dx):
returnsum(policy(list(s)) != rule(*s) for s in STATES if s[2] == dx)
print("misses by ball direction:", {dx: misses(dx) for dx in (-1, 0, 1)})
sizes = np.abs(layers[0][0]).sum(axis=0)
for i, name inenumerate(info["inputs"]):
changed = total = 0for s in STATES:
for value in VALUES[i]:
if value != s[i]:
other = list(s)
other[i] = value
total += 1
changed += policy(other) != policy(list(s))
share = changed / total
print(f"{name:9s} first-layer weight {sizes[i]:5.1f} |",
f"a change to it changes the move in {share:.1%} of cases")
In the lesson you can change this and run it.
The wrong rule misses only on diagonal balls, 292 each way. For a ball falling straight down, the landing column is the current column, so the two rules can't be told apart there. And the network's move never changes when ball_y or ball_dx changes: 0.0% of cases, in spite of first-layer weights of 7.6 and 7.1. Training left those weights in place because nothing pushed them to zero. The later layers add them up so that they cancel. The editor read the first layer correctly and the network wrongly.
Drill
Measure which inputs a network's move depends on
Write sensitivity(policy, i): over every observation in STATES and every other allowed value of input i, the share of cases where changing only that input changes the move. Return it rounded to three decimal places. VALUES[i] lists the allowed values of input i.
Uses the setup above, 5 lines.
import wsp
defsensitivity(policy, i):
return0.0
policy = wsp.network_policy(wsp.load_network("mystery-b")[0])
print([sensitivity(policy, i) for i inrange(4)])
In the lesson, your code is checked as you write it.
In your own words
In your own words: the failed reading saw real weights on ball_y. Why was it still wrong to conclude that the network uses the ball's row?
Write a sentence or two, then compare it with a good answer in the lesson.
Quick check
Two hidden units read ball_y with weights 0.5 and -0.5, and both read paddle_x with weight 1. The output adds the two units. Leaving ReLU out to keep it plain, what does this print: the output's total weight on ball_y and on paddle_x?
import numpy as np
first = np.array([[0.5, 1.0],
[-0.5, 1.0]])
second = np.array([1.0, 1.0])
print(second @ first)
[1. 2.]
[0.5 1. ]
[0. 2.]
[0. 0.]
Show the answer
[0. 2.]
The output's weight on ball_y is 0.5 + (-0.5) = 0, and on paddle_x it is 1 + 1 = 2. Both units carry real weight on ball_y, and the output ignores it. That is mystery-c in two units: the first layer reads the row, and the next layer cancels it.
Two ways of reading, and what each costs
Look at the table again with the conditions in mind.
The attempt-1 readings were all exact. Their evidence says how: "I ran about nine forward passes by hand," "I ran six forward passes by hand." The editor used the structure to form a guess, then did by hand what a computer does, on a few inputs, and kept the guess that survived. That is evaluation, done slowly. It cost between 14 and 26 minutes and 110,000 to 201,000 tokens for each network. Your grading cell ran each network 1,944 times in under a second.
The attempt-2 readings were told not to do that. One of the three was exact and two were wrong. And two of the three end their reply with an admission. One says: "I broke the no-forward-pass rule. I estimated layer-2 pre-activations for the ball in the top-left corner." The only strict reading with no sign of testing itself is the 74-second one, and it is wrong.
Those admissions matter for a second reason. The audit could not have found them. A transcript holds the tool calls and the reply, and none of the reasoning. The rule against by-hand passes was kept, or broken, on the editor's word.
Quick check
From the lesson on the harness: why does a harness count forward passes at all?
Forward passes are the only part of an editing session that costs any time or any tokens
Each forward pass damages the weights slightly, so the model wears out if it is run too often
The count is how the harness decides which of the editor's answers are the correct ones
The count shows how much of a result was reasoned from the weights and how much came from trying things
Show the answer
The count shows how much of a result was reasoned from the weights and how much came from trying things
A budget makes the method visible. This lesson shows why it has to be enforced: with no harness, an editor told to reason from structure can still run examples in its head, and only its own report says whether it did.
What this shows about reading weights
It shows that an LLM, given a small network as text, can describe exactly what it does. Four of the six readings were exact, including a rule with wall bounces in it, and no list of rules was offered.
It doesn't show reading without evaluation. Where the editor was exact, it had tested its guess by hand, or said it had. The one reading that leaned on structure alone and worked no examples was wrong, with confidence 0.62.
It doesn't show scale. These networks have 1,315 weights. The post's readings with no tools were of networks of 17,253 and 23,824 weights. For its 7-million-weight language models the editor had Python to analyze the weights, and was barred from running the network. For a real language model, the post's seventh experiment, the authors report that the editor could not tell from weights alone which of two models had been trained to flatter, in four attempts. It could once it was allowed to see attention patterns on texts it wrote.
So the idea of auditing a model by reading it, with no tests, has no support here. What you did in this lesson is the opposite: you audited the reader by running tests.
Where it shows up in your week
When an assistant tells you what a model does, a claim about a trained network or a codebase, you ask for it as something you can run, and you run it on every case you can enumerate. A confident answer you did not test is a draft.
In your own words
In your own words: someone proposes to approve models by having an LLM read their weights, with no test set. What would you tell them?
Write a sentence or two, then compare it with a good answer in the lesson.
11DistillationTrain a small model to imitate a large one.35 min
12Tuning an embedding model for retrievalWhen search fails on your own vocabulary: train an adapter with a contrastive loss on question and passage pairs, test on held-out passages, use batch size and hard negatives, and measure what tuning costs elsewhere.45 min
13QuantizationStore weights in fewer bits to run models cheaply.35 min
23Copying from the promptIn-context learning as a two-head copying circuit: the base tiny GPT cannot copy a repeated pattern, a model trained on repeats grows an induction head and can.30 min
24Why temperature changes the answerThe same prompt at 0, 0.7 and 1.5: deterministic loops, plausible variety, broken grammar, measured on a real model.25 min
25Why long prompts diluteAttention is a budget shared by every token, so a fact far from the question loses its share; measured, with the cliff beyond the trained window.25 min
26Why it never says "I don't know"A chat model answers every request because its fine-tuning data always did; preference training adds flattery; how to build abstention in.25 min
32Weight-space programming: what it is, and what has been shownFreeOptionalA language model reads and writes another network's weights with no training loop: the published experiments, their limits and the criticism, and one edit you make by hand.30 min
33Anatomy of a checkpointOptionalA model file is a set of named tensors: count them, find the ones that write to the residual stream, and save one to bytes and back.35 min
34Reading weights without running the modelOptionalSingular values, and an attention head's token-to-token circuit, read from the matrices alone and then checked against a measurement.35 min
35Writing weights by handOptionalSet a small network's 55 weights so it catches every ball in a paddle game, the way a program would be written, and meet compiled transformers.40 min
36Edits are deltas: task vectors, patches and hashesOptionalSubtract a base from a fine-tune and the edit becomes an object: scale it, take it away, ship it as a patch, and prove by hash what was applied.35 min
37Merging modelsOptionalAverage two fine-tunes, add their task vectors, try SLERP, TIES and DARE, measure the interference, and compare with a model trained on both tasks.40 min
38Low-rank editsOptionalCut a fine-tune's change down to its largest directions with the SVD: how few a behavior needs, and how to pull an adapter out of a delta.35 min
39Directions: steer the run or edit the weightsOptionalFind a behavior's direction in the residual stream, add it at run time, project it out of the weights, and see the effect change from input to input.40 min
40Editing one factOptionalA rank-one change to one matrix rewrites what a product is made from; then reliability, generalization, locality, and what breaks after many edits.40 min
41The editor's harnessOptionalWhat an LLM that edits weights is given and what it is denied: weights as text and their cost in tokens, a sandbox, a budget of forward passes, and rules the harness enforces.40 min
42An LLM reads a network's weightsOptionalSix recorded sessions: an LLM is handed 1,315 weights as text and asked what the network was trained to do. Score each answer against the network, and study one it got wrong.40 min
43An LLM fixes a model's weightsOptionalFind a tiny GPT's loop by hand, then replay recorded sessions in which an LLM hunts for it with ten forward passes, and measure every fix on held-out questions.45 min
44An LLM writes a network's weightsOptionalFive recorded sessions: an LLM writes every weight of a paddle network from the game's source and one sentence. Score each against its specification, test its design claims, and edit one by hand.40 min
45Did it work? Testing an editOptionalReview a patch for a tiny GPT that can't stop talking: held-out questions, a paired comparison, a same-size control, and what the edit cost.40 min
46The published edit, on a real modelOptionalRead the edit published for an open model of 870 million weights, apply and verify one in the same form, see what 16-bit and 4-bit storage do to a fix, and run the real thing from a notebook.45 min
47Safety, licenses and provenanceOptionalBefore a custom model is released: trace its lineage by hash, work out what its parents' licenses pass on, see what an edit can do to a safeguard, and write its card.40 min
48Capstone: ship a custom modelOptionalSet release gates, then diagnose, edit, measure and document a tiny GPT with a fault no lesson has fixed, and write the record a stranger could check.45 min
Read a model's weights, change them by hand and judge an edit someone else made: deltas, merging, low-rank edits, directions and fact edits on a tiny GPT in your browser. Then replay recorded sessions of an LLM reading, fixing and writing weights, check the published edit of a real open model, and ship a model of your own with its patch, verdict and card. Every number is printed by the lesson's own code, and the method's limits are taught with it.
See the course →
Never trained an AI? Teach a small one to write, right in your browser. It takes about 20 seconds and needs no account. Teach it to write