Weight-space programming: what it is, and what has been shown
A language model reads and writes another network's weights with no training loop: the published experiments, their limits and the criticism, and one edit you make by hand.
A model's behavior lives in its weights, and until now almost every weight was set by training. In November 2026 four researchers at MIT asked a different question: can a language model read another network's weights as numbers, reason about them, and write better ones, with no training at all? They called it weight-space programming, and the results spread fast.
The idea is new and the evidence is thin, so this course treats it the way an engineer should. You will do every step by hand on networks small enough to read, then watch a model do it, then test whether it worked. This first lesson gives you the idea, the claims with their real numbers, and the criticism.
What you'll be able to do
Say what weight-space programming is, and how it differs from training a network
Read what a small network does from its weights, and repair a planted fault by changing one number
Sort the published claims into what was shown, on what size of network, and what was not
Report an edit honestly: how many weights changed, what share, and what was measured
Quick check
A small network has 17,253 weights. A language model is shown those numbers as text. It cannot run the network. A guess: what did researchers report it could do?
Nothing useful: weights are random-looking numbers with no readable meaning
Retrain the network to play a different game
Recite the exact data the network was trained on
Name the game the network was trained to play
Show the answer
Name the game the network was trained to play
That is the first experiment in the post this lesson is about: four small game-playing networks, and the model named each one's game from the weights alone. The rest of the lesson looks at how far that result reaches.
The idea: weights are the program
This lesson rests on A matrix times a vector: a matrix times a vector gives one score per row.
Six weights and three biases
A neural network is a fixed recipe of arithmetic with adjustable numbers in it. Those numbers are the weights. Change them and the same recipe computes something else, so the weights are where the behavior lives. In the usual workflow nobody writes them. A training loop shows the network examples and nudges every weight a little, millions of times, until the outputs are good.
Weight-space programming is the name for doing it another way. Something reads the weights as numbers, works out what they compute, and writes new values on purpose. The name comes from a blog post of November 2, 2026, "Sparks of Weight-Space Programming," by four researchers at MIT. In their experiments the reader and writer is a general-purpose language model, the kind you send prompts to. It gets the weights and a goal in plain words. It uses no training data, no loss and no gradient descent.
That can sound like magic, so start with a network small enough to read yourself. A paddle has to catch a falling ball. The network sees two numbers, the ball's column and the paddle's column, and gives a score to each of three moves: left, stay, right. The paddle makes the move with the highest score. The whole network is six weights and three extra numbers called biases.
Worked example
Read a policy out of six numbers
Each row of W belongs to one move. A row's score is its two weights times the two inputs, added up, plus that row's bias. The code prints the scores for three positions.
import numpy as np
MOVES = ["left", "stay", "right"]
W = np.array([[-1.0, 1.0], # left: paddle column minus ball column
[ 0.0, 0.0], # stay: ignores both inputs
[ 1.0, -1.0]]) # right: ball column minus paddle column
b = np.array([0.0, 0.5, 0.0]) # stay gets a small head startfor ball, paddle in [(6, 2), (1, 5), (4, 4)]:
scores = W @ np.array([ball, paddle]) + b
move = MOVES[int(scores.argmax())]
print(f"ball {ball}, paddle {paddle}: scores {scores} -> {move}")
It runs right here in your browser, with nothing to install.
With the ball at 6 and the paddle at 2, "right" scores 4 and wins. With the ball at 1 and the paddle at 5, "left" scores 4. When they share a column both differences are 0, so the bias of 0.5 lets "stay" win. You read all of that from the numbers: the "right" row has +1 on the ball and −1 on the paddle, so it scores how far the ball is to the paddle's right. Nothing was run to learn it. The run only confirmed the reading.
Quick check
Using the same weights, what does this print?
import numpy as np
W = np.array([[-1.0, 1.0], [0.0, 0.0], [1.0, -1.0]])
b = np.array([0.0, 0.5, 0.0])
scores = W @ np.array([3, 7]) + b
print(int(scores.argmax()))
4
2
0
1
Show the answer
0
The ball is in column 3 and the paddle in column 7. The "left" row gives −3 + 7 = 4, "stay" gives 0.5 and "right" gives 3 − 7 = −4. The largest score is at index 0, which is the move left.
Try it now
The policy's weight matrix as a picture: blue is positive, red is negative. Move over a cell, or use the Row and Column boxes, to read each weight. The "left" and "right" rows are mirror images, and the "stay" row is blank.
Drill
Repair a fault by changing one number
This copy of the policy has a planted fault: the paddle runs away from balls on its right. Read the matrix, find the one wrong weight, and change it. Don't retrain anything.
import numpy as np
MOVES = ["left", "stay", "right"]
W = np.array([[-1.0, 1.0],
[ 0.0, 0.0],
[-1.0, -1.0]]) # one of these six numbers is wrong
b = np.array([0.0, 0.5, 0.0])
defmove(ball, paddle):
return MOVES[int((W @ np.array([ball, paddle]) + b).argmax())]
print(move(6, 2), move(1, 5), move(4, 4))
In the lesson, your code is checked as you write it.
In your own words
In your own words: what is the difference between training a network and weight-space programming?
Write a sentence or two, then compare it with a good answer in the lesson.
In code: what the post reports, in numbers
You read and repaired a nine-number network. The post asks whether a language model can do the same on networks far too big for a person. It runs eight experiments around three questions, with one editor model throughout (OpenAI's GPT-6 Astra). Every figure below is the authors' own.
Read. Four small game-playing networks of 17,253 weights each were written out as text. With no tools, the model named the game each one plays. It did the same for three small image generators, naming the shape each draws. For three character-level language models of about 7 million weights, it could analyze the weights with Python but could not run the networks. It named the language each was trained on and found their copying heads. One reading failed: given two copies of a small open model, one trained to flatter the user, it could not tell them apart from weights alone in four attempts.
Quick check
From the multi-layer networks lesson: a layer computes W @ x + b. What does one row of W hold?
One weight from every layer of the network
The weights one output unit puts on each input
The inputs for one training example
The outputs of the previous layer
Show the answer
The weights one output unit puts on each input
Each row belongs to one output unit: its weights times the inputs, added up, plus its bias, give that unit's value. That is why you could read the paddle policy row by row, and why a network's first layer says which inputs it watches.
What it fixed, and what it wrote
Fix. A network that drives a car up a ramp failed in all 256 test runs. The model changed 3.5% of its 171,664 weights in one reply, and the car then succeeded in all 1,024 runs. The best-known result is on a real open language model, Qwen3.5-0.8B, which gets stuck in loops when it reasons. Working in a Python sandbox, the model traced the loops to one direction written by two layers and removed it. That changed 131,072 weights. The authors report its score on a math test rising from 7.9% to 48.1%.
Write. Starting from all zeros, the model wrote complete networks for three small games from their rules alone. One scored far above a trained network, one about the same and one below. It also wrote a network for a small Go board.
Worked example
How big were those edits, and those prompts?
Three numbers from the post, worked out: the share of weights each fix changed, and how many tokens it takes to show weights to a model as text. (A token is the unit a language model reads; a long number costs several.) The post gives the Qwen edit as 131,072 weights and 0.016% of the model. The total used below, 819,200,000, is the size those two figures imply for a model described as 0.8 billion.
car_weights, car_share = 171_664, 0.035print(f"car fix: about {car_weights * car_share:,.0f} of {car_weights:,} weights")
qwen_changed = 2 * 64 * 1024# 2 layers x 64 columns x 1,024 rows each
qwen_weights = 819_200_000# about 0.8 billion
share = qwen_changed / qwen_weights
print(f"Qwen fix: {qwen_changed:,} weights, {share:.3%} of the model")
tokens_per_weight = 73_700 / 17_253# the first experiment's promptprint(f"tokens per weight as text: {tokens_per_weight:.2f}")
for name, n in [("the course's tiny GPT", 40_195), ("Qwen3.5-0.8B", qwen_weights)]:
print(f"{name}: about {n * tokens_per_weight:,.0f} tokens to show every weight")
In the lesson you can change this and run it.
The car fix touched about 6,008 weights. The Qwen fix touched 131,072, which is 0.016% of the model: roughly one weight in every 6,250. The last lines explain why the big experiments needed a sandbox. Weights written as text cost about 4.27 tokens each, so the 17,253-weight networks fit in one prompt of about 73,700 tokens. This course's tiny GPT would need about 172,000. Qwen would need about 3.5 billion, thousands of times more than any model can read at once. For Qwen the editor never saw the weights as text. It wrote Python that examined them.
Quick check
In which experiments did the model work from the raw weights as text, with no tools and no way to run the network?
Only on small networks, from about 17,000 to 172,000 weights
In all eight, including the 0.8-billion-weight language model
Only on the language models, because text is what a language model reads best
In none: every experiment let the model run the network first
Show the answer
Only on small networks, from about 17,000 to 172,000 weights
The tool-free experiments were the game policies, the image generators, the car fix and the from-scratch game networks, all between about 17,000 and 172,000 weights. Every experiment on a language model gave the editor Python, and the Qwen fix also gave it a budget of test runs.
In your own words
The Qwen edit raised a math score from 7.9% to 48.1%. In your own words, why is "the edit taught the model math" the wrong reading?
Write a sentence or two, then compare it with a good answer in the lesson.
In production: how to hold a new claim
The post is eight days old as this lesson is written. It is a blog post, not a reviewed paper, and its authors call the results "very early." Others raised fair points.
Few runs. Most conditions were tried one to four times, with a single editor model. A result seen once can be luck.
The framing. One critic noted "raw weights only" applied just to the tiny networks. Everything on a real language model used code and test runs.
Could a person have done it? Another argued a skilled researcher would have made the Qwen fix, and compared editing weights to patching a compiled program.
Does it need a language model? A community tool built to repeat the work reported that a two-line script matched its language-model editors on one task.
Replication. One engineer rebuilt the edited Qwen model from the published logs and measured 51.0%. Another repeated the first experiment with an open model, which got two of four games clearly right.
None of this makes the work wrong. It tells you how much to trust it. For any new result, ask what was measured, on what size of thing, how many times, by whom, and what the same method failed to do.
The last question matters most here. An edit can remove a fault. It cannot supply what the network never learned. The cell below shows that difference on the paddle policy.
Quick check
A result was produced once, by one editor model, and measured by the people who built it. Which follow-up tells you the most?
Someone else rebuilding it from the published files and measuring it again
A larger headline number, taken from the same single run as before
The authors describing the result in more detail
More people sharing the post
Show the answer
Someone else rebuilding it from the published files and measuring it again
An independent rebuild checks the whole chain: the files, the edit and the measurement. For the Qwen fix one exists, and it measured 51.0% where the authors reported 48.1%. That one result now has two sources. The other seven experiments still have one.
Worked example
A fault you can remove, and a gap you can't fill
Two broken policies, each scored on all 81 positions. In the first, a single bias is far too large. In the second, the network has no weights on the ball's column at all. The code tries to repair each by changing the biases only.
import numpy as np
from itertools import product
GOOD_W = np.array([[-1.0, 1.0], [0.0, 0.0], [1.0, -1.0]])
GOOD_B = np.array([0.0, 0.5, 0.0])
defscore(W, b):
right = 0for ball, paddle in product(range(9), repeat=2):
want = 0if ball < paddle else2if ball > paddle else1
right += int((W @ np.array([ball, paddle]) + b).argmax()) == want
return right
faulty_b = np.array([0.0, 50.0, 0.0]) # stay always winsprint("fault planted:", score(GOOD_W, faulty_b), "of 81")
print("one bias changed back:", score(GOOD_W, GOOD_B), "of 81")
blind_W = GOOD_W.copy()
blind_W[:, 0] = 0.0# no weight reads the ball's column
grid = [-8.0, -4.0, 0.0, 0.5, 4.0, 8.0]
best = max(score(blind_W, np.array(b)) for b in product(grid, repeat=3))
print("blind network, best of 216 bias settings:", best, "of 81")
In the lesson you can change this and run it.
With the planted fault the policy is right only in the 9 positions where staying is correct. Changing one bias back restores all 81. The blind network is different. Its weights never look at the ball, so no choice of biases can make it follow one: the best of 216 settings gets 56 of 81. To fix it you would have to write the missing weights, which means supplying the skill, not removing a fault. That is the Qwen result and the second model's result in miniature.
2 × 64 × 1,024 is 131,072 weights. Divided by 819,200,000 and multiplied by 100, that is 0.016 percent.
Drill
Report an edit's size
Write edit_report(before, after) for two NumPy arrays of the same shape. Return a dictionary with "changed" (how many entries differ), "total" (how many entries there are) and "share" (changed divided by total). Every honest edit report starts with these three numbers.
In the lesson, your code is checked as you write it.
Where it shows up in your week
When a colleague sends you an edited model, ask for the edit's size and share, what was measured, on what, and how many times. The fault you hit most often is a result seen once, measured on the same items the edit was built from.
In your own words
You read that an edit to 0.016% of a model's weights changed a benchmark score a lot. Name three things you would want to know before trusting it.
Write a sentence or two, then compare it with a good answer in the lesson.
11DistillationTrain a small model to imitate a large one.35 min
12Tuning an embedding model for retrievalWhen search fails on your own vocabulary: train an adapter with a contrastive loss on question and passage pairs, test on held-out passages, use batch size and hard negatives, and measure what tuning costs elsewhere.45 min
13QuantizationStore weights in fewer bits to run models cheaply.35 min
23Copying from the promptIn-context learning as a two-head copying circuit: the base tiny GPT cannot copy a repeated pattern, a model trained on repeats grows an induction head and can.30 min
24Why temperature changes the answerThe same prompt at 0, 0.7 and 1.5: deterministic loops, plausible variety, broken grammar, measured on a real model.25 min
25Why long prompts diluteAttention is a budget shared by every token, so a fact far from the question loses its share; measured, with the cliff beyond the trained window.25 min
26Why it never says "I don't know"A chat model answers every request because its fine-tuning data always did; preference training adds flattery; how to build abstention in.25 min
32Weight-space programming: what it is, and what has been shownFreeOptionalA language model reads and writes another network's weights with no training loop: the published experiments, their limits and the criticism, and one edit you make by hand.30 min
33Anatomy of a checkpointOptionalA model file is a set of named tensors: count them, find the ones that write to the residual stream, and save one to bytes and back.35 min
34Reading weights without running the modelOptionalSingular values, and an attention head's token-to-token circuit, read from the matrices alone and then checked against a measurement.35 min
35Writing weights by handOptionalSet a small network's 55 weights so it catches every ball in a paddle game, the way a program would be written, and meet compiled transformers.40 min
36Edits are deltas: task vectors, patches and hashesOptionalSubtract a base from a fine-tune and the edit becomes an object: scale it, take it away, ship it as a patch, and prove by hash what was applied.35 min
37Merging modelsOptionalAverage two fine-tunes, add their task vectors, try SLERP, TIES and DARE, measure the interference, and compare with a model trained on both tasks.40 min
38Low-rank editsOptionalCut a fine-tune's change down to its largest directions with the SVD: how few a behavior needs, and how to pull an adapter out of a delta.35 min
39Directions: steer the run or edit the weightsOptionalFind a behavior's direction in the residual stream, add it at run time, project it out of the weights, and see the effect change from input to input.40 min
40Editing one factOptionalA rank-one change to one matrix rewrites what a product is made from; then reliability, generalization, locality, and what breaks after many edits.40 min
41The editor's harnessOptionalWhat an LLM that edits weights is given and what it is denied: weights as text and their cost in tokens, a sandbox, a budget of forward passes, and rules the harness enforces.40 min
42An LLM reads a network's weightsOptionalSix recorded sessions: an LLM is handed 1,315 weights as text and asked what the network was trained to do. Score each answer against the network, and study one it got wrong.40 min
43An LLM fixes a model's weightsOptionalFind a tiny GPT's loop by hand, then replay recorded sessions in which an LLM hunts for it with ten forward passes, and measure every fix on held-out questions.45 min
44An LLM writes a network's weightsOptionalFive recorded sessions: an LLM writes every weight of a paddle network from the game's source and one sentence. Score each against its specification, test its design claims, and edit one by hand.40 min
45Did it work? Testing an editOptionalReview a patch for a tiny GPT that can't stop talking: held-out questions, a paired comparison, a same-size control, and what the edit cost.40 min
46The published edit, on a real modelOptionalRead the edit published for an open model of 870 million weights, apply and verify one in the same form, see what 16-bit and 4-bit storage do to a fix, and run the real thing from a notebook.45 min
47Safety, licenses and provenanceOptionalBefore a custom model is released: trace its lineage by hash, work out what its parents' licenses pass on, see what an edit can do to a safeguard, and write its card.40 min
48Capstone: ship a custom modelOptionalSet release gates, then diagnose, edit, measure and document a tiny GPT with a fault no lesson has fixed, and write the record a stranger could check.45 min
Read a model's weights, change them by hand and judge an edit someone else made: deltas, merging, low-rank edits, directions and fact edits on a tiny GPT in your browser. Then replay recorded sessions of an LLM reading, fixing and writing weights, check the published edit of a real open model, and ship a model of your own with its patch, verdict and card. Every number is printed by the lesson's own code, and the method's limits are taught with it.
See the course →
Never trained an AI? Teach a small one to write, right in your browser. It takes about 20 seconds and needs no account. Teach it to write