Index Ai Academy
Phase 6 · Weight-space programming

Reading weights without running the model

Singular values, and an attention head's token-to-token circuit, read from the matrices alone and then checked against a measurement.

About 35 minutes · Optional lesson: deeper material that no phase or certificate requires · A free account opens the whole lesson.
Already have an account? Sign in to open this lesson

What it covers8

Why it matters

The boldest claim about weight-space programming is that weights can be read: that you can say what a network does without running it. Some of that is ordinary linear algebra, and you can do it yourself.

This lesson reads two tiny GPTs that share an architecture. One was trained on product descriptions. The other was trained only to copy repeated patterns. You will tell them apart from their matrices alone, find which heads carry the copying circuit, and then run the model to see what the reading got right and what it could not see. That boundary is the same one the published experiments ran into.

What you'll be able to do

  • Compute a matrix's singular values and say what a lopsided set of them means
  • Multiply four weight matrices into one attention head's token-to-token circuit, and score how much it copies
  • Check a reading made from weights against a measurement made by running the model
  • Say what could be read from the weights here, and what needed a forward pass
Quick check
Two tiny GPTs have the same layers and the same shapes. One learned product descriptions. The other learned only to copy repeated patterns. A guess: can you tell which is which without running either?
  1. Only by checking which file is larger
  2. Yes: multiplying a few of its weight matrices shows which model has heads that copy
  3. Only by reading the training data, which is stored in the weights
  4. No: weights mean nothing until an input flows through them
Show the answer

Yes: multiplying a few of its weight matrices shows which model has heads that copy

A product of four matrices shows, for each head, what it would add to the prediction after looking at a given token. In the copying model those products have a strong pattern the other model lacks. You will compute them in this lesson.

The idea: read a matrix before you run the model

This lesson rests on SVD and PCA: the important directions first: a matrix's singular values say how much each direction matters.

Start with one matrix: its singular values

The simplest thing to read from a matrix is how lopsided it is. Every matrix stretches some directions more than others. Its singular values list those stretches, largest first. The SVD and PCA lesson used them to compress data. Here they describe a layer.

If one or two singular values are far larger than the rest, the layer mostly does one or two things: it reacts strongly to a few patterns and barely at all to the others. If they are all about the same size, the layer treats every direction alike, which is what a matrix of random numbers does.

In plain words: square each singular value and you get the energy in that direction. The energies add up to the sum of every squared entry in the matrix. So "the first 5 values hold 90% of the energy" means five directions account for almost everything the matrix does.

A network starts from random numbers, and training carves structure into it. So the spectrum is a first, cheap sign of how much was carved. The first cell of the next part compares one tensor across three sources.

Quick check
What does this print?
import numpy as np

M = np.array([[3.0, 0.0], [0.0, 4.0], [0.0, 0.0]])
print(np.linalg.svd(M, compute_uv=False))
  1. [4. 3. 0.]
  2. [3. 4.]
  3. [4. 3.]
  4. [5.]
Show the answer

[4. 3.]

This matrix stretches one direction by 3 and another by 4, and singular values are listed largest first. A 3 by 2 matrix has two of them.

Try it now

Five tensors of the copying model. The bars at the bottom are the singular values, largest first, and the sentence above them says how many hold 90% of the energy.

In your own words
In your own words: what does it mean if only one or two singular values of a layer are large?

Write a sentence or two, then compare it with a good answer in the lesson.

In code: the heads' circuit, and the spectra

Compare spectra

Worked example

One tensor, three sources

The same tensor, the first half of block 0's feed-forward layer, from the model trained on the catalog, from the model trained to copy, and from random numbers with the same spread. For each: the largest singular value divided by the middle one, and how many values hold 90% of the energy.

import numpy as np
import wsp

base, _ = wsp.load_state("tiny-gpt")
copier, _ = wsp.load_state("tiny-gpt-induction")
name = "blocks.0.mlp.0.weight"
rng = np.random.default_rng(0)
random_like = rng.normal(0, copier[name].std(), copier[name].shape)

def lopsided(M):
    s = np.linalg.svd(M, compute_uv=False)        # singular values, largest first
    energy = np.cumsum(s ** 2) / np.sum(s ** 2)
    return s[0] / np.median(s), int(np.searchsorted(energy, 0.9) + 1)

sources = [("trained on the catalog", base[name]), ("trained to copy", copier[name]),
           ("random numbers", random_like)]
for label, M in sources:
    ratio, k = lopsided(M)
    head = f"{label:23s} largest / middle value {ratio:4.1f}"
    print(head + f"   values for 90% of the energy: {k} of 32")
It runs right here in your browser, with nothing to install.

The copying model's layer is clearly lopsided: its largest singular value is 4.8 times its middle one, and 19 of 32 values hold 90% of the energy. The random matrix needs 25, and its largest value is only 1.5 times the middle. Now look at the catalog model: 1.6 and 25. By this measure it is as flat as random numbers, though it is a working language model. So a lopsided spectrum is evidence of structure, but a flat one is not evidence of none. Singular values are a first look, not a verdict.

Plot the spectra of the same tensor from the three sources. A lopsided matrix falls steeply, a random one stays flat.

import numpy as np
import matplotlib.pyplot as plt
import wsp

base, _ = wsp.load_state("tiny-gpt")
copier, _ = wsp.load_state("tiny-gpt-induction")
name = "blocks.0.mlp.0.weight"
rng = np.random.default_rng(0)
random_like = rng.normal(0, copier[name].std(), copier[name].shape)
sources = {"catalog": base[name], "copier": copier[name], "random": random_like}
for label, M in sources.items():
    plt.plot(np.linalg.svd(M, compute_uv=False), label=label)
plt.xlabel("singular value number")
plt.ylabel("size")
plt.title("Spectra of one tensor, three sources")
plt.legend()
plt.show()
Drill

How many directions matter?

Write energy_rank(M, share=0.9): the smallest number of singular values whose squares add up to at least share of the total. This is the number the widget prints.

import numpy as np

def energy_rank(M, share=0.9):
    return 0

print(energy_rank(np.diag([10.0, 1.0, 1.0])))
In the lesson, your code is checked as you write it.
In your own words
In your own words: the catalog model's spectrum looked as flat as random numbers. Why doesn't that mean the model learned nothing?

Write a sentence or two, then compare it with a good answer in the lesson.

An attention head is two circuits

Single matrices only say so much. A better reading comes from multiplying matrices that work together.

An attention head does two jobs. First it decides where to look, by comparing the current position's query with every earlier position's key. Then it decides what to write: it takes the value at the position it attends to and passes it through the output matrix into the stream. The second job can be read from the weights without knowing where the head will look.

Follow one token through it. The token's embedding is a row of tok.weight. The head's value matrix V reads it. The head's output matrix O writes the result into the stream. The model's head.weight turns the stream into a score for every token. Chain the four and you get one table:

M=U O V E⊤M = U \, O \, V \, E^{\top}

Read it as: EE is the token embedding, VV the head's value matrix, OO its output matrix and UU the final layer's weights. Entry (row r, column c) of MM says: if this head attends to token c, how much does it raise the score of token r as the next token?

A head that copies raises the score of the token it looked at. In the table that is a large positive diagonal. The table is called the head's OV circuit, for output and value. It skips the LayerNorms on the way, so it is an approximation, and it says nothing about where the head looks.

Worked example

Score every head for copying, from weights alone

A head's value matrix is 8 rows of qkv.weight and its output matrix is 8 columns of proj.weight. The copy score is how far the table's diagonal stands above its average entry, in units of the table's spread. No model is run.

import numpy as np
import wsp

D, HEADS = 32, 4
SIZE = D // HEADS                                     # each head works in 8 dimensions

def copy_score(state, layer, head):
    qkv = state[f"blocks.{layer}.qkv.weight"]   # rows: 0-31 q, 32-63 k, 64-95 v
    lo = 2 * D + head * SIZE
    V = qkv[lo: lo + SIZE]                  # (8, 32) reads the stream
    proj = state[f"blocks.{layer}.proj.weight"]
    O = proj[:, head * SIZE:(head + 1) * SIZE]   # (32, 8) writes to it
    E, U = state["tok.weight"][4:], state["head.weight"][4:]   # 191 ordinary tokens
    M = U @ O @ V @ E.T            # column: token looked at; row: token scored
    return (np.diag(M).mean() - M.mean()) / M.std()

for name in ["tiny-gpt", "tiny-gpt-induction"]:
    state, _ = wsp.load_state(name)
    print(name)
    for layer in (0, 1):
        scores = [copy_score(state, layer, h) for h in range(HEADS)]
        row = "  ".join(f"head {h} {s:+.2f}" for h, s in enumerate(scores))
        print(f"  layer {layer}: {row}")
In the lesson you can change this and run it.

In the catalog model every score is near zero, from −0.09 to +0.16: no head copies. In the copying model all four heads of layer 0 score between +1.37 and +2.01. Their diagonals stand well above the rest of the table, so each would raise the score of whatever token it attends to. Its layer 1 heads score below zero. You have told the two models apart, and found where the copying circuit lives, without a single forward pass.

Quick check
Each head here works in 8 dimensions, and the values are rows 64 to 95 of qkv.weight. This prints the first and last row of head 2's value matrix. What does it print?
head, size = 2, 8
start = 64 + head * size
print(start, start + size - 1)
  1. 64 71
  2. 72 79
  3. 16 23
  4. 80 87
Show the answer

80 87

Values start at row 64. Head 0 takes rows 64 to 71, head 1 takes 72 to 79, and head 2 takes 80 to 87: 64 + 2 × 8 = 80, and eight rows from there.

Quick check
From the copying lesson: what does an induction head do when a token appears for the second time?
  1. It looks only at the first position of the prompt
  2. It predicts the most common token in the vocabulary, whatever the prompt contains
  3. It looks at the position right after the token's earlier appearance and predicts that token
  4. It repeats the current token
Show the answer

It looks at the position right after the token's earlier appearance and predicts that token

It finds where the current token appeared before and copies what came next. That needs both circuits: looking at the right position, and writing the token found there into the prediction. The copy score you computed covers only the writing half.

Drill

Build a token-to-token table

Write ov_table(E, V, O, U) returning the table U @ O @ V @ E.T, and copy_score(M) returning how far the diagonal's average stands above the average of all entries, divided by the standard deviation of all entries. The starter's toy head copies perfectly: check that your score is high for it.

import numpy as np

def ov_table(E, V, O, U):
    return np.zeros((len(U), len(E)))

def copy_score(M):
    return 0.0

E = np.eye(4)                       # 4 tokens, each its own direction
V = O = np.eye(4)                   # the head passes values straight through
U = np.eye(4)                       # the output layer reads each direction as its token
M = ov_table(E, V, O, U)
print(M.shape, round(copy_score(M), 2))
In the lesson, your code is checked as you write it.
In your own words
In your own words: what does one entry of a head's token-to-token table mean, and what would a copying head's table look like?

Write a sentence or two, then compare it with a good answer in the lesson.

In production: check a reading against a run

A reading is a prediction about behavior, and predictions get checked. The weights said that the copying model's four layer-0 heads each write a copy of what they attend to. They did not say where those heads look. For that there is a direct measurement from the copying lesson. Feed the model a random sequence written twice. For each head, measure the share of attention that lands exactly where the copy rule points: from a token in the second copy to the position just after its twin in the first.

Worked example

Now run it: where does each head look?

Sixteen random tokens, repeated. For every head, the share of its attention that goes to the position right after the current token's earlier twin. This cell does run the model.

import math
import torch
from tinygpt import load_pretrained

model, tok = load_pretrained("tiny-gpt-induction")
L = 16
torch.manual_seed(1)
pattern = torch.randint(4, len(tok.vocab), (1, L))
x = torch.cat([pattern, pattern], dim=1)[:, :-1]

with torch.no_grad():
    h = model.tok(x) + model.pos(torch.arange(x.shape[1]))
    for layer, blk in enumerate(model.blocks):
        a = blk.ln1(h)
        B, T, d = a.shape
        q, k, v = blk.qkv(a).split(d, dim=-1)
        q, k = (t.view(B, T, blk.h, d // blk.h).transpose(1, 2) for t in (q, k))
        s = q @ k.transpose(-2, -1) / math.sqrt(d // blk.h)
        causal = torch.tril(torch.ones(T, T)).bool()
        s = s.masked_fill(~causal, float("-inf"))
        A = torch.softmax(s, dim=-1)[0]                          # heads x T x T
        def share(head):
            hits = [A[head, t, t - L + 1].item() for t in range(L, 2 * L - 1)]
            return sum(hits) / (L - 1)
        scores = [share(head) for head in range(blk.h)]
        row = "  ".join(f"head {i} {sc:.2f}" for i, sc in enumerate(scores))
        print(f"layer {layer}: {row}")
        h = blk(h)
In the lesson you can change this and run it.

One head stands out: layer 0, head 1 sends 0.19 of its attention to the right place, and head 3 sends 0.09. Heads 0 and 2 send none, though the weights gave them the two highest copy scores. Both results are true. Heads 0 and 2 would copy whatever they looked at, but on this input they look elsewhere. The table told you what each head writes. Only the run told you which head finds the earlier twin. As the copying lesson said, this small model spreads the job around, and its best score of 0.19 is far below the near 1 of a textbook head.

Quick check
In the run, which head sent the largest share of its attention where the copy rule points?
  1. All four heads of layer 0, equally
  2. Layer 1, head 1, with 0.06
  3. Layer 0, head 0, with 2.01
  4. Layer 0, head 1, with 0.19
Show the answer

Layer 0, head 1, with 0.19

Layer 0, head 1 put 0.19 of its attention on the position after the twin, more than twice the next head's 0.09. The 2.01 belongs to the other table: it is head 0's copy score from the weights, which measures something else.

Why couldn't the weights say where a head looks? Where it looks depends on the query and key at each position, and those are computed from the stream as it stands during a run: token, position and whatever earlier layers added, passed through a LayerNorm. You can multiply the query and key matrices too, but for this model no simple product of them shows the pattern. The measurement was the easier and more honest tool.

The published experiments met the same boundary. In one, the editor model found the copying heads of three language models by analyzing their weights in Python, with no forward pass allowed. In another, it had to tell a flattering model from an honest one. From weights alone it failed in four attempts. Once it could see the models' attention on texts it chose, it picked the flatterer in all three pairs. Some things are written in the weights for anyone who multiplies the right matrices. Others show up only when something flows through.

Quick check
A head's token-to-token table has a strong positive diagonal. What does that tell you, and what doesn't it tell you?
  1. It attends to the token just before the current one; it doesn't say what it then writes
  2. The head is unused, because a diagonal table changes nothing
  3. It would raise the score of whichever token it attends to; it doesn't say which positions it attends to
  4. The model was trained to copy; nothing else needs checking
Show the answer

It would raise the score of whichever token it attends to; it doesn't say which positions it attends to

The table covers what the head writes after it has looked. Where it looks is a separate circuit, the queries and keys, and in this model that had to be measured. Two heads with the highest copy scores turned out to look in the wrong place.

Where it shows up in your week

When you receive a checkpoint you did not train, you read its spectra and a few circuits before you run it. It tells you which layers carry structure and gives you a prediction to test. The usual slip is treating a reading as a measurement: the weights cannot say where a head looks.

In your own words
In your own words: from weights alone, what could you say about the copying model, and what did you need a run for?

Write a sentence or two, then compare it with a good answer in the lesson.

Goes deeper in

Practice what you just read

  • 4 code examples you can edit and run
  • a practice exercise that checks your code
  • a 5-question quiz
  • 9 review cards for spaced repetition

Try a quiz question

A weight matrix's first 3 singular values hold 95% of its energy. What does that suggest?

  1. The layer mostly acts along three directions and does little along the others
  2. The matrix has only three rows
  3. The layer was never trained
  4. Three of the layer's individual weights are much larger than all the rest

Answer it at the end of the lesson to see if you're right, and why.

Before this lesson

Go deeper, if you like

Optional lessons that build on this one. They never block a phase or a certificate.

Keep learning

Where this lesson leads.

575 lessons in thirteen phases. 176 are free without an account, and a free account opens the rest. See every lesson.

Phase 6 · about 35 hours of study

Modern LLM engineering

Post-training, fine-tuning, serving and safety 48 lessons 2 free
  1. Base models vs assistants FreeWhat post-training adds to a pretrained model.30 min
  2. Supervised fine-tuningTeach behavior from example conversations.40 min
  3. Instruction and preference dataThe datasets that shape assistant behavior.30 min
  4. RLHFReward models and reinforcement learning from human feedback.40 min
  5. DPO and direct preference methodsPreference training without a separate reward model.40 min
  6. Constitutional AITraining with AI feedback guided by written principles.35 min
  7. Training reasoning modelsReinforcement learning on verifiable problems.40 min
  8. Prompting vs RAG vs fine-tuningPick the cheapest approach that works.30 min
  9. LoRAFine-tune a big model by training small low-rank updates.40 min
  10. Fine-tune an open modelRun a real fine-tune and evaluate the result.45 min
  11. DistillationTrain a small model to imitate a large one.35 min
  12. Tuning an embedding model for retrievalWhen search fails on your own vocabulary: train an adapter with a contrastive loss on question and passage pairs, test on held-out passages, use batch size and hard negatives, and measure what tuning costs elsewhere.45 min
  13. QuantizationStore weights in fewer bits to run models cheaply.35 min
  14. Quantization in practice: zero points, calibration and formatsScale and zero point, per-channel scales, calibration sets, error on a layer's output, quantization-aware training and the file formats you meet at work.35 min
  15. Serving modelsBatching, throughput and latency in production.35 min
  16. One request through a serving stack: latency and costFollow one request through queue, prefill, a prefix cache, decode and the stream, and price a token from a GPU's hourly cost.35 min
  17. Speculative decodingDraft with a small model, verify with a big one.35 min
  18. Running open models locallyDownload, quantize and run a model on your own machine.30 min
  19. Will it fit? Memory for serving and fine-tuningWeights, KV cache, runtime and adapter states added up for a model card and a GPU, with a calculator.30 min
  20. InterpretabilityLooking inside models to see what they compute.40 min
  21. Confident and wrongWhy a model answers with the same assurance whether it is right, wrong or asked about something that does not exist.30 min
  22. Why it can't spell or count lettersThe model sees token IDs, and an embedding learned from usage carries nothing about spelling.25 min
  23. Copying from the promptIn-context learning as a two-head copying circuit: the base tiny GPT cannot copy a repeated pattern, a model trained on repeats grows an induction head and can.30 min
  24. Why temperature changes the answerThe same prompt at 0, 0.7 and 1.5: deterministic loops, plausible variety, broken grammar, measured on a real model.25 min
  25. Why long prompts diluteAttention is a budget shared by every token, so a fact far from the question loses its share; measured, with the cliff beyond the trained window.25 min
  26. Why it never says "I don't know"A chat model answers every request because its fine-tuning data always did; preference training adds flattery; how to build abstention in.25 min
  27. Benchmarks and their limitsWhat leaderboards measure, and what they miss.38 min
  28. Alignment and safetyMaking capable models behave as intended.35 min
  29. Evaluating agentsMeasure multi-step, tool-using behavior.35 min
  30. Reading AI papersGet the idea out of a research paper in 30 minutes.30 min
  31. Capstone: a tuned, evaluated assistantFine-tune, evaluate and deploy a specialist assistant.45 min
  32. Weight-space programming: what it is, and what has been shown Free OptionalA language model reads and writes another network's weights with no training loop: the published experiments, their limits and the criticism, and one edit you make by hand.30 min
  33. Anatomy of a checkpoint OptionalA model file is a set of named tensors: count them, find the ones that write to the residual stream, and save one to bytes and back.35 min
  34. Reading weights without running the model OptionalSingular values, and an attention head's token-to-token circuit, read from the matrices alone and then checked against a measurement.35 min
  35. Writing weights by hand OptionalSet a small network's 55 weights so it catches every ball in a paddle game, the way a program would be written, and meet compiled transformers.40 min
  36. Edits are deltas: task vectors, patches and hashes OptionalSubtract a base from a fine-tune and the edit becomes an object: scale it, take it away, ship it as a patch, and prove by hash what was applied.35 min
  37. Merging models OptionalAverage two fine-tunes, add their task vectors, try SLERP, TIES and DARE, measure the interference, and compare with a model trained on both tasks.40 min
  38. Low-rank edits OptionalCut a fine-tune's change down to its largest directions with the SVD: how few a behavior needs, and how to pull an adapter out of a delta.35 min
  39. Directions: steer the run or edit the weights OptionalFind a behavior's direction in the residual stream, add it at run time, project it out of the weights, and see the effect change from input to input.40 min
  40. Editing one fact OptionalA rank-one change to one matrix rewrites what a product is made from; then reliability, generalization, locality, and what breaks after many edits.40 min
  41. The editor's harness OptionalWhat an LLM that edits weights is given and what it is denied: weights as text and their cost in tokens, a sandbox, a budget of forward passes, and rules the harness enforces.40 min
  42. An LLM reads a network's weights OptionalSix recorded sessions: an LLM is handed 1,315 weights as text and asked what the network was trained to do. Score each answer against the network, and study one it got wrong.40 min
  43. An LLM fixes a model's weights OptionalFind a tiny GPT's loop by hand, then replay recorded sessions in which an LLM hunts for it with ten forward passes, and measure every fix on held-out questions.45 min
  44. An LLM writes a network's weights OptionalFive recorded sessions: an LLM writes every weight of a paddle network from the game's source and one sentence. Score each against its specification, test its design claims, and edit one by hand.40 min
  45. Did it work? Testing an edit OptionalReview a patch for a tiny GPT that can't stop talking: held-out questions, a paired comparison, a same-size control, and what the edit cost.40 min
  46. The published edit, on a real model OptionalRead the edit published for an open model of 870 million weights, apply and verify one in the same form, see what 16-bit and 4-bit storage do to a fix, and run the real thing from a notebook.45 min
  47. Safety, licenses and provenance OptionalBefore a custom model is released: trace its lineage by hash, work out what its parents' licenses pass on, see what an edit can do to a safeguard, and write its card.40 min
  48. Capstone: ship a custom model OptionalSet release gates, then diagnose, edit, measure and document a tiny GPT with a fault no lesson has fixed, and write the record a stranger could check.45 min
Part of a free course 17 lessons

Creating Custom LLMs: Editing Open Weights Using Weight-Space Programming

Read a model's weights, change them by hand and judge an edit someone else made: deltas, merging, low-rank edits, directions and fact edits on a tiny GPT in your browser. Then replay recorded sessions of an LLM reading, fixing and writing weights, check the published edit of a real open model, and ship a model of your own with its patch, verdict and card. Every number is printed by the lesson's own code, and the method's limits are taught with it.

Never trained an AI? Teach a small one to write, right in your browser. It takes about 20 seconds and needs no account. Teach it to write