In-context learning as a two-head copying circuit: the base tiny GPT cannot copy a repeated pattern, a model trained on repeats grows an induction head and can.
Put three examples in a prompt and a model follows their format. Show it a made-up code word once and it uses it correctly a paragraph later. None of that was in its training, so how does a fixed network do it? Interpretability research found a two-head circuit, an induction head and its partner, that implements "find where this token appeared before and predict what followed it". That one move is most of what people call in-context learning.
The course's base tiny GPT lacks the circuit, which is why a fact in its prompt did nothing in the confident-and-wrong lesson. A second model, trained only on random patterns repeated once, learns the rule, and you will measure it: the loss on the second half of a repeated sequence, and where each head looks. The mechanism explains few-shot prompting, why examples must be consistent, and what grounding relies on.
What you'll be able to do
Show that the base tiny GPT cannot copy a repeated pattern, and that a model trained on repeats can
Score every head for the copy rule and read the loss on repeated text, the two ways to see the circuit
Explain the two-head circuit behind copying, and why it is the mechanism of in-context learning
Use the mechanism to write few-shot prompts that work, and to know what grounding depends on
Quick check
A list of sixteen random tokens is written out twice in a row. A model trained only on such repeated lists is shown the second copy as it starts. A guess at what the model can do:
Predict the second copy almost perfectly, by looking up where each token appeared in the first
Predict the second copy only if it memorized this particular list during training
Predict the most frequent token in its vocabulary, the best guess for random input
Nothing: random tokens cannot be predicted, whether or not they have appeared before
Show the answer
Predict the second copy almost perfectly, by looking up where each token appeared in the first
The tokens are random, so nothing about them can be learned; what can be learned is the rule 'this token appeared before: predict what followed it then'. A model with that rule copies any repeated list, including ones it has never seen.
The idea: copying from the prompt
This lesson rests on Causal masking: a language model reads only the token it is on and the tokens before it.
Few-shot prompting works. Why?
In the examples-in-prompts lesson you learned that a few worked examples steer a model's output more reliably than instructions. The interpretability lesson showed heads that look back at a product word. This lesson joins the two: a model can follow a pattern it has never seen because particular heads implement a copying rule.
Start with what the base tiny GPT does on the cleanest possible test. Take sixteen random tokens, write them twice, and ask the model to predict each token given the ones before it. On the first copy there is nothing to go on; the loss should sit near ln191≈5.25 nats, the loss of guessing uniformly among the 191 content tokens. On the second copy, every token has already appeared sixteen positions earlier, followed by the very token that comes next. A model that can copy will have a much lower loss there. The code part below measures both halves, and where each attention head looks from the second copy.
The current token is "b"; its earlier occurrence is at index 1, and the token that followed it there is "c". That is exactly what an induction head predicts.
The copying circuit
How can attention implement "find my earlier twin and predict what followed it"? Not in one head. A single head compares the current token's query with every earlier token's key, so it can find the earlier twin itself, but the twin's successor is a different position. The circuit uses two heads in two layers:
A previous-token head in the first layer attends from every position to the position just before it, and writes "the token before me was X" into each position's vector.
An induction head in the second layer attends from the current token (say "b") to the positions whose previous token was "b", which is where the earlier "b"'s successor sits, and copies that token into the prediction.
The induction score in the code part measures exactly the second head's job: from a token in the second copy, how much attention lands on the position right after its twin. In large models, a head doing the job scores near 1, and the previous-token head's job can be measured the same way, as attention to the position just before. A model as small as ours can also share the work out across heads and layers, which the code part shows; what matters is that the rule is implemented somewhere, and the loss tells you when it is.
Try it now
The real attention of layer 0, head 0 in tiny-gpt-induction, on four product words written twice (the scores are the logs of its real weights, so the percentages are what the model computed). The model is causal, so a token reads only itself and the tokens before it. Read a row from the second copy: it looks at the token that followed its twin.
In your own words
In your own words: why does the copying circuit need two heads in two layers rather than one?
Write a sentence or two, then compare it with a good answer in the lesson.
In code: measure the circuit
The cells below run two models on the same repeated sequence. The first is the base tiny GPT. The second is trained only on random patterns written twice.
Worked example
The base model on a repeated random sequence
Sixteen random content tokens, repeated. The loss per token on each half, then for every head the share of attention that lands exactly where the copy rule points: from a token in the second copy to the position just after its twin in the first copy.
import math
import torch
from tinygpt import load_pretrained
model, tok = load_pretrained("tiny-gpt")
L = 16
torch.manual_seed(1)
# ids 0-3 are the special tokens
pattern = torch.randint(4, len(tok.vocab), (1, L))
seq = torch.cat([pattern, pattern], dim=1) # the pattern, twicedefhalves_and_heads(model, seq):
x, y = seq[:, :-1], seq[0, 1:]
with torch.no_grad():
nll = -torch.log_softmax(model(x), dim=-1)[0, torch.arange(x.shape[1]), y]
first, second = nll[:L - 1].mean().item(), nll[L:].mean().item()
h = model.tok(x) + model.pos(torch.arange(x.shape[1]))
rows = []
for layer, blk inenumerate(model.blocks):
a = blk.ln1(h)
B, T, d = a.shape
q, k, v = blk.qkv(a).split(d, dim=-1)
q, k = (t.view(B, T, blk.h, d // blk.h).transpose(1, 2) for t in (q, k))
s = (q @ k.transpose(-2, -1) / math.sqrt(d // blk.h)).masked_fill(
~torch.tril(torch.ones(T, T)).bool(), float("-inf"))
A = torch.softmax(s, dim=-1)[0] # heads x T x Tfor head inrange(blk.h):
score = sum(A[head, t, t - L + 1].item()
for t inrange(L, 2 * L - 1)) / (L - 1)
rows.append(f"layer {layer} head {head}: {score:.2f}")
h = blk(h)
return first, second, rows
first, second, rows = halves_and_heads(model, seq)
print(f"loss per token: first copy {first:.2f}, second copy {second:.2f}")
print("induction score per head:", ", ".join(rows))
It runs right here in your browser, with nothing to install.
The base model's loss is no better on the second copy than on the first: 12.75 against 13.20 nats, both far above even uniform guessing, because random content is as alien to this model as a sentence in another language. No head puts more than 0.06 of its attention where the copy rule points. Trained on 300 product descriptions in which nothing repeats this way, it never had a reason to learn copying, and it cannot do it.
Worked example
A model trained to copy
The same measurement on tiny-gpt-induction, a model with the base model's architecture trained only on random patterns of changing length, each repeated once. Both scores for every head in both layers.
import math
import torch
from tinygpt import load_pretrained
model, tok = load_pretrained("tiny-gpt-induction")
L = 16
torch.manual_seed(1)
pattern = torch.randint(4, len(tok.vocab), (1, L))
seq = torch.cat([pattern, pattern], dim=1)
x, y = seq[:, :-1], seq[0, 1:]
with torch.no_grad():
nll = -torch.log_softmax(model(x), dim=-1)[0, torch.arange(x.shape[1]), y]
print(f"loss per token: first copy {nll[:L - 1].mean().item():.2f}, second copy "f"{nll[L:].mean().item():.2f}")
h = model.tok(x) + model.pos(torch.arange(x.shape[1]))
for layer, blk inenumerate(model.blocks):
a = blk.ln1(h)
B, T, d = a.shape
q, k, v = blk.qkv(a).split(d, dim=-1)
q, k = (t.view(B, T, blk.h, d // blk.h).transpose(1, 2) for t in (q, k))
s = (q @ k.transpose(-2, -1) / math.sqrt(d // blk.h)).masked_fill(
~torch.tril(torch.ones(T, T)).bool(), float("-inf"))
A = torch.softmax(s, dim=-1)[0]
for head inrange(blk.h):
prev = sum(A[head, t, t - 1].item()
for t inrange(1, 2 * L - 1)) / (2 * L - 2)
ind = sum(A[head, t, t - L + 1].item()
for t inrange(L, 2 * L - 1)) / (L - 1)
print(f"layer {layer} head {head}: previous-token {prev:.2f}, induction "f"{ind:.2f}")
h = blk(h)
In the lesson you can change this and run it.
The second copy is now almost free: 5.45 nats on the first copy, where nothing can be predicted, and 0.21 on the second, against 12.75 for the base model. The heads tell a subtler story. The strongest single signal is layer 0 head 1, which puts 0.19 of its attention where the copy rule points, three times the base model's best, but nowhere near the scores close to 1 that single induction heads reach in large models, and no head is a clean previous-token head either. In 40,195 parameters the job is spread: several heads carry a share, the feed-forward layers finish the lookup, and the behavior is sharp even though no single head is. Same architecture and size as the base model, different training data, and the data was enough to build a copying machine, just not a tidy one.
Quick check
In the copying model's table, the best induction score is 0.19 while the loss on the second copy is 0.21 nats. Which reading is right?
The rule is implemented, but spread over several heads and the feed-forward layers, not in one head
Layer 0 head 1 is a complete induction head, and 0.19 is the highest score the measure allows
The rule is mostly absent: a best score of 0.19 means the model is barely copying at all
The loss figure must be wrong, since only a head scoring near 1 can produce a loss near 0
Show the answer
The rule is implemented, but spread over several heads and the feed-forward layers, not in one head
A loss of 0.21 nats on random tokens means the second copy is predicted almost perfectly; that is behavior, and it is sharp. The head scores are a probe for one clean implementation, and this small model found a messier one.
Watch it form
tiny-gpt-induction was trained from scratch on one kind of sequence: a random pattern of 4 to 28 content tokens, written twice. Its loss on the first copy can never fall; random tokens have no structure. Its loss on the second copy falls only once the model implements the rule, and when it does, it falls for every pattern length it trained on at once, because the rule does not care how far back the twin is. In the training log that shows as a plateau at the uniform loss followed by a drop, the signature that first drew researchers to these heads in real models.
The payoff is that the rule is general in content. The model never saw a sentence, yet it copies one. The next cell gives it a product sentence repeated, with a material that is false so that copying and knowledge disagree, and a made-up mapping it has never seen; the base model gets the same prompts.
Worked example
Copying something it never saw
Two prompts to both models. The sentence repeats "Our Kestrel tent is made from enamel", a material no tent in the catalog has; the mapping pairs products with materials in a format the models never trained on. The top prediction and its probability.
import torch
from tinygpt import load_pretrained
base, tok = load_pretrained("tiny-gpt")
copier, _ = load_pretrained("tiny-gpt-induction")
deftop(model, text):
with torch.no_grad():
p = torch.softmax(model(torch.tensor([tok.encode(text)]))[0, -1], dim=-1)
returnf"{tok.vocab[int(p.argmax())]} {p.max().item():.2f}"for text in ["Our Kestrel tent is made from enamel . Our Kestrel tent is made from",
"tent silnylon mug enamel tent"]:
print(f"{text!r}")
print(f" base: {top(base, text)}")
print(f" copier: {top(copier, text)}")
In the lesson you can change this and run it.
On the repeated sentence the copier predicts "enamel" with 0.95: the current token is "from", its earlier twin was followed by "enamel", so that is what it copies, catalog or no catalog. The base model answers "silnylon" (0.58) from its training data and ignores the sentence in front of it. On the made-up mapping, "tent silnylon mug enamel tent", the copier predicts "silnylon" with 0.96, the token that followed the earlier "tent": a pairing it learned from the prompt alone, while the base model offers a comma (0.70). That is in-context learning in its smallest form, and the base model, with no copying circuit, cannot do it.
Quick check
During the copying model's training, the loss on the first copy of each pattern stays at about 5.25 nats while the loss on the second copy falls toward zero. Why can the first half never improve?
The first half is too short for attention to find any useful pattern in it
The learning rate is too low for the first half and too high for the second
The tokens are drawn at random, so no rule predicts a token the first time it appears
The model is not trained on the first half, so its loss there is never pushed down
Show the answer
The tokens are drawn at random, so no rule predicts a token the first time it appears
Only structure can be learned. The first copy has none; the second copy's structure is 'the same as the first', and that is what the heads learn to exploit.
Drill
Score one head
Write induction_score(A, L): A is a square NumPy array of attention weights for one head (rows attend to columns) over a sequence of length 2·L − 1 that holds a pattern of length L followed by the pattern again (minus its last token). Return the average, over the positions t from L to 2L − 2, of the attention from t to position t − L + 1, the token after t's twin.
import numpy as np
definduction_score(A, L):
return0.0
L = 3
A = np.zeros((5, 5))
for t inrange(1, 5):
A[t, t - 1] = 1.0# a head that only looks at the previous tokenprint(round(induction_score(A, L), 2))
In the lesson, your code is checked as you write it.
Plot the loss at each position for the copying model on one 16-token pattern written twice. The first copy sits near the uniform loss, because nothing can be predicted yet. After the second copy starts, the loss falls to almost nothing.
import torch
import matplotlib.pyplot as plt
from tinygpt import load_pretrained
model, tok = load_pretrained("tiny-gpt-induction")
L = 16
torch.manual_seed(1)
pattern = torch.randint(4, len(tok.vocab), (1, L))
seq = torch.cat([pattern, pattern], dim=1)
x, y = seq[:, :-1], seq[0, 1:]
with torch.no_grad():
nll = -torch.log_softmax(model(x), dim=-1)[0, torch.arange(x.shape[1]), y]
plt.plot(range(1, 2 * L), nll.tolist(), marker="o", label="loss at each position")
plt.axvline(L + 0.5, linestyle="--", label="second copy starts")
plt.xlabel("position")
plt.ylabel("loss (nats)")
plt.legend()
print(round(nll[:L - 1].mean().item(), 2), round(nll[L:].mean().item(), 2))
In your own words
In your own words: why does the loss on the first copy stay high while the loss on the second copy falls?
Write a sentence or two, then compare it with a good answer in the lesson.
In production: what copying justifies
From copying to learning in context
Exact copying is the clearest case, and large models go further. Their induction heads match on similar tokens as well as identical ones, and they operate on structure: a label that followed one kind of example is predicted after another example of the same kind. Give a model three lines of "review → sentiment" and it produces a sentiment after the fourth review; the heads that do this are the same circuit, matching the pattern "review, arrow" and copying the kind of token that followed it. Research on large models finds that the appearance of induction heads during training coincides with the appearance of in-context learning, and that removing them removes most of it.
The circuit also explains the limits. It copies what is in the context, including mistakes: an example with the wrong label teaches the wrong mapping for the rest of the prompt. It needs the twin to be in the context, so it is bounded by the window and weakened by the dilution of long prompts. And it is a pattern-matcher, not a reasoner: it reproduces the shape of what it saw.
Quick check
From the examples-in-prompts lesson: why do a few worked examples steer the output more reliably than a description of the format?
The model copies the pattern the examples show, which is exactly what its induction circuit does
The model stores the examples in its weights for the rest of the conversation
Examples are shorter than instructions, so they fit in the context more easily
Instructions are ignored by language models unless they come with examples
Show the answer
The model copies the pattern the examples show, which is exactly what its induction circuit does
A worked example is a pattern in the context, and copying patterns from the context is the machinery the model has. Instructions have to be translated into behavior; examples are the behavior.
Drill
The copy rule as a function
Write copy_predict(tokens): find the most recent earlier occurrence of the last token and return the token that followed it. Return None when the last token has not appeared before. Search from the end, so the nearest twin wins.
In the lesson, your code is checked as you write it.
In your own words
A few-shot prompt contains one example with a wrong label. What does the copying circuit do with it, and why?
Write a sentence or two, then compare it with a good answer in the lesson.
What this justifies
Few-shot examples: consistent, correct, close. The circuit copies format and mapping, so every example must share one format, every label must be right, and the examples should sit near the question, inside the budget the long-prompts lesson measured.
Grounding depends on copying. Retrieval puts the fact in the context; the induction circuit is what moves it into the answer. The confident-and-wrong lesson showed a model without the circuit ignoring a fact placed right in front of it. Production models have the circuit, which is why grounding works for them and why it weakens exactly where copying does: far from the question, in long prompts.
Expect pattern matching, not understanding. A model that copies "tent → silnylon" from the prompt has learned nothing about tents. In-context learning is powerful because patterns carry a great deal, and limited for the same reason.
Worked example
Three pairs, then a question
A longer made-up mapping on the copying model: three product-material pairs, then one product repeated. The top prediction and the share the correct material gets. Watch the third line.
import torch
from tinygpt import load_pretrained
copier, tok = load_pretrained("tiny-gpt-induction")
pairs = "tent silnylon mug enamel pants organic"for product, material in [("tent", "silnylon"), ("mug", "enamel"),
("pants", "organic")]:
with torch.no_grad():
p = torch.softmax(
copier(torch.tensor([tok.encode(f"{pairs} {product}")]))[0, -1], dim=-1)
print(f"{pairs} {product} -> {tok.vocab[int(p.argmax())]} (p of {material}: "f"{p[tok.token_id(material)].item():.2f})")
In the lesson you can change this and run it.
"tent" recalls silnylon with 0.95 and "mug" recalls enamel with 0.94: the model finds the earlier twin, six and four tokens back, and copies what followed it. Nothing about tents or enamel was learned here; it learned a rule about repeats, and pairs in a prompt are repeats. The third line fails: for "pants" the twin is only two tokens back, and the correct material gets 0.00. The training patterns were 4 to 28 tokens long, so the model never saw a twin closer than four positions and never learned to look there. The rule is general in content but was learned only over the distances the training data covered, a limit that large models, trained on text with repeats at every distance, do not share.
In your own words
The copying model copies sentences and mappings it never saw, yet misses a twin two tokens back. What did it learn, and what did it not learn?
Write a sentence or two, then compare it with a good answer in the lesson.
Quick check
The copy rule applied to a few-shot prompt. What does this print?
tokens = ["ok", "good", "slow", "bad", "fast", "good", "slow"]
current = tokens[-1]
earlier = [i for i, t inenumerate(tokens[:-1]) if t == current]
print(tokens[earlier[-1] + 1], len(earlier))
bad 1
fast 1
slow 2
good 1
Show the answer
bad 1
The last token is "slow"; it appeared once before, at index 2, followed by "bad". The circuit would predict "bad", the label the prompt attached to "slow".
In your own words
Why does grounding work on production models but not on the base tiny GPT, and what does that tell you about where in a prompt to put retrieved text?
Write a sentence or two, then compare it with a good answer in the lesson.
11DistillationTrain a small model to imitate a large one.35 min
12Tuning an embedding model for retrievalWhen search fails on your own vocabulary: train an adapter with a contrastive loss on question and passage pairs, test on held-out passages, use batch size and hard negatives, and measure what tuning costs elsewhere.45 min
13QuantizationStore weights in fewer bits to run models cheaply.35 min
23Copying from the promptIn-context learning as a two-head copying circuit: the base tiny GPT cannot copy a repeated pattern, a model trained on repeats grows an induction head and can.30 min
24Why temperature changes the answerThe same prompt at 0, 0.7 and 1.5: deterministic loops, plausible variety, broken grammar, measured on a real model.25 min
25Why long prompts diluteAttention is a budget shared by every token, so a fact far from the question loses its share; measured, with the cliff beyond the trained window.25 min
26Why it never says "I don't know"A chat model answers every request because its fine-tuning data always did; preference training adds flattery; how to build abstention in.25 min
32Weight-space programming: what it is, and what has been shownFreeOptionalA language model reads and writes another network's weights with no training loop: the published experiments, their limits and the criticism, and one edit you make by hand.30 min
33Anatomy of a checkpointOptionalA model file is a set of named tensors: count them, find the ones that write to the residual stream, and save one to bytes and back.35 min
34Reading weights without running the modelOptionalSingular values, and an attention head's token-to-token circuit, read from the matrices alone and then checked against a measurement.35 min
35Writing weights by handOptionalSet a small network's 55 weights so it catches every ball in a paddle game, the way a program would be written, and meet compiled transformers.40 min
36Edits are deltas: task vectors, patches and hashesOptionalSubtract a base from a fine-tune and the edit becomes an object: scale it, take it away, ship it as a patch, and prove by hash what was applied.35 min
37Merging modelsOptionalAverage two fine-tunes, add their task vectors, try SLERP, TIES and DARE, measure the interference, and compare with a model trained on both tasks.40 min
38Low-rank editsOptionalCut a fine-tune's change down to its largest directions with the SVD: how few a behavior needs, and how to pull an adapter out of a delta.35 min
39Directions: steer the run or edit the weightsOptionalFind a behavior's direction in the residual stream, add it at run time, project it out of the weights, and see the effect change from input to input.40 min
40Editing one factOptionalA rank-one change to one matrix rewrites what a product is made from; then reliability, generalization, locality, and what breaks after many edits.40 min
41The editor's harnessOptionalWhat an LLM that edits weights is given and what it is denied: weights as text and their cost in tokens, a sandbox, a budget of forward passes, and rules the harness enforces.40 min
42An LLM reads a network's weightsOptionalSix recorded sessions: an LLM is handed 1,315 weights as text and asked what the network was trained to do. Score each answer against the network, and study one it got wrong.40 min
43An LLM fixes a model's weightsOptionalFind a tiny GPT's loop by hand, then replay recorded sessions in which an LLM hunts for it with ten forward passes, and measure every fix on held-out questions.45 min
44An LLM writes a network's weightsOptionalFive recorded sessions: an LLM writes every weight of a paddle network from the game's source and one sentence. Score each against its specification, test its design claims, and edit one by hand.40 min
45Did it work? Testing an editOptionalReview a patch for a tiny GPT that can't stop talking: held-out questions, a paired comparison, a same-size control, and what the edit cost.40 min
46The published edit, on a real modelOptionalRead the edit published for an open model of 870 million weights, apply and verify one in the same form, see what 16-bit and 4-bit storage do to a fix, and run the real thing from a notebook.45 min
47Safety, licenses and provenanceOptionalBefore a custom model is released: trace its lineage by hash, work out what its parents' licenses pass on, see what an edit can do to a safeguard, and write its card.40 min
48Capstone: ship a custom modelOptionalSet release gates, then diagnose, edit, measure and document a tiny GPT with a fault no lesson has fixed, and write the record a stranger could check.45 min
Confident wrong answers, miscounted letters, copying from the prompt, temperature, long prompts that dilute, and a model that never says "I don't know": each behavior reproduced on a tiny GPT with every number printed by code, then the mechanism behind it and the Phase 1 technique it justifies.
See the course →
Never trained an AI? Teach a small one to write, right in your browser. It takes about 20 seconds and needs no account. Teach it to write