Index Ai Academy
Phase 6 · Why models do that

Copying from the prompt

In-context learning as a two-head copying circuit: the base tiny GPT cannot copy a repeated pattern, a model trained on repeats grows an induction head and can.

About 30 minutes · A free account opens the whole lesson.
Already have an account? Sign in to open this lesson

What it covers9

Why it matters

Put three examples in a prompt and a model follows their format. Show it a made-up code word once and it uses it correctly a paragraph later. None of that was in its training, so how does a fixed network do it? Interpretability research found a two-head circuit, an induction head and its partner, that implements "find where this token appeared before and predict what followed it". That one move is most of what people call in-context learning.

The course's base tiny GPT lacks the circuit, which is why a fact in its prompt did nothing in the confident-and-wrong lesson. A second model, trained only on random patterns repeated once, learns the rule, and you will measure it: the loss on the second half of a repeated sequence, and where each head looks. The mechanism explains few-shot prompting, why examples must be consistent, and what grounding relies on.

What you'll be able to do

  • Show that the base tiny GPT cannot copy a repeated pattern, and that a model trained on repeats can
  • Score every head for the copy rule and read the loss on repeated text, the two ways to see the circuit
  • Explain the two-head circuit behind copying, and why it is the mechanism of in-context learning
  • Use the mechanism to write few-shot prompts that work, and to know what grounding depends on
Quick check
A list of sixteen random tokens is written out twice in a row. A model trained only on such repeated lists is shown the second copy as it starts. A guess at what the model can do:
  1. Predict the second copy almost perfectly, by looking up where each token appeared in the first
  2. Predict the second copy only if it memorized this particular list during training
  3. Predict the most frequent token in its vocabulary, the best guess for random input
  4. Nothing: random tokens cannot be predicted, whether or not they have appeared before
Show the answer

Predict the second copy almost perfectly, by looking up where each token appeared in the first

The tokens are random, so nothing about them can be learned; what can be learned is the rule 'this token appeared before: predict what followed it then'. A model with that rule copies any repeated list, including ones it has never seen.

The idea: copying from the prompt

This lesson rests on Causal masking: a language model reads only the token it is on and the tokens before it.

Few-shot prompting works. Why?

In the examples-in-prompts lesson you learned that a few worked examples steer a model's output more reliably than instructions. The interpretability lesson showed heads that look back at a product word. This lesson joins the two: a model can follow a pattern it has never seen because particular heads implement a copying rule.

Start with what the base tiny GPT does on the cleanest possible test. Take sixteen random tokens, write them twice, and ask the model to predict each token given the ones before it. On the first copy there is nothing to go on; the loss should sit near ln⁡191≈5.25\ln 191 \approx 5.25 nats, the loss of guessing uniformly among the 191 content tokens. On the second copy, every token has already appeared sixteen positions earlier, followed by the very token that comes next. A model that can copy will have a much lower loss there. The code part below measures both halves, and where each attention head looks from the second copy.

Quick check
The copy rule, in Python. What does this print?
seq = ["a", "b", "c", "a", "b"]
current = seq[-1]
earlier = seq.index(current)
print(seq[earlier + 1])
  1. an error
  2. a
  3. c
  4. b
Show the answer

c

The current token is "b"; its earlier occurrence is at index 1, and the token that followed it there is "c". That is exactly what an induction head predicts.

The copying circuit

How can attention implement "find my earlier twin and predict what followed it"? Not in one head. A single head compares the current token's query with every earlier token's key, so it can find the earlier twin itself, but the twin's successor is a different position. The circuit uses two heads in two layers:

  1. A previous-token head in the first layer attends from every position to the position just before it, and writes "the token before me was X" into each position's vector.
  2. An induction head in the second layer attends from the current token (say "b") to the positions whose previous token was "b", which is where the earlier "b"'s successor sits, and copies that token into the prediction.

The induction score in the code part measures exactly the second head's job: from a token in the second copy, how much attention lands on the position right after its twin. In large models, a head doing the job scores near 1, and the previous-token head's job can be measured the same way, as attention to the position just before. A model as small as ours can also share the work out across heads and layers, which the code part shows; what matters is that the rule is implemented somewhere, and the loss tells you when it is.

Try it now

The real attention of layer 0, head 0 in tiny-gpt-induction, on four product words written twice (the scores are the logs of its real weights, so the percentages are what the model computed). The model is causal, so a token reads only itself and the tokens before it. Read a row from the second copy: it looks at the token that followed its twin.

In your own words
In your own words: why does the copying circuit need two heads in two layers rather than one?

Write a sentence or two, then compare it with a good answer in the lesson.

In code: measure the circuit

The cells below run two models on the same repeated sequence. The first is the base tiny GPT. The second is trained only on random patterns written twice.

Worked example

The base model on a repeated random sequence

Sixteen random content tokens, repeated. The loss per token on each half, then for every head the share of attention that lands exactly where the copy rule points: from a token in the second copy to the position just after its twin in the first copy.

import math
import torch
from tinygpt import load_pretrained

model, tok = load_pretrained("tiny-gpt")
L = 16
torch.manual_seed(1)
# ids 0-3 are the special tokens
pattern = torch.randint(4, len(tok.vocab), (1, L))
seq = torch.cat([pattern, pattern], dim=1)                  # the pattern, twice

def halves_and_heads(model, seq):
    x, y = seq[:, :-1], seq[0, 1:]
    with torch.no_grad():
        nll = -torch.log_softmax(model(x), dim=-1)[0, torch.arange(x.shape[1]), y]
        first, second = nll[:L - 1].mean().item(), nll[L:].mean().item()
        h = model.tok(x) + model.pos(torch.arange(x.shape[1]))
        rows = []
        for layer, blk in enumerate(model.blocks):
            a = blk.ln1(h)
            B, T, d = a.shape
            q, k, v = blk.qkv(a).split(d, dim=-1)
            q, k = (t.view(B, T, blk.h, d // blk.h).transpose(1, 2) for t in (q, k))
            s = (q @ k.transpose(-2, -1) / math.sqrt(d // blk.h)).masked_fill(
                ~torch.tril(torch.ones(T, T)).bool(), float("-inf"))
            A = torch.softmax(s, dim=-1)[0]                   # heads x T x T
            for head in range(blk.h):
                score = sum(A[head, t, t - L + 1].item()
                            for t in range(L, 2 * L - 1)) / (L - 1)
                rows.append(f"layer {layer} head {head}: {score:.2f}")
            h = blk(h)
    return first, second, rows

first, second, rows = halves_and_heads(model, seq)
print(f"loss per token: first copy {first:.2f}, second copy {second:.2f}")
print("induction score per head:", ", ".join(rows))
It runs right here in your browser, with nothing to install.

The base model's loss is no better on the second copy than on the first: 12.75 against 13.20 nats, both far above even uniform guessing, because random content is as alien to this model as a sentence in another language. No head puts more than 0.06 of its attention where the copy rule points. Trained on 300 product descriptions in which nothing repeats this way, it never had a reason to learn copying, and it cannot do it.

Worked example

A model trained to copy

The same measurement on tiny-gpt-induction, a model with the base model's architecture trained only on random patterns of changing length, each repeated once. Both scores for every head in both layers.

import math
import torch
from tinygpt import load_pretrained

model, tok = load_pretrained("tiny-gpt-induction")
L = 16
torch.manual_seed(1)
pattern = torch.randint(4, len(tok.vocab), (1, L))
seq = torch.cat([pattern, pattern], dim=1)
x, y = seq[:, :-1], seq[0, 1:]
with torch.no_grad():
    nll = -torch.log_softmax(model(x), dim=-1)[0, torch.arange(x.shape[1]), y]
    print(f"loss per token: first copy {nll[:L - 1].mean().item():.2f}, second copy "
          f"{nll[L:].mean().item():.2f}")
    h = model.tok(x) + model.pos(torch.arange(x.shape[1]))
    for layer, blk in enumerate(model.blocks):
        a = blk.ln1(h)
        B, T, d = a.shape
        q, k, v = blk.qkv(a).split(d, dim=-1)
        q, k = (t.view(B, T, blk.h, d // blk.h).transpose(1, 2) for t in (q, k))
        s = (q @ k.transpose(-2, -1) / math.sqrt(d // blk.h)).masked_fill(
            ~torch.tril(torch.ones(T, T)).bool(), float("-inf"))
        A = torch.softmax(s, dim=-1)[0]
        for head in range(blk.h):
            prev = sum(A[head, t, t - 1].item()
                       for t in range(1, 2 * L - 1)) / (2 * L - 2)
            ind = sum(A[head, t, t - L + 1].item()
                      for t in range(L, 2 * L - 1)) / (L - 1)
            print(f"layer {layer} head {head}: previous-token {prev:.2f}, induction "
                  f"{ind:.2f}")
        h = blk(h)
In the lesson you can change this and run it.

The second copy is now almost free: 5.45 nats on the first copy, where nothing can be predicted, and 0.21 on the second, against 12.75 for the base model. The heads tell a subtler story. The strongest single signal is layer 0 head 1, which puts 0.19 of its attention where the copy rule points, three times the base model's best, but nowhere near the scores close to 1 that single induction heads reach in large models, and no head is a clean previous-token head either. In 40,195 parameters the job is spread: several heads carry a share, the feed-forward layers finish the lookup, and the behavior is sharp even though no single head is. Same architecture and size as the base model, different training data, and the data was enough to build a copying machine, just not a tidy one.

Quick check
In the copying model's table, the best induction score is 0.19 while the loss on the second copy is 0.21 nats. Which reading is right?
  1. The rule is implemented, but spread over several heads and the feed-forward layers, not in one head
  2. Layer 0 head 1 is a complete induction head, and 0.19 is the highest score the measure allows
  3. The rule is mostly absent: a best score of 0.19 means the model is barely copying at all
  4. The loss figure must be wrong, since only a head scoring near 1 can produce a loss near 0
Show the answer

The rule is implemented, but spread over several heads and the feed-forward layers, not in one head

A loss of 0.21 nats on random tokens means the second copy is predicted almost perfectly; that is behavior, and it is sharp. The head scores are a probe for one clean implementation, and this small model found a messier one.

Watch it form

tiny-gpt-induction was trained from scratch on one kind of sequence: a random pattern of 4 to 28 content tokens, written twice. Its loss on the first copy can never fall; random tokens have no structure. Its loss on the second copy falls only once the model implements the rule, and when it does, it falls for every pattern length it trained on at once, because the rule does not care how far back the twin is. In the training log that shows as a plateau at the uniform loss followed by a drop, the signature that first drew researchers to these heads in real models.

The payoff is that the rule is general in content. The model never saw a sentence, yet it copies one. The next cell gives it a product sentence repeated, with a material that is false so that copying and knowledge disagree, and a made-up mapping it has never seen; the base model gets the same prompts.

Worked example

Copying something it never saw

Two prompts to both models. The sentence repeats "Our Kestrel tent is made from enamel", a material no tent in the catalog has; the mapping pairs products with materials in a format the models never trained on. The top prediction and its probability.

import torch
from tinygpt import load_pretrained

base, tok = load_pretrained("tiny-gpt")
copier, _ = load_pretrained("tiny-gpt-induction")

def top(model, text):
    with torch.no_grad():
        p = torch.softmax(model(torch.tensor([tok.encode(text)]))[0, -1], dim=-1)
    return f"{tok.vocab[int(p.argmax())]} {p.max().item():.2f}"

for text in ["Our Kestrel tent is made from enamel . Our Kestrel tent is made from",
             "tent silnylon mug enamel tent"]:
    print(f"{text!r}")
    print(f"   base:   {top(base, text)}")
    print(f"   copier: {top(copier, text)}")
In the lesson you can change this and run it.

On the repeated sentence the copier predicts "enamel" with 0.95: the current token is "from", its earlier twin was followed by "enamel", so that is what it copies, catalog or no catalog. The base model answers "silnylon" (0.58) from its training data and ignores the sentence in front of it. On the made-up mapping, "tent silnylon mug enamel tent", the copier predicts "silnylon" with 0.96, the token that followed the earlier "tent": a pairing it learned from the prompt alone, while the base model offers a comma (0.70). That is in-context learning in its smallest form, and the base model, with no copying circuit, cannot do it.

Quick check
During the copying model's training, the loss on the first copy of each pattern stays at about 5.25 nats while the loss on the second copy falls toward zero. Why can the first half never improve?
  1. The first half is too short for attention to find any useful pattern in it
  2. The learning rate is too low for the first half and too high for the second
  3. The tokens are drawn at random, so no rule predicts a token the first time it appears
  4. The model is not trained on the first half, so its loss there is never pushed down
Show the answer

The tokens are drawn at random, so no rule predicts a token the first time it appears

Only structure can be learned. The first copy has none; the second copy's structure is 'the same as the first', and that is what the heads learn to exploit.

Drill

Score one head

Write induction_score(A, L): A is a square NumPy array of attention weights for one head (rows attend to columns) over a sequence of length 2·L − 1 that holds a pattern of length L followed by the pattern again (minus its last token). Return the average, over the positions t from L to 2L − 2, of the attention from t to position t − L + 1, the token after t's twin.

import numpy as np

def induction_score(A, L):
    return 0.0

L = 3
A = np.zeros((5, 5))
for t in range(1, 5):
    A[t, t - 1] = 1.0            # a head that only looks at the previous token
print(round(induction_score(A, L), 2))
In the lesson, your code is checked as you write it.

Plot the loss at each position for the copying model on one 16-token pattern written twice. The first copy sits near the uniform loss, because nothing can be predicted yet. After the second copy starts, the loss falls to almost nothing.

import torch
import matplotlib.pyplot as plt
from tinygpt import load_pretrained

model, tok = load_pretrained("tiny-gpt-induction")
L = 16
torch.manual_seed(1)
pattern = torch.randint(4, len(tok.vocab), (1, L))
seq = torch.cat([pattern, pattern], dim=1)
x, y = seq[:, :-1], seq[0, 1:]
with torch.no_grad():
    nll = -torch.log_softmax(model(x), dim=-1)[0, torch.arange(x.shape[1]), y]
plt.plot(range(1, 2 * L), nll.tolist(), marker="o", label="loss at each position")
plt.axvline(L + 0.5, linestyle="--", label="second copy starts")
plt.xlabel("position")
plt.ylabel("loss (nats)")
plt.legend()
print(round(nll[:L - 1].mean().item(), 2), round(nll[L:].mean().item(), 2))
In your own words
In your own words: why does the loss on the first copy stay high while the loss on the second copy falls?

Write a sentence or two, then compare it with a good answer in the lesson.

In production: what copying justifies

From copying to learning in context

Exact copying is the clearest case, and large models go further. Their induction heads match on similar tokens as well as identical ones, and they operate on structure: a label that followed one kind of example is predicted after another example of the same kind. Give a model three lines of "review → sentiment" and it produces a sentiment after the fourth review; the heads that do this are the same circuit, matching the pattern "review, arrow" and copying the kind of token that followed it. Research on large models finds that the appearance of induction heads during training coincides with the appearance of in-context learning, and that removing them removes most of it.

The circuit also explains the limits. It copies what is in the context, including mistakes: an example with the wrong label teaches the wrong mapping for the rest of the prompt. It needs the twin to be in the context, so it is bounded by the window and weakened by the dilution of long prompts. And it is a pattern-matcher, not a reasoner: it reproduces the shape of what it saw.

Quick check
From the examples-in-prompts lesson: why do a few worked examples steer the output more reliably than a description of the format?
  1. The model copies the pattern the examples show, which is exactly what its induction circuit does
  2. The model stores the examples in its weights for the rest of the conversation
  3. Examples are shorter than instructions, so they fit in the context more easily
  4. Instructions are ignored by language models unless they come with examples
Show the answer

The model copies the pattern the examples show, which is exactly what its induction circuit does

A worked example is a pattern in the context, and copying patterns from the context is the machinery the model has. Instructions have to be translated into behavior; examples are the behavior.

Drill

The copy rule as a function

Write copy_predict(tokens): find the most recent earlier occurrence of the last token and return the token that followed it. Return None when the last token has not appeared before. Search from the end, so the nearest twin wins.

def copy_predict(tokens):
    return None

print(copy_predict(["tent", "silnylon", "mug", "enamel", "tent"]))
In the lesson, your code is checked as you write it.
In your own words
A few-shot prompt contains one example with a wrong label. What does the copying circuit do with it, and why?

Write a sentence or two, then compare it with a good answer in the lesson.

What this justifies

Few-shot examples: consistent, correct, close. The circuit copies format and mapping, so every example must share one format, every label must be right, and the examples should sit near the question, inside the budget the long-prompts lesson measured.

Grounding depends on copying. Retrieval puts the fact in the context; the induction circuit is what moves it into the answer. The confident-and-wrong lesson showed a model without the circuit ignoring a fact placed right in front of it. Production models have the circuit, which is why grounding works for them and why it weakens exactly where copying does: far from the question, in long prompts.

Expect pattern matching, not understanding. A model that copies "tent → silnylon" from the prompt has learned nothing about tents. In-context learning is powerful because patterns carry a great deal, and limited for the same reason.

Worked example

Three pairs, then a question

A longer made-up mapping on the copying model: three product-material pairs, then one product repeated. The top prediction and the share the correct material gets. Watch the third line.

import torch
from tinygpt import load_pretrained

copier, tok = load_pretrained("tiny-gpt-induction")
pairs = "tent silnylon mug enamel pants organic"
for product, material in [("tent", "silnylon"), ("mug", "enamel"),
                          ("pants", "organic")]:
    with torch.no_grad():
        p = torch.softmax(
            copier(torch.tensor([tok.encode(f"{pairs} {product}")]))[0, -1], dim=-1)
    print(f"{pairs} {product} -> {tok.vocab[int(p.argmax())]} (p of {material}: "
          f"{p[tok.token_id(material)].item():.2f})")
In the lesson you can change this and run it.

"tent" recalls silnylon with 0.95 and "mug" recalls enamel with 0.94: the model finds the earlier twin, six and four tokens back, and copies what followed it. Nothing about tents or enamel was learned here; it learned a rule about repeats, and pairs in a prompt are repeats. The third line fails: for "pants" the twin is only two tokens back, and the correct material gets 0.00. The training patterns were 4 to 28 tokens long, so the model never saw a twin closer than four positions and never learned to look there. The rule is general in content but was learned only over the distances the training data covered, a limit that large models, trained on text with repeats at every distance, do not share.

In your own words
The copying model copies sentences and mappings it never saw, yet misses a twin two tokens back. What did it learn, and what did it not learn?

Write a sentence or two, then compare it with a good answer in the lesson.

Quick check
The copy rule applied to a few-shot prompt. What does this print?
tokens = ["ok", "good", "slow", "bad", "fast", "good", "slow"]
current = tokens[-1]
earlier = [i for i, t in enumerate(tokens[:-1]) if t == current]
print(tokens[earlier[-1] + 1], len(earlier))
  1. bad 1
  2. fast 1
  3. slow 2
  4. good 1
Show the answer

bad 1

The last token is "slow"; it appeared once before, at index 2, followed by "bad". The circuit would predict "bad", the label the prompt attached to "slow".

In your own words
Why does grounding work on production models but not on the base tiny GPT, and what does that tell you about where in a prompt to put retrieved text?

Write a sentence or two, then compare it with a good answer in the lesson.

Practice what you just read

  • 5 code examples you can edit and run
  • a practice exercise that checks your code
  • a 5-question quiz
  • 9 review cards for spaced repetition

Try a quiz question

What does an induction head do?

  1. Attends from the current token to the position after that token's earlier occurrence and copies it
  2. Attends to the previous token and copies it forward, one position at a time
  3. Counts how many times each token has appeared and predicts the most frequent
  4. Attends to the first token of the sequence, where documents state their subject

Answer it at the end of the lesson to see if you're right, and why.

Before this lesson

Go deeper, if you like

Optional lessons that build on this one. They never block a phase or a certificate.

Keep learning

Where this lesson leads.

575 lessons in thirteen phases. 176 are free without an account, and a free account opens the rest. See every lesson.

Phase 6 · about 35 hours of study

Modern LLM engineering

Post-training, fine-tuning, serving and safety 48 lessons 2 free
  1. Base models vs assistants FreeWhat post-training adds to a pretrained model.30 min
  2. Supervised fine-tuningTeach behavior from example conversations.40 min
  3. Instruction and preference dataThe datasets that shape assistant behavior.30 min
  4. RLHFReward models and reinforcement learning from human feedback.40 min
  5. DPO and direct preference methodsPreference training without a separate reward model.40 min
  6. Constitutional AITraining with AI feedback guided by written principles.35 min
  7. Training reasoning modelsReinforcement learning on verifiable problems.40 min
  8. Prompting vs RAG vs fine-tuningPick the cheapest approach that works.30 min
  9. LoRAFine-tune a big model by training small low-rank updates.40 min
  10. Fine-tune an open modelRun a real fine-tune and evaluate the result.45 min
  11. DistillationTrain a small model to imitate a large one.35 min
  12. Tuning an embedding model for retrievalWhen search fails on your own vocabulary: train an adapter with a contrastive loss on question and passage pairs, test on held-out passages, use batch size and hard negatives, and measure what tuning costs elsewhere.45 min
  13. QuantizationStore weights in fewer bits to run models cheaply.35 min
  14. Quantization in practice: zero points, calibration and formatsScale and zero point, per-channel scales, calibration sets, error on a layer's output, quantization-aware training and the file formats you meet at work.35 min
  15. Serving modelsBatching, throughput and latency in production.35 min
  16. One request through a serving stack: latency and costFollow one request through queue, prefill, a prefix cache, decode and the stream, and price a token from a GPU's hourly cost.35 min
  17. Speculative decodingDraft with a small model, verify with a big one.35 min
  18. Running open models locallyDownload, quantize and run a model on your own machine.30 min
  19. Will it fit? Memory for serving and fine-tuningWeights, KV cache, runtime and adapter states added up for a model card and a GPU, with a calculator.30 min
  20. InterpretabilityLooking inside models to see what they compute.40 min
  21. Confident and wrongWhy a model answers with the same assurance whether it is right, wrong or asked about something that does not exist.30 min
  22. Why it can't spell or count lettersThe model sees token IDs, and an embedding learned from usage carries nothing about spelling.25 min
  23. Copying from the promptIn-context learning as a two-head copying circuit: the base tiny GPT cannot copy a repeated pattern, a model trained on repeats grows an induction head and can.30 min
  24. Why temperature changes the answerThe same prompt at 0, 0.7 and 1.5: deterministic loops, plausible variety, broken grammar, measured on a real model.25 min
  25. Why long prompts diluteAttention is a budget shared by every token, so a fact far from the question loses its share; measured, with the cliff beyond the trained window.25 min
  26. Why it never says "I don't know"A chat model answers every request because its fine-tuning data always did; preference training adds flattery; how to build abstention in.25 min
  27. Benchmarks and their limitsWhat leaderboards measure, and what they miss.38 min
  28. Alignment and safetyMaking capable models behave as intended.35 min
  29. Evaluating agentsMeasure multi-step, tool-using behavior.35 min
  30. Reading AI papersGet the idea out of a research paper in 30 minutes.30 min
  31. Capstone: a tuned, evaluated assistantFine-tune, evaluate and deploy a specialist assistant.45 min
  32. Weight-space programming: what it is, and what has been shown Free OptionalA language model reads and writes another network's weights with no training loop: the published experiments, their limits and the criticism, and one edit you make by hand.30 min
  33. Anatomy of a checkpoint OptionalA model file is a set of named tensors: count them, find the ones that write to the residual stream, and save one to bytes and back.35 min
  34. Reading weights without running the model OptionalSingular values, and an attention head's token-to-token circuit, read from the matrices alone and then checked against a measurement.35 min
  35. Writing weights by hand OptionalSet a small network's 55 weights so it catches every ball in a paddle game, the way a program would be written, and meet compiled transformers.40 min
  36. Edits are deltas: task vectors, patches and hashes OptionalSubtract a base from a fine-tune and the edit becomes an object: scale it, take it away, ship it as a patch, and prove by hash what was applied.35 min
  37. Merging models OptionalAverage two fine-tunes, add their task vectors, try SLERP, TIES and DARE, measure the interference, and compare with a model trained on both tasks.40 min
  38. Low-rank edits OptionalCut a fine-tune's change down to its largest directions with the SVD: how few a behavior needs, and how to pull an adapter out of a delta.35 min
  39. Directions: steer the run or edit the weights OptionalFind a behavior's direction in the residual stream, add it at run time, project it out of the weights, and see the effect change from input to input.40 min
  40. Editing one fact OptionalA rank-one change to one matrix rewrites what a product is made from; then reliability, generalization, locality, and what breaks after many edits.40 min
  41. The editor's harness OptionalWhat an LLM that edits weights is given and what it is denied: weights as text and their cost in tokens, a sandbox, a budget of forward passes, and rules the harness enforces.40 min
  42. An LLM reads a network's weights OptionalSix recorded sessions: an LLM is handed 1,315 weights as text and asked what the network was trained to do. Score each answer against the network, and study one it got wrong.40 min
  43. An LLM fixes a model's weights OptionalFind a tiny GPT's loop by hand, then replay recorded sessions in which an LLM hunts for it with ten forward passes, and measure every fix on held-out questions.45 min
  44. An LLM writes a network's weights OptionalFive recorded sessions: an LLM writes every weight of a paddle network from the game's source and one sentence. Score each against its specification, test its design claims, and edit one by hand.40 min
  45. Did it work? Testing an edit OptionalReview a patch for a tiny GPT that can't stop talking: held-out questions, a paired comparison, a same-size control, and what the edit cost.40 min
  46. The published edit, on a real model OptionalRead the edit published for an open model of 870 million weights, apply and verify one in the same form, see what 16-bit and 4-bit storage do to a fix, and run the real thing from a notebook.45 min
  47. Safety, licenses and provenance OptionalBefore a custom model is released: trace its lineage by hash, work out what its parents' licenses pass on, see what an edit can do to a safeguard, and write its card.40 min
  48. Capstone: ship a custom model OptionalSet release gates, then diagnose, edit, measure and document a tiny GPT with a fault no lesson has fixed, and write the record a stranger could check.45 min
Part of a free course 6 lessons

Why Models Do That: six behaviors, traced to the mechanism

Confident wrong answers, miscounted letters, copying from the prompt, temperature, long prompts that dilute, and a model that never says "I don't know": each behavior reproduced on a tiny GPT with every number printed by code, then the mechanism behind it and the Phase 1 technique it justifies.

Never trained an AI? Teach a small one to write, right in your browser. It takes about 20 seconds and needs no account. Teach it to write