In Phase 1 you met hallucinations: fluent, confident, wrong. You learned to ground answers in retrieved text and to ask for structured output with an explicit "unknown". This lesson shows why those techniques are needed, by reproducing the behavior on the course's own tiny GPT and measuring it: the model's top probability is about the same whether it is right or wrong, and a product that does not exist gets a material with the same assurance as a real one.
Knowing the cause changes how you build. You stop asking a model whether it is sure, you stop reading a high probability as knowledge, and you design the check into the system instead: a source in the prompt, an "unknown" option, an evaluation set. Those are the tools from Phase 1; here is the mechanism behind them.
What you'll be able to do
Reproduce a confident wrong answer on a real model and read the probability it puts on it
Explain why next-token prediction has no place for "I don't know"
Measure how little a model's top probability tells you about whether it is right
Say where the confidence comes from, and which Phase 1 techniques attack the cause
Quick check
A model that describes outdoor products is asked for the material of a product that does not exist. Compared with the probability it puts on its answer for real products, a guess: the top probability for the invented product will be
Spread evenly over every word in the vocabulary
Exactly zero: the model declines
About the same, sometimes higher
Much lower, close to zero
Show the answer
About the same, sometimes higher
The model has no way to notice that the product is new. It falls back on what products of that type are usually made of, and that pattern can be sharper than any single real product's. You will measure this in a moment.
The idea: fluent, confident, wrong
Purpose. Why a next-token model has no way to say "none of these", and what its top probability does and does not mean.
In the hallucination lesson you saw a model invent a court case, a citation, a product feature, in the same even tone it uses for facts. The usual advice followed: ground the answer in retrieved text, ask for a structured output with an "unknown" option, measure with an evaluation set. All of that works. None of it says why the model does this in the first place.
The why is simple once you look at the mechanism. A language model is a next-token predictor. For every prompt it produces a probability for every token in its vocabulary, and those probabilities sum to 1. It was trained to put probability on whatever token came next in its training text. Nowhere in that objective is there a step that checks a claim against the world, and nowhere is there an output that means "none of these".
This lesson reproduces the behavior on the course's tiny GPT: the 2-layer model with 40,195 parameters, trained on 300 product descriptions, that Interpretability took apart. It is small enough to measure completely, and it behaves exactly like its big cousins here.
Worked example
A real product and an invented one
Two prompts in the catalog's own phrasing. The River sleeping bag exists (its description says "made from a synthetic insulation"); the Kestrel umbrella does not. For each, the three most likely next tokens with their probabilities.
import torch
from tinygpt import load_pretrained
model, tok = load_pretrained("tiny-gpt")
deftop3(prompt):
with torch.no_grad():
p = torch.softmax(model(torch.tensor([tok.encode(prompt)]))[0, -1], dim=-1)
best = p.topk(3)
return", ".join(f"{tok.vocab[i]} {v:.2f}"for v, i inzip(best.values.tolist(), best.indices.tolist()))
print("real: ", top3("Our River sleeping bag is made from"))
print("invented:", top3("Our Kestrel umbrella is made from"))
print("unknown words in the second prompt:",
[w for w in tok.tokenize("Our Kestrel umbrella is made from")
if w notin tok.stoi])
It runs right here in your browser, with nothing to install.
For the real product the model's favorite is "responsibly" at 0.70, the start of "responsibly sourced down", a phrase that follows "made from" in many descriptions. The truth, "a synthetic insulation", gets 0.27 through its first token "a". For the umbrella, a product type the model has never seen (the word is not even in its vocabulary, so it arrives as <unk>), the model is more sure: 0.95 on "a". It is not hedging. It is completing the most common pattern after "made from" with nothing to pull it elsewhere.
Quick check
Softmax turns the model's scores into probabilities. Before running it, what does this print?
Three equal scores give three equal shares of one third, and the shares sum to 1. Whatever the scores, softmax hands back a complete distribution: there is never a leftover share for "none of these".
No place for "I don't know"
Look at what the training objective rewards. For every position in the training text, the loss is the cross-entropy between the model's distribution and the token that actually came next (Surprise, entropy and cross-entropy). The model lowers the loss by putting probability on the tokens that tend to follow such a context. It never sees a signal that says "this context is about something that does not exist"; the training text only ever describes things that do.
So at inference, when the context is "Our Kestrel umbrella is made from", the model does the only thing it can: it estimates which tokens followed "is made from" in similar contexts. The estimate is a full distribution over all 195 tokens. The one labeled <unk> exists only because the tokenizer needs somewhere to put unknown words; it is not an answer. There is no token for "I have no information about this product", and no part of the computation that would produce it.
The probability the model puts on its top token therefore measures one thing: how peaked the learned pattern is for contexts like this one. It does not measure whether the completion is true. The next step is to show that on every fact in the catalog, not on two hand-picked prompts.
Try it now
The model's shares for an invented camp mug (the logits are chosen to reproduce them: enamel 0.58, titanium 0.39). Drag one logit and watch the others move: the total never changes.
In your own words
In your own words: why does a next-token model answer a question about a product that does not exist, instead of saying it has no information?
Write a sentence or two, then compare it with a good answer in the lesson.
In code: measure it on 300 facts
Purpose. Ask the tiny GPT every fact in the catalog and compare how sure it is with how often it is right.
Every description in the catalog names a material: "Our Ridgeline tent is made from recycled ripstop nylon", "Meet the Harbor camp mug, made from enamel steel". That gives 300 facts of the form product → first word of its material. Ask the model each one in the same phrasing, take its top token and its probability, and record whether the top token was right.
Two numbers matter. The accuracy: how often the top token is the right word. And the confidence: the top probability, averaged separately over the answers that were right and the ones that were wrong. If the probability carried information about correctness, the second average would be clearly lower than the first.
Code flow. Before the full run, here is the idea on six answers: group them by confidence, then count how often each group was actually right.
Worked example
Confidence when right, confidence when wrong
All 300 catalog facts, asked in one phrasing. The code reads each description's material with a regular expression, asks the model, and compares its top token with the truth.
import re
import torch
import requests
from tinygpt import load_pretrained
model, tok = load_pretrained("tiny-gpt")
text = requests.get("https://api.gradient.test/v1/corpus").json()["text"]
pattern = re.compile(r"^(?:Meet the|Our|The) ([A-Z][a-z]+) (.+?)(?:,| is| are) made "r"(?:from|of|with) (?:a |an )?(\w+)")
facts = [m.groups() for d in text.strip().split("\n") if (m := pattern.match(d))]
right, wrong, pairs = [], [], []
for name, product, material in facts:
with torch.no_grad():
ids = torch.tensor([tok.encode(f"Our {name} {product} is made from")])
p = torch.softmax(model(ids)[0, -1], dim=-1)
top = tok.vocab[int(p.argmax())]
(right if top == material else wrong).append(float(p.max()))
pairs.append((float(p.max()), top == material))
print(f"{len(facts)} facts, top token right {len(right)} times "f"({len(right) / len(facts):.0%})")
print(f"average top probability when right {sum(right) / len(right):.2f}, when wrong "f"{sum(wrong) / len(wrong):.2f}")
print(f"wrong answers with a probability above 0.5: {sum(1 for p in wrong if p > 0.5)} "f"of {len(wrong)}")
import matplotlib.pyplot as plt
edges = [0.0, 0.5, 0.7, 0.9, 1.01]
shares = []
for lo, hi inzip(edges, edges[1:]):
inside = [ok for p, ok in pairs if lo <= p < hi]
shares.append(sum(inside) / len(inside) if inside else0.0)
print(f"confidence {lo:.1f} to {min(hi, 1.0):.1f}: {len(inside)} answers, "f"{shares[-1]:.0%} right")
mids = [(lo + min(hi, 1.0)) / 2for lo, hi inzip(edges, edges[1:])]
plt.bar(mids, shares, width=0.15)
plt.plot([0, 1], [0, 1], "k--", label="a model that knew when it was right")
plt.xlabel("top probability")
plt.ylabel("share of answers right")
plt.title("Confidence does not track accuracy")
plt.legend()
plt.show()
In the lesson you can change this and run it.
The model gets 137 of 300 right, 46%. Its average top probability is 0.63 when it is right and 0.68 when it is wrong: slightly higher for the wrong answers. Of the 163 wrong answers, 156 come with a probability above 0.5. Read a probability as "how sure the model is" and you would trust the wrong answers a little more than the right ones. The probability is real; it just measures the pattern, not the fact. Binned by confidence the pattern is stranger still: the 28 answers above 0.9 are all wrong, and the answers between 0.5 and 0.7 are right half the time. A reliable model would climb from left to right.
Quick check
From the hallucination lesson: which design move attacks this cause directly, rather than hoping the model will know when it does not know?
Put the relevant source text in the prompt and ask the model to answer from it
Ask the same question twice and keep the longer of the two answers it gives
Tell the model to be careful and honest, and to say when it is unsure
Raise the temperature so the model explores more answers before settling on one
Show the answer
Put the relevant source text in the prompt and ask the model to answer from it
Grounding changes the context the model predicts from. With the fact in the prompt, the learned pattern now points at the right continuation. The other moves change the sampling or the wording, not the information the prediction is based on.
Drill
Count the confident wrong answers
Write confidently_wrong(rows, threshold=0.5). Each row is (probability, right), the model's top probability and whether its top token was correct. Return how many rows were wrong and had a probability above the threshold.
In the lesson, your code is checked as you write it.
Where the confidence comes from
If the probability is not knowledge, what is it? It is the learned frequency of continuations in contexts like the prompt. The training text has twelve product types, and each type is made of a small set of materials. Camp mugs are enamel steel or titanium; tents are silnylon or recycled ripstop nylon; sleeping bags have "responsibly sourced down" or "a synthetic insulation". When the model meets a new product of a known type, its distribution over materials mirrors how often each material appeared with that type.
That is why an invented camp mug gets enamel about 0.6 and titanium about 0.4, the split in the widget. It is not a coin flip. It is the model reporting how the training text divides mugs between the two materials, with no information about this mug to move it, which is the honest shape for its uncertainty between known options. What it cannot do is report that the option "this mug does not exist" should get all the probability. The next cell checks the base-rate story against the training text.
Worked example
Base rates in the text, shares in the model
For three product types: how often each material follows "made from" in the training descriptions, next to the model's shares for an invented product of that type.
import re
from collections import Counter
import torch
import requests
from tinygpt import load_pretrained
model, tok = load_pretrained("tiny-gpt")
text = requests.get("https://api.gradient.test/v1/corpus").json()["text"]
pattern = re.compile(r"^(?:Meet the|Our|The) [A-Z][a-z]+ (.+?)(?:,| is| are) made "r"(?:from|of|with) (?:a |an )?(\w+)")
counts = {}
for d in text.strip().split("\n"):
m = pattern.match(d)
if m:
counts.setdefault(m.group(1), Counter())[m.group(2)] += 1for product in ["camp mug", "tent", "hiking pants"]:
seen = ", ".join(f"{w} {n}"for w, n in counts[product].most_common(2))
with torch.no_grad():
ids = torch.tensor([tok.encode(f"Our Orbit {product} is made from")])
p = torch.softmax(model(ids)[0, -1], dim=-1)
best = p.topk(2)
shares = ", ".join(f"{tok.vocab[i]} {v:.2f}"for v, i inzip(best.values.tolist(), best.indices.tolist()))
print(f"{product:<9} training text: {seen:<28} model for an invented {product}: "f"{shares}")
In the lesson you can change this and run it.
Each line pairs counts with shares, and they move together. Camp mugs: 14 enamel to 10 titanium in the text, 0.59 to 0.39 in the model. Tents: 14 silnylon to 11 recycled, 0.60 to 0.39. Hiking pants: 16 organic to 11 stretch, 0.61 to 0.38. The "confidence" on an invented product is the base rate of its type, read off the training data and sharpened a little. The name "Orbit" adds nothing, because the model has never seen it.
Quick check
For the invented Orbit camp mug the model gives enamel 0.59 and titanium 0.39. Which statement explains that?
The word "Orbit" is associated with metal, so the model picks metals
The model knows the Orbit mug and is unsure which of its two versions is meant
The training text has 14 enamel mugs and 10 titanium ones, and the model reports that split
The probabilities are random because the name is unknown
Show the answer
The training text has 14 enamel mugs and 10 titanium ones, and the model reports that split
The distribution over materials is the learned base rate for the product type. The unknown name contributes nothing, so what remains is the type's pattern.
Drill
Base rates from the descriptions
Write base_rates(docs, product): given a list of descriptions and a product type, return a Counter of the first material word that follows "made from" (or "made of" / "made with") in the descriptions of that product. Skip an article ("a" or "an") before the material. A description names its product type right after the brand name, as in "Our Ridgeline tent is made from".
import re
from collections import Counter
defbase_rates(docs, product):
return Counter()
docs = [
"Our Ridgeline tent is made from recycled ripstop nylon. It is roomy.",
"Meet the Kestrel tent, made from silnylon. It is light.",
"The Harbor tent is made from recycled ripstop nylon. It packs small.",
"Meet the Harbor camp mug, made from enamel steel. It is durable.",
"Our River sleeping bag is made from a synthetic insulation. It is warm.",
]
print(base_rates(docs, "tent"))
In the lesson, your code is checked as you write it.
In your own words
A colleague says: "The model gave 0.9, so it must know." Explain in two sentences what the 0.9 measures and what it does not.
Write a sentence or two, then compare it with a good answer in the lesson.
In production: what this justifies
Purpose. Each Phase 1 technique now has a reason, and the real tool to read a model's probabilities.
Each Phase 1 technique now has a reason.
Grounding. Retrieval puts the source text in the prompt, so the right continuation is sitting in the context. For that to help, the model must be able to read a fact off its context and copy it forward, which large models do through attention heads built for copying. The next cell tries it on the tiny GPT and finds the limit: with the Ridgeline description in front of the question, its answer does not improve. It has no copying machinery, and the lesson on copying from the prompt shows what that machinery is and how it forms. Grounding works on production models because they have it; it is not a property of any next-token predictor.
Structured output with "unknown". Since there is no built-in token for "no information", you add one: an output schema with an explicit unknown value, and instructions that make it the right completion when the source does not contain the answer. Post-training can teach a model to use such an option; the lesson on why models never say "I don't know" takes that up.
Evals, not confidence. Because the probability measures the pattern and not the truth, the only way to know how often a model is right on your task is to measure it on a labeled set, as the evals lesson does. Token log-probabilities, where a provider exposes them, help with known options (the mug's two materials) and fail as a hallucination detector.
Quick check
Which Phase 1 technique gives a model a completion to use when the source text does not contain the answer?
Reading the top probability and refusing whenever it falls below 0.5
A schema with an explicit unknown value, made the right answer when the source is silent
A longer prompt that repeats the question, so the model notices it is unsure
A higher temperature, so unlikely tokens get a chance to express doubt
Show the answer
A schema with an explicit unknown value, made the right answer when the source is silent
Since no token means 'no information', you add one: a schema field that can hold unknown, and training or instructions that make it the likely completion when the source is silent.
Worked example
Does the fact in the prompt help this model?
The Ridgeline tent is made from recycled ripstop nylon. Ask without context, then with the catalog sentence placed just before the question.
import torch
from tinygpt import load_pretrained
model, tok = load_pretrained("tiny-gpt")
defp_recycled(prompt):
with torch.no_grad():
p = torch.softmax(model(torch.tensor([tok.encode(prompt)]))[0, -1], dim=-1)
return p[tok.token_id("recycled")].item(), tok.vocab[int(p.argmax())]
cold = p_recycled("Our Ridgeline tent is made from")
grounded = p_recycled("Our Ridgeline tent is made from recycled ripstop nylon. It is ""roomy. The Ridgeline tent is made from")
print(f"cold: p(recycled) = {cold[0]:.2f}, top token {cold[1]}")
print(f"grounded: p(recycled) = {grounded[0]:.2f}, top token {grounded[1]}")
In the lesson you can change this and run it.
Cold, the model prefers "silnylon" and gives "recycled" 0.46: the two tent materials are close in the training text and the Ridgeline name does not settle it. With the description in the prompt, "recycled" falls to 0.34 and "silnylon" stays on top. The source is right there and the model cannot use it, because nothing in this 2-layer model looks back for "Ridgeline ... made from" and copies what followed. Retrieval-augmented generation depends on that copying ability, and the tiny GPT shows what a model without it does: it keeps completing the base-rate pattern.
Quick check
A calibration check compares average confidence with accuracy. What does this print?
rows = [(0.9, True), (0.8, False), (0.6, True), (0.7, False)]
confidence = sum(p for p, _ in rows) / len(rows)
accuracy = sum(1for _, right in rows if right) / len(rows)
print(round(confidence - accuracy, 2))
0.25
-0.25
0.5
0.75
Show the answer
0.25
Average confidence is 0.75 and accuracy is 0.5, so the model is overconfident by 0.25. On the catalog facts the gap was similar: an average top probability near 0.65 against 46% accuracy.
Where this shows up in an engineer's week
Someone proposes "ask the model how sure it is" or "reject answers below 0.8 probability". You answer with numbers from your own labeled set: accuracy by confidence bin, as the plot above. If the bins are flat, a threshold filters nothing. The fix is outside the model: a source in the prompt, an explicit unknown, and an eval that scores abstaining.
Grounding is the standard fix for hallucination, yet it did nothing for the tiny GPT. What must a model be able to do for a fact in the prompt to change its answer?
Write a sentence or two, then compare it with a good answer in the lesson.
On 300 catalog facts the tiny GPT's top token was right 46% of the time, with an average top probability of 0.63 when right and 0.68 when wrong. What does that show?
The top probability tells you almost nothing about whether the answer is right
The model is well calibrated: confidence and accuracy match
The model is right whenever its probability is above 0.5
A higher probability means the fact appeared more often in training, so it is more likely true
Answer it at the end of the lesson to see if you're right, and why.
11DistillationTrain a small model to imitate a large one.35 min
12Tuning an embedding model for retrievalWhen search fails on your own vocabulary: train an adapter with a contrastive loss on question and passage pairs, test on held-out passages, use batch size and hard negatives, and measure what tuning costs elsewhere.45 min
13QuantizationStore weights in fewer bits to run models cheaply.35 min
23Copying from the promptIn-context learning as a two-head copying circuit: the base tiny GPT cannot copy a repeated pattern, a model trained on repeats grows an induction head and can.30 min
24Why temperature changes the answerThe same prompt at 0, 0.7 and 1.5: deterministic loops, plausible variety, broken grammar, measured on a real model.25 min
25Why long prompts diluteAttention is a budget shared by every token, so a fact far from the question loses its share; measured, with the cliff beyond the trained window.25 min
26Why it never says "I don't know"A chat model answers every request because its fine-tuning data always did; preference training adds flattery; how to build abstention in.25 min
32Weight-space programming: what it is, and what has been shownFreeOptionalA language model reads and writes another network's weights with no training loop: the published experiments, their limits and the criticism, and one edit you make by hand.30 min
33Anatomy of a checkpointOptionalA model file is a set of named tensors: count them, find the ones that write to the residual stream, and save one to bytes and back.35 min
34Reading weights without running the modelOptionalSingular values, and an attention head's token-to-token circuit, read from the matrices alone and then checked against a measurement.35 min
35Writing weights by handOptionalSet a small network's 55 weights so it catches every ball in a paddle game, the way a program would be written, and meet compiled transformers.40 min
36Edits are deltas: task vectors, patches and hashesOptionalSubtract a base from a fine-tune and the edit becomes an object: scale it, take it away, ship it as a patch, and prove by hash what was applied.35 min
37Merging modelsOptionalAverage two fine-tunes, add their task vectors, try SLERP, TIES and DARE, measure the interference, and compare with a model trained on both tasks.40 min
38Low-rank editsOptionalCut a fine-tune's change down to its largest directions with the SVD: how few a behavior needs, and how to pull an adapter out of a delta.35 min
39Directions: steer the run or edit the weightsOptionalFind a behavior's direction in the residual stream, add it at run time, project it out of the weights, and see the effect change from input to input.40 min
40Editing one factOptionalA rank-one change to one matrix rewrites what a product is made from; then reliability, generalization, locality, and what breaks after many edits.40 min
41The editor's harnessOptionalWhat an LLM that edits weights is given and what it is denied: weights as text and their cost in tokens, a sandbox, a budget of forward passes, and rules the harness enforces.40 min
42An LLM reads a network's weightsOptionalSix recorded sessions: an LLM is handed 1,315 weights as text and asked what the network was trained to do. Score each answer against the network, and study one it got wrong.40 min
43An LLM fixes a model's weightsOptionalFind a tiny GPT's loop by hand, then replay recorded sessions in which an LLM hunts for it with ten forward passes, and measure every fix on held-out questions.45 min
44An LLM writes a network's weightsOptionalFive recorded sessions: an LLM writes every weight of a paddle network from the game's source and one sentence. Score each against its specification, test its design claims, and edit one by hand.40 min
45Did it work? Testing an editOptionalReview a patch for a tiny GPT that can't stop talking: held-out questions, a paired comparison, a same-size control, and what the edit cost.40 min
46The published edit, on a real modelOptionalRead the edit published for an open model of 870 million weights, apply and verify one in the same form, see what 16-bit and 4-bit storage do to a fix, and run the real thing from a notebook.45 min
47Safety, licenses and provenanceOptionalBefore a custom model is released: trace its lineage by hash, work out what its parents' licenses pass on, see what an edit can do to a safeguard, and write its card.40 min
48Capstone: ship a custom modelOptionalSet release gates, then diagnose, edit, measure and document a tiny GPT with a fault no lesson has fixed, and write the record a stranger could check.45 min
Confident wrong answers, miscounted letters, copying from the prompt, temperature, long prompts that dilute, and a model that never says "I don't know": each behavior reproduced on a tiny GPT with every number printed by code, then the mechanism behind it and the Phase 1 technique it justifies.
See the course →
Never trained an AI? Teach a small one to write, right in your browser. It takes about 20 seconds and needs no account. Teach it to write