After the last block, "The cat sat" is three rows of eight numbers, and the model has to turn that into one decision: which token comes next. Everything in this section has been leading here. One matrix turns the last row into a score for every entry in the vocabulary, a softmax turns the scores into probabilities, and a sampling rule picks one. Then the chosen token joins the prompt and the whole pipeline runs again.
This is also where the settings you meet in every API live: temperature, top-k and top-p are nothing more than small changes to the arithmetic in this lesson, and you'll compute each of them by hand.
What you'll be able to do
Project the last token's row to a logit for every vocabulary entry, and give the shapes involved
Turn logits into probabilities with softmax, and read a probability ratio off a logit gap
Apply a temperature to the logits and predict how the distribution sharpens or flattens
Pick the next token by argmax, sampling, top-k or top-p, and feed it back for the next step
The last row becomes a score for every token, softmax makes them probabilities and the chosen token joins the prompt; the boards show " on" at 0.813 and what temperature does to it. Transcript
Quick check
Two logits, 6.1 for " on" and 3.9 for " down", through a softmax. A guess: what does this print?
import numpy as np
z = np.array([6.1, 3.9])
e = np.exp(z - z.max())
print((e / e.sum()).round(2))
[0.61 0.39]
[0.5 0.5]
[6.1 3.9]
[0.9 0.1]
Show the answer
[0.9 0.1]
A gap of 2.2 in the logits is a factor of e^2.2 ≈ 9 in the probabilities, so the split is about 90/10. Softmax turns differences into ratios.
After the last block, the token that matters is the last one, " sat". Its row z∈Rdmodel (normalized once more by a final layer norm) is multiplied by the unembedding or LM head matrix:
logits=zWout+b∈RV,Wout∈Rdmodel×V
Eight numbers in, one raw score, a logit, for each of the V=1000 vocabulary entries out. GPT-2's head is 768 × 50,257, and it reuses the embedding matrix transposed (weight tying), so those 38.6 million numbers do double duty: the same row that encodes " cat" on the way in scores " cat" on the way out.
Only the last row is used when generating, because the next token comes after the last one. During training every row is used at once, each predicting the token after it, which is why one pass over a text gives a training signal at every position.
The example takes " sat"'s output row from the feed-forward lesson as z and uses a seeded random head, so its top entry is a random ID; the shapes are the point here. The rest of the lesson then uses logits with meaning.
Quick check
When a model generates text, only the last token's row goes through the LM head. Why that row?
The LM head can only multiply one row at a time, so it takes the last
The other rows are deleted after the last block to save memory
The first row would be too small to hold a prediction over the vocabulary
The last row has read the whole prompt and predicts the token after the end
Show the answer
The last row has read the whole prompt and predicts the token after the end
Each row predicts the token after its own position. At generation time the only unknown token is the one after the end, so the last row is the one that matters. During training every row is used, each predicting its successor.
Worked example
From the last row to V logits
The last token's row, a seeded 8 × 1000 head, and the resulting logits: one number per vocabulary entry.
import numpy as np
np.set_printoptions(precision=2, suppress=True)
# " sat"'s row after the last block
z = np.array([-1.23, -1.47, 0.79, -0.95, 0.67, 1.09, 1.84, -0.59])
V, d_model = 1000, 8
rng = np.random.default_rng(5)
W_out = rng.normal(0.0, 0.5, size=(d_model, V)).round(2)
b = np.zeros(V)
logits = z @ W_out + b
print(z.shape, "@", W_out.shape, "->", logits.shape)
top = np.argsort(-logits)[:3]
print("highest-scoring IDs:", top, "with logits", logits[top])
print("the logit of ID 511 (\" on\" in our vocabulary):", round(float(logits[511]), 2))
It runs right here in your browser, with nothing to install.
(8,) @ (8, 1000) -> (1000,): a thousand logits from eight numbers, one per token in the vocabulary. With random weights the top IDs mean nothing, and that's the honest picture of an untrained model: training is what moves the row for " on" to the top after "The cat sat". From here on we use logits a trained head might give.
Quick check
GPT-2 has d_model = 768 and V = 50,257. What are the shapes of its LM head and of the logits for one next-token prediction?
50,257 × 768 and (768,)
768 × 768 and (768,)
T × 50,257 and (T,)
768 × 50,257 and (50,257,)
Show the answer
768 × 50,257 and (50,257,)
The head maps d_model numbers to one score per vocabulary entry, so it has V columns, and the logits are a vector of length V.
Softmax: scores to probabilities
Logits are raw numbers on no particular scale. Softmax turns the whole vector into a probability distribution over the vocabulary:
pi=∑j=1Vezjezi,i∑pi=1
Every entry becomes positive, the largest logit gets the largest share, and only differences matter: adding the same constant to every logit changes nothing, which is why the code subtracts the maximum first. The ratio between two probabilities is the exponential of their logit gap:
pjpi=ezi−zj
Here are logits a trained head might give for eight candidates after "The cat sat". They're illustrative, chosen to show the shape of a real distribution: a clear favorite, a few plausible runners-up, and a long tail (the other 992 tokens, left out here, would share a fraction of a percent).
Token
" on"
" down"
" there"
" quietly"
" by"
" under"
" in"
" up"
Logit
6.1
3.9
3.2
2.4
2.0
1.7
1.1
0.6
The gap between " on" and " down" is 2.2, so " on" is e2.2≈9 times as likely.
Worked example
Eight candidates after "The cat sat"
Softmax over the eight logits, the probability ratio from the top gap, and the check that the shares sum to 1.
" on" takes 0.813, " down" 0.090, " there" 0.045, and the rest share the remaining 5%. The ratio 0.813 / 0.090 is 9.03, exactly e2.2. The shares sum to 1 because the softmax divides by the total.
Quick check
The top two logits are 6.1 and 3.9. How many times as likely is the top token? What does this print?
import numpy as np
print(round(float(np.exp(6.1 - 3.9)), 2))
0.9
2.2
1.56
9.03
Show the answer
9.03
The ratio of two softmax probabilities is e to the gap between their logits: e^2.2 = 9.03.
Try it now
The eight candidates. Drag a logit and watch every share move, since they always sum to 1; then move the temperature.
Temperature
Temperature divides every logit by a number T before the softmax:
pi(T)=∑jezj/Tezi/T
Dividing by T scales every gap by 1/T. With T<1 the gaps grow and the favorite takes more; as T→0 the distribution becomes a single spike on the largest logit, the same as always picking the top token. With T>1 the gaps shrink and the probability spreads toward the tail. For our eight candidates, " on" gets 1.000 at T=0.1, 0.984 at 0.5, 0.813 at 1, 0.610 at 1.5 and 0.474 at 2. The order never changes; only the margins do.
One way to put a number on the spread is entropy, in bits: 0.0 at T=0.1, 0.14 at 0.5, 1.09 at 1, 1.92 at 1.5, 2.37 at 2. Higher means the draw is less predictable.
The bar in the video reads 98%: that's this distribution at a temperature of 0.5. Temperature is set by the caller, not learned, which is why the same model can sound careful at 0.3 and inventive at 1.2.
Worked example
The same logits at five temperatures
The top candidate's share and the entropy of the whole distribution, as T rises.
import numpy as np
words = [" on", " down", " there", " quietly", " by", " under", " in", " up"]
logits = np.array([6.1, 3.9, 3.2, 2.4, 2.0, 1.7, 1.1, 0.6])
defwith_temperature(logits, T):
e = np.exp(logits / T - (logits / T).max())
return e / e.sum()
for T in (0.1, 0.5, 1.0, 1.5, 2.0):
p = with_temperature(logits, T)
nz = p[p > 0]
entropy = float(-(nz * np.log2(nz)).sum())
print(f"T = {T:3.1f}: on {p[0]:.3f} down {p[1]:.3f} "f"there {p[2]:.3f} up {p[7]:.3f} entropy {entropy:.2f} bits")
In the lesson you can change this and run it.
At T=0.1 everything piles onto " on". At T=2 " on" keeps less than half, " down" rises to 0.158 and even " up" reaches 0.030. The entropy climbs from 0 to 2.37 bits as the draw gets harder to predict.
Quick check
At T = 0.1 the top probability prints as 1.000. Why?
The logits themselves are changed by training at low temperature, raising the top one
The softmax stops dividing by the sum at low temperatures, so the top stays at 1
Temperature 0.1 removes all but the top token from the vocabulary before the softmax
Dividing by 0.1 turns the 2.2 gap into 22, a ratio of e^22, so the runner-up rounds to 0
Show the answer
Dividing by 0.1 turns the 2.2 gap into 22, a ratio of e^22, so the runner-up rounds to 0
Temperature only rescales the logits before the softmax. A tenfold gap in logits is a ratio of e^22 ≈ 3.6 billion in probability, far below three decimals.
Choosing the token, and the loop
A probability distribution isn't a token yet. Four rules are in common use:
Greedy (argmax): take the largest probability, " on". Deterministic, and prone to repeating itself in long outputs.
Sampling: draw a token at random with the given probabilities. " on" about 81 times in 100, " down" about 9.
Top-k: keep only the k largest, renormalize, then sample. With k=3: " on" 0.857, " down" 0.095, " there" 0.047.
Top-p (nucleus): keep the smallest set of tokens whose probabilities add up to at least p, then sample. With p=0.9: " on" + " down" = 0.903 ≥ 0.9, so those two survive, renormalized to 0.900 and 0.100.
kept = {" on": 0.813, " down": 0.090} # the smallest set reaching p = 0.9
total = sum(kept.values())
print({tok: round(p / total, 3) for tok, p in kept.items()})
Whichever rule picks it, the chosen token's ID (511 for " on") is appended to the sequence. "The cat sat" becomes "The cat sat on", T goes from 3 to 4, and the whole pipeline runs again from the tokenizer's output: new embeddings, a 4 × 4 attention matrix, four rows through every block, and the LM head on the new last row. That repeat is the autoregressive loop, one token per pass, until a stop token or a length limit. A real server avoids redoing the old rows' keys and values each time; that's the KV cache, in the inference lesson.
In your own words
What does temperature change about the distribution, and what does it leave alone?
Write a sentence or two, then compare it with a good answer in the lesson.
In PyTorch: the head on a trained model
The tiny GPT has the same last step as GPT-2: a final norm, a linear head, and a softmax over the last row. Here is the loop, written out, on a model whose answers mean something.
Drill
Logits to probabilities
Write softmax_probs(logits) that returns the probabilities for a NumPy array of logits. Subtract the maximum before exponentiating, so very large logits can't overflow.
In the lesson, your code is checked as you write it.
Drill
Write the temperature softmax
Write with_temperature(logits, T) that divides the logits by T and returns the softmax probabilities.
import numpy as np
defwith_temperature(logits, T):
e = np.exp(logits - logits.max())
return e / e.sum()
z = np.array([6.1, 3.9, 3.2, 2.4, 2.0, 1.7, 1.1, 0.6])
print(with_temperature(z, 2.0).round(3))
In the lesson, your code is checked as you write it.
Worked example
The last row, the chances and a five-token loop
model(ids) gives logits for every position. Only the last row is used. The loop appends the likeliest token five times.
import torch
import torch.nn.functional as F
from tinygpt import load_pretrained
model, tok = load_pretrained("tiny-gpt")
ids = torch.tensor([tok.encode("The Lark headlamp is")])
logits = model(ids)
last = logits[:, -1, :]
print(tuple(logits.shape), "->", tuple(last.shape))
p = F.softmax(last, dim=-1)[0]
top = torch.topk(p, 3)
pairs = zip(top.values, top.indices)
print([(tok.vocab[int(i)], round(float(v), 3)) for v, i in pairs])
out = ids
for step inrange(5):
nxt = model(out[:, -model.context:])[:, -1, :].argmax(dim=-1, keepdim=True)
out = torch.cat([out, nxt], dim=1)
print(step + 1, tok.vocab[int(nxt)], tuple(out.shape))
print(tok.decode(out[0].tolist()))
same = model.generate(ids, 5, temperature=0)
print("generate agrees:", torch.equal(same, out))
In the lesson you can change this and run it.
The prompt is four tokens, so the logits are (1, 4, 195): a score for each of the 195 tokens at each of the four positions, and the last row is (1, 195). The softmax gives " made" 0.986 and almost nothing to the rest: the model has seen this phrase in its catalog. Each loop pass appends one token and the shape grows from (1, 5) to (1, 9): made, from, a, light, polymer. generate with temperature=0 runs the same loop.
Worked example
Temperature on the model's own logits
The same last row, divided by three temperatures before the softmax. The top chance falls as the temperature rises.
import torch
import torch.nn.functional as F
from tinygpt import load_pretrained
model, tok = load_pretrained("tiny-gpt")
ids = torch.tensor([tok.encode("The Lark headlamp is")])
last = model(ids)[:, -1, :]
for T in (0.5, 1.0, 2.0):
q = F.softmax(last / T, dim=-1)[0]
print(T, "top chance", round(float(q.max()), 3), tok.vocab[int(q.argmax())])
In the lesson you can change this and run it.
At temperature 0.5 the favorite " made" has a chance of 1.0 (rounded), at 1 it has 0.986 and at 2 it falls to 0.591. The ranking never changes, only the margins. For real outputs this is the knob: below 1 the model repeats its best guess, above 1 it wanders.
Quick check
From the sampling lesson: with p = 0.9 on these probabilities, 0.813, 0.090, 0.045, 0.020, …, which tokens does top-p keep?
" on" alone, since 0.813 is by far the largest single share
The top three, because 0.9 rounds up to three tokens in a list of eight
Top-p adds tokens from the largest down until the running total reaches p. 0.813 falls short of 0.9; adding 0.090 passes it, so two tokens survive.
Worked example
Greedy, top-k and top-p on the eight candidates
The four rules on the same probabilities, with a seeded draw so the sampled token is repeatable.
import numpy as np
words = [" on", " down", " there", " quietly", " by", " under", " in", " up"]
logits = np.array([6.1, 3.9, 3.2, 2.4, 2.0, 1.7, 1.1, 0.6])
e = np.exp(logits - logits.max()); probs = e / e.sum()
print("greedy:", words[int(np.argmax(probs))])
rng = np.random.default_rng(0)
print("sampled:", rng.choice(words, p=probs))
k = 3
keep = np.argsort(-probs)[:k]
top_k = np.zeros_like(probs); top_k[keep] = probs[keep]; top_k /= top_k.sum()
print("top-k = 3:", [(words[i], round(float(top_k[i]), 3)) for i in keep])
order = np.argsort(-probs)
cumulative = np.cumsum(probs[order])
nucleus = order[: int(np.searchsorted(cumulative, 0.9)) + 1]
top_p = np.zeros_like(probs); top_p[nucleus] = probs[nucleus]; top_p /= top_p.sum()
print("top-p = 0.9:", [(words[i], round(float(top_p[i]), 3)) for i in nucleus])
In the lesson you can change this and run it.
Greedy always says " on". The seeded draw picks one token by the probabilities; run it again with another seed and the pick can change. Top-k keeps three tokens and spreads the removed 5% over them. Top-p keeps the smallest set reaching 0.9: two tokens, because " on" alone is 0.813 and " on" plus " down" is 0.903.
In your own words
Why does only the last row go through the LM head when generating, but every row during training?
Write a sentence or two, then compare it with a good answer in the lesson.
In real models: logits and generate in practice
In Hugging Face the model returns logits for every position, and generate wraps this lesson's loop with the sampling settings you saw. The map shows the end of the pipeline.
Where this shows up in an engineer's week
On the job: most output bugs here are settings. A temperature left at 1 on an extraction job makes answers differ between runs. A temperature of 0 on a creative job makes every answer the same. Log the sampling settings with every response, and for models that fix them for you, record which ones the provider ignores.
Two more habits help. Check that a stop token is set, since a loop with no stop runs to the length limit. And read the top few chances, not only the chosen token: a flat spread is the model telling you it is unsure.
03A bigram modelThe simplest language model, built from counts.35 min
04Sampling strategiesTemperature, top-k and top-p: turning probabilities into text.35 min
05Beam search and constrained decodingWhy the likeliest word isn't the likeliest sentence, beam search with log probabilities, its taste for short and repetitive text, and forcing output into a grammar: what that guarantees and what it doesn't.40 min
14The GPT architectureA stack of decoder blocks that predicts the next token.40 min
15Build a tiny GPTWrite and train a small GPT from scratch.45 min
16From your GPT to GPT-2 in Hugging FaceMatch your GPT's tensors to GPT-2's names and shapes, count its 124,439,808 parameters, load renamed weights into your own class, pad a batch and read a config.40 min
17Training a GPT wellWarmup and cosine decay, gradient clipping, a validation loss you can trust, saving and loading, and two training mistakes read off the loss curve.35 min
27The block in today's open modelsOptionalRMSNorm, SwiGLU, rotary positions and grouped-query attention swapped into the block you built, with the parameter and cache counts before and after.35 min
28Long contextEfficient attention and million-token windows.35 min
29Mixture of expertsRoute each token to a few specialist sub-networks.35 min
30Capstone: diagnose your own GPTEntropy per position, a head's attention pattern and a head ablation on the tiny GPT, then the same three measurements on a GPT you train yourself.45 min
31Jacobians: derivatives with shapesOptionalThe derivative of a vector function is a matrix; the chain rule multiplies them, and backprop multiplies by them without building them.35 min
32Backprop through a linear layerOptionaldX = dY Wᵀ, dW = Xᵀ dY, db = column sums: derived by shape and by entry, checked on the tiny GPT's own projection.35 min
33The softmax Jacobian and p − yOptionalEvery output depends on every input: the full Jacobian, its cheap vector product, and why cross-entropy's gradient is p − y.35 min
34Backprop through attentionOptionalFive formulas from the output back to queries, keys and values; why masked scores get zero gradient; what FlashAttention recomputes.40 min
36Object detection: boxes, overlap and suppressionEvery object and where it is: boxes in two formats, intersection over union, why one object gets many boxes, non-maximum suppression and its threshold, and grading with average precision.45 min
37Segmentation: a label for every pixelMasks, and how to grade them with IoU and Dice; why a pixel's label needs context and too much context blurs; the shrink-then-enlarge network with skip connections; and turning a mask into countable objects.45 min
38Build a vision transformerPatch embeddings, a class token, position embeddings and an unmasked block, trained in the page; then two failures that explain it: no positions means no layout, and few images means losing to a CNN a fraction of its size.45 min
39Image-text models in depth: retrieval, negatives and limitsHow to test a shared image-text space with recall at k, what the rest of the batch teaches, what hard and false negatives do, and why a similarity score can't see word order, counts or "no".45 min
40Self-supervised learning: labels from the data itselfPretrain an encoder with no labels by matching two views of each image, test it with a linear probe, and see what the augmentations remove, what collapse looks like, and how filling in masked parts differs.45 min
41Diffusion 1: turning data into noiseThe easy half of a diffusion model: one step of noise, why the data is shrunk as noise is added, the jump to any step with alpha bar, how to read a noise schedule, and the training examples a denoiser learns from.40 min
42Diffusion 2: learning to denoise, and samplingTrain a network to predict noise, then run it a hundred times from pure noise and watch a ring assemble: the sampling loop, why fresh noise goes back in, fewer steps and where they break, and a check that samples are new.45 min
43Diffusion 3: images, labels and guidanceFrom points to small images, a label the denoiser can be asked with, and classifier-free guidance: why a model that is told what to draw still often doesn't, how the guidance scale fixes it, and what it breaks when turned up too far.45 min
44Diffusion 4: latent diffusion and the whole pipelineWhy image generators diffuse an autoencoder's codes and not pixels: a latent diffusion model in miniature, what the autoencoder must be trained to do, the three parts of a text-to-image system, and where the cost of an image goes.40 min
45Sound as numbers: waveforms and spectrogramsA recording as an array of samples, why the sampling rate must be twice the highest pitch, spectrograms and the window trade-off, and the log-mel features a speech model is given.40 min
46Recognition and synthesis as a pipelineA whole speech recognizer for a made-up language: features, a guess per frame, collapsing repeats and blanks, a dictionary to repair mistakes, word error rate, and synthesis as the same stages run backward.40 min
47The voice agent loop and its latency budgetOne turn of a voice agent, timed: why every stage must stream, how to decide that someone has finished speaking, what to do when they interrupt, and why the slow turns matter more than the average.45 min
Follow one prompt, "The cat sat", through a language model with every number shown: tokens to IDs, embeddings and positional waves, the attention matrix, the feed-forward layer, and the probability over the next word.
See the course →
Never trained an AI? Teach a small one to write, right in your browser. It takes about 20 seconds and needs no account. Teach it to write