Capstone: diagnose your own GPT
Entropy per position, a head's attention pattern and a head ablation on the tiny GPT, then the same three measurements on a GPT you train yourself.
What it covers9
Why it matters
You built a GPT in this phase and followed one prompt through a transformer number by number. This capstone turns that into a method: three measurements that say what a model has learned and where it is weak, each a few lines of code. The entropy of the next-token distribution at every position shows where the model is certain and where it guesses. A head's attention pattern shows which earlier tokens it reads. An ablation, zeroing one head and measuring the loss, shows what the head is worth.
On the tiny GPT the three agree: 4.68 bits of uncertainty after "Our" and 0.01 after "is made"; three heads that put 0.97 of their attention on "tent" when the material is due; one head whose removal doubles the loss. Then it is your turn: train the same model on your own text and write up what you find.
What you'll be able to do
- Measure where a GPT is sure and unsure with the entropy of its next-token distribution at every position
- Read a head's attention pattern on a prompt and say what it looks up
- Ablate one head at a time, measure what each costs the loss, and rank them
- Train the Phase 5 GPT on your own text and report three findings, each backed by one measurement
Show the answer
At the first it is nearly certain of the next token; at the second it is spread over dozens
Entropy measures the spread of the distribution: 0.01 bits is one token with nearly all the probability; 4.68 bits is the uncertainty of a choice among about 26 equally likely tokens (2^4.68).
The brief: three measurements and a report
The job. A trained model is a black box until you measure it. You will take the tiny GPT you met in this phase and run three measurements on it, then do the same on a GPT you train yourself, and write three findings, each with its evidence.
What you build. Entropy of the next-token distribution at every position, the attention of every head from one position, and a head-by-head ablation that ranks the heads by what zeroing them costs. Each is a few lines of PyTorch on the model you built.
What you ship. A short report: an entropy map with the positions that surprised you, an attention pattern named with care or marked as having none, and a ranking of the heads. A finding names the measurement, the numbers and the comparison. The tutor marks the report against the rubric.
Show the answer
3 bits: log₂ 8
Entropy of a uniform distribution over n outcomes is log₂ n: 3 bits for 8. The tiny GPT's 4.68 bits after 'Our' is the spread of about 26 equally likely tokens.
Write a sentence or two, then compare it with a good answer in the lesson.
Build it: measure the tiny GPT
Measurement 1: where the model is sure
A language model gives a probability to every token in its vocabulary at every position. The entropy of that distribution, from the information track, , measures how spread out it is: near 0 bits when one token has almost all the probability, bits if the tiny GPT were guessing uniformly over its vocabulary. Computed at every position of a sentence, it is a map of where the model is confident and where it is not.
The next cell runs it on a catalog sentence followed by a tagline. Watch the pattern: the first positions are uncertain (after "Our", any of thirty product names could follow), the formulaic middle is near zero ("is" is followed by "made" with probability 1.00), and the content slots are in between: the material after "from" has 1.12 bits of uncertainty, with "silnylon" at 0.53. Entropy separates what the model has learned as structure from what it still weighs as content.
Entropy at every position
The tiny GPT's next-token entropy in bits after each token of a catalog sentence and its tagline, with the top prediction and its probability.
import math import torch from tinygpt import load_pretrained model, tok = load_pretrained("tiny-gpt") text = "Our Kestrel tent is made from silnylon . Made for cold nights ." ids = torch.tensor([tok.encode(text)]) with torch.no_grad(): p = torch.softmax(model(ids)[0], dim=-1) # one distribution per position H = -(p * (p + 1e-9).log()).sum(-1) / math.log(2) # entropy in bits for i, word in enumerate(text.split()): top = int(p[i].argmax()) print(f"after {word!r:12s} entropy {H[i].item():.2f} bits " f"top {tok.vocab[top]!r} {p[i, top].item():.2f}") import matplotlib.pyplot as plt fig, ax = plt.subplots(figsize=(6.5, 3)) ax.bar(range(len(H)), H.tolist()) ax.set_xticks(range(len(H)), text.split(), rotation=60) ax.set_ylabel("entropy (bits)") ax.set_title("Next-token entropy after each word")
After "Our" the model has 4.68 bits of uncertainty, the equivalent of about 26 equally likely tokens, and its top guess, "Ember", has only 0.09: any product name could come next. After "Kestrel", 2.97 bits: it has seen "Kestrel" with several product types. After "tent", 0.64 bits and "is" at 0.85; after "is" and "made", 0.02 and 0.01 bits: the catalog's sentence frame is memorized. The material slot after "from" has 1.12 bits with "silnylon" at 0.53, a real choice among tent materials. The tagline's slot after "for" has 1.82 bits ("storms" 0.49), the one after "cold" only 0.52 ("mornings" 0.91). Structure near zero, content in between, openings high.
import math p = [0.25, 0.25, 0.25, 0.25] print(round(-sum(x * math.log2(x) for x in p), 2))
Show the answer
2.0
Four equally likely outcomes carry log₂ 4 = 2 bits. The tiny GPT's 4.68 bits after 'Our' is the spread of about 2^4.68 ≈ 26 such outcomes.
Measurement 2: what a head reads
Attention weights say where each position looks. For one head, the row for the last token is a distribution over the earlier tokens, and reading it tells you what that head fetches at the moment of prediction. The next cell prints that row for all eight heads of the tiny GPT, from "from" in "Our Kestrel tent is made from", the position where the material is about to be chosen.
The two layers behave differently. Layer 0's heads spread their attention, 0.11 to 0.34 per token, leaning toward "tent" and "made". Layer 1's heads 1, 2 and 3 put 0.96 to 0.97 of their attention on "tent", one token, and head 0 splits between "tent" (0.61) and "made" (0.23). The material depends on the product type, and these heads are fetching it: the model has learned where the answer is. The "How transformers think" lessons showed this for one head on one prompt; a diagnosis checks every head, and compares prompts, before naming what a head does.
Every head's attention from "from"
For both layers and all four heads, the attention row of the last token of "Our Kestrel tent is made from".
import math import torch from tinygpt import load_pretrained model, tok = load_pretrained("tiny-gpt") ids = torch.tensor([tok.encode("Our Kestrel tent is made from")]) with torch.no_grad(): h = model.tok(ids) + model.pos(torch.arange(ids.shape[1])) rows = [] for layer, blk in enumerate(model.blocks): a = blk.ln1(h); B, T, d = a.shape q, k, v = blk.qkv(a).split(d, dim=-1); dh = d // blk.h q, k = (t.view(B, T, blk.h, dh).transpose(1, 2) for t in (q, k)) s = (q @ k.transpose(-2, -1) / math.sqrt(dh)).masked_fill( ~torch.tril(torch.ones(T, T)).bool(), float("-inf")) A = torch.softmax(s, dim=-1)[0] # heads x T x T for head in range(blk.h): print(f"layer {layer} head {head}:", [ f"{tok.vocab[int(t)]} {A[head, -1, j].item():.2f}" for j, t in enumerate(ids[0])]) h = blk(h) rows.append(A[:, -1, :]) import matplotlib.pyplot as plt heat = torch.cat(rows).tolist() # 8 heads x T fig, ax = plt.subplots(figsize=(6, 3.4)) im = ax.imshow(heat, cmap="Blues", vmin=0, vmax=1) ax.set_xticks(range(len(ids[0])), [tok.vocab[int(t)] for t in ids[0]]) ax.set_yticks(range(8), [f"L{i // 4} H{i % 4}" for i in range(8)]) ax.set_title("Attention from the last token, every head") fig.colorbar(im, ax=ax)
Layer 0 is diffuse: head 1 gives "tent" 0.34 and "made" 0.29, head 3 "tent" 0.34 and "Kestrel" 0.22, nothing dominant. Layer 1 is sharp: heads 1, 2 and 3 put 0.97, 0.97 and 0.96 on "tent", and head 0 puts 0.61 on "tent" and 0.23 on "made". By the second layer the model knows that the material it is about to name depends on the product type and reads that token almost exclusively. A diagnosis would now test the claim on other prompts: do these heads attend to "mug" in a mug sentence, and to "pants" in a pants sentence? If so, "fetch the product type" is a fair name for them.
Show the answer
They fetch the product type the material depends on; test it on other types before naming them
One prompt supports a hypothesis; checking it on mugs and pants is what turns it into a finding. Redundant heads are common in trained models.
The tokens a head reads most
Write top_attended(weights, tokens, k=2): weights is a list of attention weights for one position (one per earlier token) and tokens the matching list of token strings. Return the k tokens with the largest weights as a list of (token, weight) pairs, largest first.
def top_attended(weights, tokens, k=2): return [] weights = [0.00, 0.00, 0.97, 0.00, 0.02, 0.01] tokens = ["Our", "Kestrel", "tent", "is", "made", "from"] print(top_attended(weights, tokens))
Measurement 3: what a head is worth
Attention says where a head reads; it does not say whether the model needs it. An ablation does: zero the head's output (its value vectors, so it contributes nothing to the residual stream) and measure the loss on a set of sentences. The rise in loss is what the rest of the network could not do without it.
The next cell does this for all eight heads over 40 catalog sentences. The baseline loss is 0.458 nats per token. Zeroing layer 1 head 0 raises it to 0.959, more than double; heads 1, 2 and 3 of layer 1 cost 0.150 to 0.225; no layer 0 head costs more than 0.037. The two measurements tell one story and correct each other: layer 1 does the work and layer 0 prepares it, and the head that looked least focused in the attention printout, head 0 of layer 1, is the one the model can least afford to lose. Attention weights are not importance.
A caveat belongs in every write-up. An ablation measures what the network cannot compensate for, with every other head still in place; three redundant heads can each look cheap because the other two cover for them. Removing pairs, or all of a layer's heads, is the next experiment when the single-head costs look small.
Zero one head at a time
Mean loss per token over 40 catalog sentences with every head in place, then with each head's values zeroed in turn.
import math import requests import torch from tinygpt import load_pretrained model, tok = load_pretrained("tiny-gpt") corpus = requests.get("https://api.gradient.test/v1/corpus").json()["text"] sentences = [s.strip() for s in corpus.split("\n") if s.strip()][:40] def mean_loss(zero=None): # zero = (layer, head) to silence, or None total, count = 0.0, 0 for s in sentences: ids = torch.tensor([tok.encode(s)]) x, y = ids[:, :-1], ids[0, 1:] with torch.no_grad(): h = model.tok(x) + model.pos(torch.arange(x.shape[1])) for layer, blk in enumerate(model.blocks): a = blk.ln1(h); B, T, d = a.shape q, k, v = blk.qkv(a).split(d, dim=-1); dh = d // blk.h if zero is not None and zero[0] == layer: # the head contributes nothing v = v.clone(); v[:, :, zero[1] * dh:(zero[1] + 1) * dh] = 0.0 q, k, v = (t.view(B, T, blk.h, dh).transpose(1, 2) for t in (q, k, v)) sc = (q @ k.transpose(-2, -1) / math.sqrt(dh)).masked_fill( ~torch.tril(torch.ones(T, T)).bool(), float("-inf")) h = h + blk.proj( (torch.softmax(sc, dim=-1) @ v).transpose(1, 2).reshape(B, T, d)) h = h + blk.mlp(blk.ln2(h)) nll = -torch.log_softmax(model.head(model.ln_f(h)), dim=-1)[0, torch.arange(x.shape[1]), y] total += nll.sum().item(); count += nll.numel() return total / count costs = {} base = mean_loss() print(f"baseline loss {base:.3f} over {len(sentences)} sentences") for layer in range(2): for head in range(4): loss = mean_loss((layer, head)) print(f"layer {layer} head {head} zeroed: loss {loss:.3f} (+{loss - base:.3f})") costs[f"L{layer} H{head}"] = loss - base import matplotlib.pyplot as plt order = sorted(costs, key=costs.get, reverse=True) fig, ax = plt.subplots(figsize=(6.5, 3)) ax.bar(order, [costs[n] for n in order]) ax.set_ylabel("loss added when zeroed") ax.set_title("What each head is worth, largest first")
Baseline 0.458 nats per token. Layer 0's heads cost between 0.013 and 0.037 each: the network barely notices. Layer 1's heads cost 0.150, 0.207 and 0.225 for heads 1 to 3, and 0.501 for head 0, which takes the loss to 0.959. The head whose attention looked least decisive carries the most: its 0.61 on "tent" and 0.23 on "made" may be combining two pieces of information the other heads do not. That is the kind of finding worth a paragraph in a report, with this table as its evidence and a pair ablation as the next step.
base = 0.458 losses = {"L1H0": 0.959, "L1H1": 0.608, "L0H2": 0.495} print(max(losses, key=lambda name: losses[name] - base))
Show the answer
L1H0
max with a key picks the name whose cost (loss minus baseline) is largest: L1H0 at +0.501.
Show the answer
With the other heads in place the network compensates for it; it may still matter with others
Single-head ablation measures what the rest cannot cover. Redundant heads look cheap one at a time; remove them in pairs or as a layer to find out.
Rank heads by their cost
Write rank_heads(losses, base): losses maps a head's name to the loss with that head zeroed, base is the loss with every head in place. Return a list of (name, cost) pairs, cost = loss − base rounded to 3 decimals, largest cost first.
def rank_heads(losses, base): return [] losses = {"layer 1 head 0": 0.959, "layer 1 head 1": 0.608, "layer 1 head 2": 0.665, "layer 1 head 3": 0.683, "layer 0 head 2": 0.495} for name, cost in rank_heads(losses, 0.458): print(name, cost)
What a sharp head looks like (illustrative scores shaped like layer 1 head 1): from the position where the material is chosen, almost all of the attention goes to the product type.
Write a sentence or two, then compare it with a good answer in the lesson.
Ship it: diagnose a GPT you trained
The brief below is the capstone. Its starter is the training loop from the build-a-GPT lesson with the three measurements as functions. By default it trains on the practice catalog; replace the text with a few hundred lines of your own (notes, invented product copy, a synthetic log, anything you may use freely), train it, and then investigate the model you made.
A good report has three findings, each one a claim with its measurement attached: an entropy map that shows where your model is certain and where it guesses, with the two or three positions that surprised you; an attention pattern for one head on at least two prompts, with a careful name for what it fetches or an honest "no consistent pattern"; and an ablation table that ranks the heads, with a sentence on what the top head's removal breaks. End with one sentence on what you would change, more data, more steps, a bigger model, and the measurement that would show whether it helped. The tutor reviews it against the rubric; signed out, the rubric is a checklist to tick yourself.
Diagnose your own GPT
A small team ships a tiny language model trained on their own product copy, and the lead wants more than a loss number before anyone trusts it: where is it confident, what has it actually learned to look up, and which parts of it carry the weight? The starter trains the Phase 5 GPT (2 layers, 4 heads, 32-wide) on a text of your choosing; by default it uses the practice catalog, which you should replace with a few hundred lines of your own.
What makes this hard is discipline, not code: each claim in your report must point at a number from one of the three measurements, a pattern seen once must be checked on a second prompt before it gets a name, and an ablation's cost must be read with the caveat that the other heads may compensate. Invented or freely usable text only; nothing personal.
It must
- Train the model on your own text (at least 300 lines) and report the training and validation loss against the untrained baseline ln(vocabulary size)
- An entropy map for at least two sentences from your text, with the highest- and lowest-entropy positions named and explained
- The attention pattern of one head from the last token of at least two different prompts, and a careful statement of what it fetches, or that it has no consistent pattern
- An ablation table over all eight heads with the baseline, sorted by cost, and a sentence on what the costliest head's removal breaks
- A closing paragraph: three findings, each with its evidence, and one change you would try with the measurement that would judge it
Go further
- Ablate pairs of heads and compare the pair's cost with the sum of the two single costs
- Plot entropy against position across many sentences to find systematic sure and unsure slots
- Retrain with a different seed and check which findings survive
import math import re import requests import torch import torch.nn as nn # 1. Your text. Replace this with a few hundred lines of your own # (invented or freely usable). text = requests.get("https://api.gradient.test/v1/corpus").json()["text"] tokens = re.findall(r"[A-Za-z]+|[0-9]+|[^\sA-Za-z0-9]|\n", text) vocab = sorted(set(tokens)) stoi = {t: i for i, t in enumerate(vocab)} data = torch.tensor([stoi[t] for t in tokens]) split = int(0.9 * len(data)) train_data, val_data = data[:split], data[split:] print(f"{len(tokens):,} tokens, vocabulary of {len(vocab)}, " f"untrained loss about {math.log(len(vocab)):.2f}") def get_batch(d, block=24, batch=16): starts = torch.randint(0, len(d) - block - 1, (batch,)).tolist() x = torch.stack([d[i:i + block] for i in starts]) y = torch.stack([d[i + 1:i + block + 1] for i in starts]) return x, y # 2. The Phase 5 model. class Block(nn.Module): def __init__(self, d, h): super().__init__() self.h = h self.ln1, self.ln2 = nn.LayerNorm(d), nn.LayerNorm(d) self.qkv, self.proj = nn.Linear(d, 3 * d), nn.Linear(d, d) self.mlp = nn.Sequential(nn.Linear(d, 4 * d), nn.GELU(), nn.Linear(4 * d, d)) def attention(self, x, zero=None): B, T, d = x.shape q, k, v = self.qkv(self.ln1(x)).split(d, dim=-1) dh = d // self.h if zero is not None: v = v.clone(); v[:, :, zero * dh:(zero + 1) * dh] = 0.0 q, k, v = (t.view(B, T, self.h, dh).transpose(1, 2) for t in (q, k, v)) s = (q @ k.transpose(-2, -1) / math.sqrt(dh)).masked_fill( ~torch.tril(torch.ones(T, T)).bool(), float("-inf")) A = torch.softmax(s, dim=-1) return self.proj((A @ v).transpose(1, 2).reshape(B, T, d)), A[0] def forward(self, x, zero=None): out, _ = self.attention(x, zero) x = x + out return x + self.mlp(self.ln2(x)) class GPT(nn.Module): def __init__(self, vocab, context, d, n_layers, n_heads): super().__init__() self.context = context self.tok, self.pos = nn.Embedding(vocab, d), nn.Embedding(context, d) self.blocks = nn.ModuleList([Block(d, n_heads) for _ in range(n_layers)]) self.ln_f, self.head = nn.LayerNorm(d), nn.Linear(d, vocab) def forward(self, idx, zero=None): x = self.tok(idx) + self.pos(torch.arange(idx.shape[1])) for layer, blk in enumerate(self.blocks): x = blk(x, zero[1] if zero is not None and zero[0] == layer else None) return self.head(self.ln_f(x)) torch.manual_seed(0) model = GPT(len(vocab), context=24, d=32, n_layers=2, n_heads=4) opt = torch.optim.AdamW(model.parameters(), lr=3e-3, weight_decay=0.01) # 3. Train. for step in range(201): x, y = get_batch(train_data) loss = nn.functional.cross_entropy(model(x).view(-1, len(vocab)), y.view(-1)) opt.zero_grad(); loss.backward(); opt.step() if step % 50 == 0: with torch.no_grad(): xv, yv = get_batch(val_data, batch=64) val = nn.functional.cross_entropy(model(xv).view(-1, len(vocab)), yv.view(-1)).item() print(f"step {step:>3} training loss {loss.item():.3f} " f"validation loss {val:.3f}") # 4. The three measurements. def encode(s): ids = [stoi[t] for t in re.findall(r"[A-Za-z]+|[0-9]+|[^\sA-Za-z0-9]|\n", s) if t in stoi] # the model reads at most 24 tokens, its context return torch.tensor([ids[:model.context]]) def entropy_map(sentence): ids = encode(sentence) with torch.no_grad(): p = torch.softmax(model(ids)[0], dim=-1) H = -(p * (p + 1e-9).log()).sum(-1) / math.log(2) return [(vocab[int(t)], round(H[i].item(), 2), vocab[int(p[i].argmax())]) for i, t in enumerate(ids[0])] def attention_from_last(sentence, layer, head): ids = encode(sentence) with torch.no_grad(): x = model.tok(ids) + model.pos(torch.arange(ids.shape[1])) for l, blk in enumerate(model.blocks): out, A = blk.attention(x) if l == layer: return [(vocab[int(t)], round(A[head, -1, j].item(), 2)) for j, t in enumerate(ids[0])] x = x + out x = x + blk.mlp(blk.ln2(x)) def ablation_table(sentences): def mean_loss(zero=None): total, count = 0.0, 0 for s in sentences: ids = encode(s) if ids.shape[1] < 3: continue with torch.no_grad(): logp = torch.log_softmax(model(ids[:, :-1], zero), dim=-1) nll = -logp[0, torch.arange(ids.shape[1] - 1), ids[0, 1:]] total += nll.sum().item(); count += nll.numel() return total / count base = mean_loss() rows = [(f"layer {l} head {h}", round(mean_loss((l, h)) - base, 3)) for l in range(2) for h in range(4)] return base, sorted(rows, key=lambda r: r[1], reverse=True) sentences = [s.strip() for s in text.split("\n") if s.strip()][:20] print(entropy_map(sentences[0])) print(attention_from_last(sentences[0], 1, 0)) base, table = ablation_table(sentences) print(f"baseline {base:.3f}", table[:3]) # your code: run the three measurements on your own text, check each attention # claim on a second prompt, and write your three findings as comments or prints below.
Show the answer
"Zeroing layer 1 head 0 raises the mean loss over 40 sentences from 0.458 to 0.959; no layer 0 head adds more than 0.037"
A finding names the measurement, the numbers and the comparison. The others are impressions, an unquantified loss claim, and a one-prompt hypothesis.
Where this shows up in an engineer's week
On the job: these three measurements are the first things to run on a model that behaves oddly. Entropy tells you where it is guessing, attention tells you what it is reading, and an ablation tells you which part to blame. Keep the baseline loss next to every ablation, and measure on held-out text, not the text the model trained on.
Write a sentence or two, then compare it with a good answer in the lesson.
You can now
- Map a model's certainty with the entropy of its next-token distribution at every position, and separate structure from content.
- Read every head's attention from a position and test a hypothesis about what a head fetches on a second prompt.
- Ablate heads one at a time, rank them by cost, and read the ranking with the compensation caveat.
- Train the Phase 5 GPT on your own text and write up three findings with their evidence.
This closes Phase 5's main path. The optional "Math under the GPT" lessons, from Jacobians: derivatives with shapes on, derive its gradients by hand. Phase 6 takes the same model into post-training, adaptation and inference at scale.
Practice what you just read
- 3 code examples you can edit and run
- a practice exercise that checks your code
- a 5-question quiz
- 9 review cards for spaced repetition
Try a quiz question
The tiny GPT's next-token entropy is 0.01 bits after "is made" and 1.12 bits after "from". The best description is:
Before this lesson
Where this lesson leads.
575 lessons in thirteen phases. 176 are free without an account, and a free account opens the rest. See every lesson.
Phase 5 · about 36 hours of study
Transformers
Attention, GPT from scratch and pretraining
47 lessons 1 free
- Language modeling FreePredict the next token, and score it with cross-entropy.30 min
- Tokenization and BPEHow tokenizers are built from data.40 min
- A bigram modelThe simplest language model, built from counts.35 min
- Sampling strategiesTemperature, top-k and top-p: turning probabilities into text.35 min
- Beam search and constrained decodingWhy the likeliest word isn't the likeliest sentence, beam search with log probabilities, its taste for short and repetitive text, and forcing output into a grammar: what that guarantees and what it doesn't.40 min
- Neural language modelsWhy counting fails, and what embeddings fix.35 min
- Attention: the intuitionEach token decides which other tokens matter to it.30 min
- Queries, keys and valuesThe three projections behind attention.35 min
- Scaled dot-product attentionThe attention formula, line by line.35 min
- Causal maskingStop tokens from seeing the future.25 min
- Multi-head attentionSeveral attention patterns in parallel.40 min
- Positional encodingsTell the model where each token sits.30 min
- The transformer blockAttention, MLP, residuals and norms together.40 min
- The GPT architectureA stack of decoder blocks that predicts the next token.40 min
- Build a tiny GPTWrite and train a small GPT from scratch.45 min
- From your GPT to GPT-2 in Hugging FaceMatch your GPT's tensors to GPT-2's names and shapes, count its 124,439,808 parameters, load renamed weights into your own class, pad a batch and read a config.40 min
- Training a GPT wellWarmup and cosine decay, gradient clipping, a validation loss you can trust, saving and loading, and two training mistakes read off the loss curve.35 min
- Encoders and embedding modelsBERT-style models and where they fit.40 min
- Tokenization: text to integer IDsFollow "The cat sat" into the three integers a model receives.30 min
- Embeddings and positional wavesFrom three IDs to the 3 × 8 matrix that enters the first block.35 min
- Inside the attention matrixEvery entry of the 3 × 3 attention matrix, computed and steered.40 min
- The feed-forward layerExpand to four times d_model and back: shapes, parameters and what the layer stores.35 min
- The LM head and samplingFrom the last row to a probability for every token, and the next word.35 min
- Pretraining at scaleData, compute and what models learn from the internet.40 min
- Scaling lawsHow loss falls predictably with size, data and compute.40 min
- Inference and the KV cacheWhy generation is fast after the first token.40 min
- The block in today's open models OptionalRMSNorm, SwiGLU, rotary positions and grouped-query attention swapped into the block you built, with the parameter and cache counts before and after.35 min
- Long contextEfficient attention and million-token windows.35 min
- Mixture of expertsRoute each token to a few specialist sub-networks.35 min
- Capstone: diagnose your own GPTEntropy per position, a head's attention pattern and a head ablation on the tiny GPT, then the same three measurements on a GPT you train yourself.45 min
- Jacobians: derivatives with shapes OptionalThe derivative of a vector function is a matrix; the chain rule multiplies them, and backprop multiplies by them without building them.35 min
- Backprop through a linear layer OptionaldX = dY Wᵀ, dW = Xᵀ dY, db = column sums: derived by shape and by entry, checked on the tiny GPT's own projection.35 min
- The softmax Jacobian and p − y OptionalEvery output depends on every input: the full Jacobian, its cheap vector product, and why cross-entropy's gradient is p − y.35 min
- Backprop through attention OptionalFive formulas from the output back to queries, keys and values; why masked scores get zero gradient; what FlashAttention recomputes.40 min
- Multimodal transformersHow models see images and hear audio.35 min
- Object detection: boxes, overlap and suppressionEvery object and where it is: boxes in two formats, intersection over union, why one object gets many boxes, non-maximum suppression and its threshold, and grading with average precision.45 min
- Segmentation: a label for every pixelMasks, and how to grade them with IoU and Dice; why a pixel's label needs context and too much context blurs; the shrink-then-enlarge network with skip connections; and turning a mask into countable objects.45 min
- Build a vision transformerPatch embeddings, a class token, position embeddings and an unmasked block, trained in the page; then two failures that explain it: no positions means no layout, and few images means losing to a CNN a fraction of its size.45 min
- Image-text models in depth: retrieval, negatives and limitsHow to test a shared image-text space with recall at k, what the rest of the batch teaches, what hard and false negatives do, and why a similarity score can't see word order, counts or "no".45 min
- Self-supervised learning: labels from the data itselfPretrain an encoder with no labels by matching two views of each image, test it with a linear probe, and see what the augmentations remove, what collapse looks like, and how filling in masked parts differs.45 min
- Diffusion 1: turning data into noiseThe easy half of a diffusion model: one step of noise, why the data is shrunk as noise is added, the jump to any step with alpha bar, how to read a noise schedule, and the training examples a denoiser learns from.40 min
- Diffusion 2: learning to denoise, and samplingTrain a network to predict noise, then run it a hundred times from pure noise and watch a ring assemble: the sampling loop, why fresh noise goes back in, fewer steps and where they break, and a check that samples are new.45 min
- Diffusion 3: images, labels and guidanceFrom points to small images, a label the denoiser can be asked with, and classifier-free guidance: why a model that is told what to draw still often doesn't, how the guidance scale fixes it, and what it breaks when turned up too far.45 min
- Diffusion 4: latent diffusion and the whole pipelineWhy image generators diffuse an autoencoder's codes and not pixels: a latent diffusion model in miniature, what the autoencoder must be trained to do, the three parts of a text-to-image system, and where the cost of an image goes.40 min
- Sound as numbers: waveforms and spectrogramsA recording as an array of samples, why the sampling rate must be twice the highest pitch, spectrograms and the window trade-off, and the log-mel features a speech model is given.40 min
- Recognition and synthesis as a pipelineA whole speech recognizer for a made-up language: features, a guess per frame, collapsing repeats and blanks, a dictionary to repair mistakes, word error rate, and synthesis as the same stages run backward.40 min
- The voice agent loop and its latency budgetOne turn of a voice agent, timed: why every stage must stream, how to decide that someone has finished speaking, what to do when they interrupt, and why the slow turns matter more than the average.45 min
Never trained an AI? Teach a small one to write, right in your browser. It takes about 20 seconds and needs no account. Teach it to write