Index Ai Academy
Phase 5 · Scale & inference

Capstone: diagnose your own GPT

Entropy per position, a head's attention pattern and a head ablation on the tiny GPT, then the same three measurements on a GPT you train yourself.

About 45 minutes · A free account opens the whole lesson.
Already have an account? Sign in to open this lesson

What it covers9

Why it matters

You built a GPT in this phase and followed one prompt through a transformer number by number. This capstone turns that into a method: three measurements that say what a model has learned and where it is weak, each a few lines of code. The entropy of the next-token distribution at every position shows where the model is certain and where it guesses. A head's attention pattern shows which earlier tokens it reads. An ablation, zeroing one head and measuring the loss, shows what the head is worth.

On the tiny GPT the three agree: 4.68 bits of uncertainty after "Our" and 0.01 after "is made"; three heads that put 0.97 of their attention on "tent" when the material is due; one head whose removal doubles the loss. Then it is your turn: train the same model on your own text and write up what you find.

What you'll be able to do

  • Measure where a GPT is sure and unsure with the entropy of its next-token distribution at every position
  • Read a head's attention pattern on a prompt and say what it looks up
  • Ablate one head at a time, measure what each costs the loss, and rank them
  • Train the Phase 5 GPT on your own text and report three findings, each backed by one measurement
Quick check
A model's next-token distribution has entropy 0.01 bits at one position and 4.68 bits at another. What does that tell you?
  1. The second position has a larger vocabulary to choose the next token from
  2. At the first it is nearly certain of the next token; at the second it is spread over dozens
  3. At the first position it will be right and at the second it will be wrong
  4. The first position is early in the text and the second one is late in it
Show the answer

At the first it is nearly certain of the next token; at the second it is spread over dozens

Entropy measures the spread of the distribution: 0.01 bits is one token with nearly all the probability; 4.68 bits is the uncertainty of a choice among about 26 equally likely tokens (2^4.68).

The brief: three measurements and a report

The job. A trained model is a black box until you measure it. You will take the tiny GPT you met in this phase and run three measurements on it, then do the same on a GPT you train yourself, and write three findings, each with its evidence.

What you build. Entropy of the next-token distribution at every position, the attention of every head from one position, and a head-by-head ablation that ranks the heads by what zeroing them costs. Each is a few lines of PyTorch on the model you built.

What you ship. A short report: an entropy map with the positions that surprised you, an attention pattern named with care or marked as having none, and a ranking of the heads. A finding names the measurement, the numbers and the comparison. The tutor marks the report against the rubric.

Quick check
From the entropy lesson: a fair choice among 8 equally likely outcomes carries how many bits?
  1. 0.125 bits: the probability of each
  2. 1 bit: it is one choice
  3. 8 bits: one per outcome
  4. 3 bits: log₂ 8
Show the answer

3 bits: log₂ 8

Entropy of a uniform distribution over n outcomes is log₂ n: 3 bits for 8. The tiny GPT's 4.68 bits after 'Our' is the spread of about 26 equally likely tokens.

In your own words
Before you start: what would make a claim about a head a finding rather than an impression?

Write a sentence or two, then compare it with a good answer in the lesson.

Build it: measure the tiny GPT

Measurement 1: where the model is sure

A language model gives a probability to every token in its vocabulary at every position. The entropy of that distribution, from the information track, H=−∑kpklog⁡2pkH = -\sum_k p_k \log_2 p_k, measures how spread out it is: near 0 bits when one token has almost all the probability, log⁡2195=7.6\log_2 195 = 7.6 bits if the tiny GPT were guessing uniformly over its vocabulary. Computed at every position of a sentence, it is a map of where the model is confident and where it is not.

The next cell runs it on a catalog sentence followed by a tagline. Watch the pattern: the first positions are uncertain (after "Our", any of thirty product names could follow), the formulaic middle is near zero ("is" is followed by "made" with probability 1.00), and the content slots are in between: the material after "from" has 1.12 bits of uncertainty, with "silnylon" at 0.53. Entropy separates what the model has learned as structure from what it still weighs as content.

Worked example

Entropy at every position

The tiny GPT's next-token entropy in bits after each token of a catalog sentence and its tagline, with the top prediction and its probability.

import math
import torch
from tinygpt import load_pretrained

model, tok = load_pretrained("tiny-gpt")
text = "Our Kestrel tent is made from silnylon . Made for cold nights ."
ids = torch.tensor([tok.encode(text)])
with torch.no_grad():
    p = torch.softmax(model(ids)[0], dim=-1)             # one distribution per position
H = -(p * (p + 1e-9).log()).sum(-1) / math.log(2)           # entropy in bits
for i, word in enumerate(text.split()):
    top = int(p[i].argmax())
    print(f"after {word!r:12s} entropy {H[i].item():.2f} bits  "
          f"top {tok.vocab[top]!r} {p[i, top].item():.2f}")

import matplotlib.pyplot as plt
fig, ax = plt.subplots(figsize=(6.5, 3))
ax.bar(range(len(H)), H.tolist())
ax.set_xticks(range(len(H)), text.split(), rotation=60)
ax.set_ylabel("entropy (bits)")
ax.set_title("Next-token entropy after each word")
It runs right here in your browser, with nothing to install.

After "Our" the model has 4.68 bits of uncertainty, the equivalent of about 26 equally likely tokens, and its top guess, "Ember", has only 0.09: any product name could come next. After "Kestrel", 2.97 bits: it has seen "Kestrel" with several product types. After "tent", 0.64 bits and "is" at 0.85; after "is" and "made", 0.02 and 0.01 bits: the catalog's sentence frame is memorized. The material slot after "from" has 1.12 bits with "silnylon" at 0.53, a real choice among tent materials. The tagline's slot after "for" has 1.82 bits ("storms" 0.49), the one after "cold" only 0.52 ("mornings" 0.91). Structure near zero, content in between, openings high.

Quick check
Entropy of a uniform choice among four tokens, in bits. What does this print?
import math
p = [0.25, 0.25, 0.25, 0.25]
print(round(-sum(x * math.log2(x) for x in p), 2))
  1. 1.0
  2. 2.0
  3. 0.25
  4. 4.0
Show the answer

2.0

Four equally likely outcomes carry log₂ 4 = 2 bits. The tiny GPT's 4.68 bits after 'Our' is the spread of about 2^4.68 ≈ 26 such outcomes.

Measurement 2: what a head reads

Attention weights say where each position looks. For one head, the row for the last token is a distribution over the earlier tokens, and reading it tells you what that head fetches at the moment of prediction. The next cell prints that row for all eight heads of the tiny GPT, from "from" in "Our Kestrel tent is made from", the position where the material is about to be chosen.

The two layers behave differently. Layer 0's heads spread their attention, 0.11 to 0.34 per token, leaning toward "tent" and "made". Layer 1's heads 1, 2 and 3 put 0.96 to 0.97 of their attention on "tent", one token, and head 0 splits between "tent" (0.61) and "made" (0.23). The material depends on the product type, and these heads are fetching it: the model has learned where the answer is. The "How transformers think" lessons showed this for one head on one prompt; a diagnosis checks every head, and compares prompts, before naming what a head does.

Worked example

Every head's attention from "from"

For both layers and all four heads, the attention row of the last token of "Our Kestrel tent is made from".

import math
import torch
from tinygpt import load_pretrained

model, tok = load_pretrained("tiny-gpt")
ids = torch.tensor([tok.encode("Our Kestrel tent is made from")])
with torch.no_grad():
    h = model.tok(ids) + model.pos(torch.arange(ids.shape[1]))
    rows = []
    for layer, blk in enumerate(model.blocks):
        a = blk.ln1(h); B, T, d = a.shape
        q, k, v = blk.qkv(a).split(d, dim=-1); dh = d // blk.h
        q, k = (t.view(B, T, blk.h, dh).transpose(1, 2) for t in (q, k))
        s = (q @ k.transpose(-2, -1) / math.sqrt(dh)).masked_fill(
            ~torch.tril(torch.ones(T, T)).bool(), float("-inf"))
        A = torch.softmax(s, dim=-1)[0]                     # heads x T x T
        for head in range(blk.h):
            print(f"layer {layer} head {head}:", [
                f"{tok.vocab[int(t)]} {A[head, -1, j].item():.2f}"
                for j, t in enumerate(ids[0])])
        h = blk(h)
        rows.append(A[:, -1, :])

import matplotlib.pyplot as plt
heat = torch.cat(rows).tolist()                          # 8 heads x T
fig, ax = plt.subplots(figsize=(6, 3.4))
im = ax.imshow(heat, cmap="Blues", vmin=0, vmax=1)
ax.set_xticks(range(len(ids[0])), [tok.vocab[int(t)] for t in ids[0]])
ax.set_yticks(range(8), [f"L{i // 4} H{i % 4}" for i in range(8)])
ax.set_title("Attention from the last token, every head")
fig.colorbar(im, ax=ax)
In the lesson you can change this and run it.

Layer 0 is diffuse: head 1 gives "tent" 0.34 and "made" 0.29, head 3 "tent" 0.34 and "Kestrel" 0.22, nothing dominant. Layer 1 is sharp: heads 1, 2 and 3 put 0.97, 0.97 and 0.96 on "tent", and head 0 puts 0.61 on "tent" and 0.23 on "made". By the second layer the model knows that the material it is about to name depends on the product type and reads that token almost exclusively. A diagnosis would now test the claim on other prompts: do these heads attend to "mug" in a mug sentence, and to "pants" in a pants sentence? If so, "fetch the product type" is a fair name for them.

Quick check
Layer 1 heads 1, 2 and 3 each put about 0.97 of their attention on "tent" from the position where the material is predicted. The most careful reading is:
  1. They are broken, since three heads doing the same thing is a waste of the model's capacity
  2. They have memorized the word 'tent' and attend to it wherever it appears in any sentence
  3. They predict that the word 'tent' will come next, since that is where they look
  4. They fetch the product type the material depends on; test it on other types before naming them
Show the answer

They fetch the product type the material depends on; test it on other types before naming them

One prompt supports a hypothesis; checking it on mugs and pants is what turns it into a finding. Redundant heads are common in trained models.

Drill

The tokens a head reads most

Write top_attended(weights, tokens, k=2): weights is a list of attention weights for one position (one per earlier token) and tokens the matching list of token strings. Return the k tokens with the largest weights as a list of (token, weight) pairs, largest first.

def top_attended(weights, tokens, k=2):
    return []

weights = [0.00, 0.00, 0.97, 0.00, 0.02, 0.01]
tokens = ["Our", "Kestrel", "tent", "is", "made", "from"]
print(top_attended(weights, tokens))
In the lesson, your code is checked as you write it.

Measurement 3: what a head is worth

Attention says where a head reads; it does not say whether the model needs it. An ablation does: zero the head's output (its value vectors, so it contributes nothing to the residual stream) and measure the loss on a set of sentences. The rise in loss is what the rest of the network could not do without it.

The next cell does this for all eight heads over 40 catalog sentences. The baseline loss is 0.458 nats per token. Zeroing layer 1 head 0 raises it to 0.959, more than double; heads 1, 2 and 3 of layer 1 cost 0.150 to 0.225; no layer 0 head costs more than 0.037. The two measurements tell one story and correct each other: layer 1 does the work and layer 0 prepares it, and the head that looked least focused in the attention printout, head 0 of layer 1, is the one the model can least afford to lose. Attention weights are not importance.

A caveat belongs in every write-up. An ablation measures what the network cannot compensate for, with every other head still in place; three redundant heads can each look cheap because the other two cover for them. Removing pairs, or all of a layer's heads, is the next experiment when the single-head costs look small.

Worked example

Zero one head at a time

Mean loss per token over 40 catalog sentences with every head in place, then with each head's values zeroed in turn.

import math
import requests
import torch
from tinygpt import load_pretrained

model, tok = load_pretrained("tiny-gpt")
corpus = requests.get("https://api.gradient.test/v1/corpus").json()["text"]
sentences = [s.strip() for s in corpus.split("\n") if s.strip()][:40]

def mean_loss(zero=None):                     # zero = (layer, head) to silence, or None
    total, count = 0.0, 0
    for s in sentences:
        ids = torch.tensor([tok.encode(s)])
        x, y = ids[:, :-1], ids[0, 1:]
        with torch.no_grad():
            h = model.tok(x) + model.pos(torch.arange(x.shape[1]))
            for layer, blk in enumerate(model.blocks):
                a = blk.ln1(h); B, T, d = a.shape
                q, k, v = blk.qkv(a).split(d, dim=-1); dh = d // blk.h
                if zero is not None and zero[0] == layer:
                    # the head contributes nothing
                    v = v.clone(); v[:, :, zero[1] * dh:(zero[1] + 1) * dh] = 0.0
                q, k, v = (t.view(B, T, blk.h, dh).transpose(1, 2) for t in (q, k, v))
                sc = (q @ k.transpose(-2, -1) / math.sqrt(dh)).masked_fill(
                    ~torch.tril(torch.ones(T, T)).bool(), float("-inf"))
                h = h + blk.proj(
                    (torch.softmax(sc, dim=-1) @ v).transpose(1, 2).reshape(B, T, d))
                h = h + blk.mlp(blk.ln2(h))
            nll = -torch.log_softmax(model.head(model.ln_f(h)),
                                     dim=-1)[0, torch.arange(x.shape[1]), y]
        total += nll.sum().item(); count += nll.numel()
    return total / count

costs = {}
base = mean_loss()
print(f"baseline loss {base:.3f} over {len(sentences)} sentences")
for layer in range(2):
    for head in range(4):
        loss = mean_loss((layer, head))
        print(f"layer {layer} head {head} zeroed: loss {loss:.3f} (+{loss - base:.3f})")
        costs[f"L{layer} H{head}"] = loss - base

import matplotlib.pyplot as plt
order = sorted(costs, key=costs.get, reverse=True)
fig, ax = plt.subplots(figsize=(6.5, 3))
ax.bar(order, [costs[n] for n in order])
ax.set_ylabel("loss added when zeroed")
ax.set_title("What each head is worth, largest first")
In the lesson you can change this and run it.

Baseline 0.458 nats per token. Layer 0's heads cost between 0.013 and 0.037 each: the network barely notices. Layer 1's heads cost 0.150, 0.207 and 0.225 for heads 1 to 3, and 0.501 for head 0, which takes the loss to 0.959. The head whose attention looked least decisive carries the most: its 0.61 on "tent" and 0.23 on "made" may be combining two pieces of information the other heads do not. That is the kind of finding worth a paragraph in a report, with this table as its evidence and a pair ablation as the next step.

Quick check
Ranking heads by what zeroing them costs. What does this print?
base = 0.458
losses = {"L1H0": 0.959, "L1H1": 0.608, "L0H2": 0.495}
print(max(losses, key=lambda name: losses[name] - base))
  1. L1H0
  2. L1H1
  3. 0.501
  4. L0H2
Show the answer

L1H0

max with a key picks the name whose cost (loss minus baseline) is largest: L1H0 at +0.501.

Quick check
A head's single ablation costs almost nothing. Which conclusion is safe?
  1. The head does nothing at all and could be deleted from the model without any cost
  2. The head's attention pattern must be uniform over every earlier token
  3. The head is in layer 0, since those were the cheap ones in the tiny GPT
  4. With the other heads in place the network compensates for it; it may still matter with others
Show the answer

With the other heads in place the network compensates for it; it may still matter with others

Single-head ablation measures what the rest cannot cover. Redundant heads look cheap one at a time; remove them in pairs or as a layer to find out.

Drill

Rank heads by their cost

Write rank_heads(losses, base): losses maps a head's name to the loss with that head zeroed, base is the loss with every head in place. Return a list of (name, cost) pairs, cost = loss − base rounded to 3 decimals, largest cost first.

def rank_heads(losses, base):
    return []

losses = {"layer 1 head 0": 0.959, "layer 1 head 1": 0.608, "layer 1 head 2": 0.665,
          "layer 1 head 3": 0.683, "layer 0 head 2": 0.495}
for name, cost in rank_heads(losses, 0.458):
    print(name, cost)
In the lesson, your code is checked as you write it.
Try it now

What a sharp head looks like (illustrative scores shaped like layer 1 head 1): from the position where the material is chosen, almost all of the attention goes to the product type.

In your own words
The head that looked least focused in the attention printout cost the most when ablated. What does that teach about reading attention maps as evidence of importance?

Write a sentence or two, then compare it with a good answer in the lesson.

Ship it: diagnose a GPT you trained

The brief below is the capstone. Its starter is the training loop from the build-a-GPT lesson with the three measurements as functions. By default it trains on the practice catalog; replace the text with a few hundred lines of your own (notes, invented product copy, a synthetic log, anything you may use freely), train it, and then investigate the model you made.

A good report has three findings, each one a claim with its measurement attached: an entropy map that shows where your model is certain and where it guesses, with the two or three positions that surprised you; an attention pattern for one head on at least two prompts, with a careful name for what it fetches or an honest "no consistent pattern"; and an ablation table that ranks the heads, with a sentence on what the top head's removal breaks. End with one sentence on what you would change, more data, more steps, a bigger model, and the measurement that would show whether it helped. The tutor reviews it against the rubric; signed out, the rubric is a checklist to tick yourself.

Portfolio brief

Diagnose your own GPT

A small team ships a tiny language model trained on their own product copy, and the lead wants more than a loss number before anyone trusts it: where is it confident, what has it actually learned to look up, and which parts of it carry the weight? The starter trains the Phase 5 GPT (2 layers, 4 heads, 32-wide) on a text of your choosing; by default it uses the practice catalog, which you should replace with a few hundred lines of your own.

What makes this hard is discipline, not code: each claim in your report must point at a number from one of the three measurements, a pattern seen once must be checked on a second prompt before it gets a name, and an ablation's cost must be read with the caveat that the other heads may compensate. Invented or freely usable text only; nothing personal.

It must

  • Train the model on your own text (at least 300 lines) and report the training and validation loss against the untrained baseline ln(vocabulary size)
  • An entropy map for at least two sentences from your text, with the highest- and lowest-entropy positions named and explained
  • The attention pattern of one head from the last token of at least two different prompts, and a careful statement of what it fetches, or that it has no consistent pattern
  • An ablation table over all eight heads with the baseline, sorted by cost, and a sentence on what the costliest head's removal breaks
  • A closing paragraph: three findings, each with its evidence, and one change you would try with the measurement that would judge it

Go further

  • Ablate pairs of heads and compare the pair's cost with the sum of the two single costs
  • Plot entropy against position across many sentences to find systematic sure and unsure slots
  • Retrain with a different seed and check which findings survive
import math
import re
import requests
import torch
import torch.nn as nn

# 1. Your text. Replace this with a few hundred lines of your own
# (invented or freely usable).
text = requests.get("https://api.gradient.test/v1/corpus").json()["text"]

tokens = re.findall(r"[A-Za-z]+|[0-9]+|[^\sA-Za-z0-9]|\n", text)
vocab = sorted(set(tokens))
stoi = {t: i for i, t in enumerate(vocab)}
data = torch.tensor([stoi[t] for t in tokens])
split = int(0.9 * len(data))
train_data, val_data = data[:split], data[split:]
print(f"{len(tokens):,} tokens, vocabulary of {len(vocab)}, "
      f"untrained loss about {math.log(len(vocab)):.2f}")

def get_batch(d, block=24, batch=16):
    starts = torch.randint(0, len(d) - block - 1, (batch,)).tolist()
    x = torch.stack([d[i:i + block] for i in starts])
    y = torch.stack([d[i + 1:i + block + 1] for i in starts])
    return x, y

# 2. The Phase 5 model.
class Block(nn.Module):
    def __init__(self, d, h):
        super().__init__()
        self.h = h
        self.ln1, self.ln2 = nn.LayerNorm(d), nn.LayerNorm(d)
        self.qkv, self.proj = nn.Linear(d, 3 * d), nn.Linear(d, d)
        self.mlp = nn.Sequential(nn.Linear(d, 4 * d), nn.GELU(), nn.Linear(4 * d, d))

    def attention(self, x, zero=None):
        B, T, d = x.shape
        q, k, v = self.qkv(self.ln1(x)).split(d, dim=-1)
        dh = d // self.h
        if zero is not None:
            v = v.clone(); v[:, :, zero * dh:(zero + 1) * dh] = 0.0
        q, k, v = (t.view(B, T, self.h, dh).transpose(1, 2) for t in (q, k, v))
        s = (q @ k.transpose(-2, -1) / math.sqrt(dh)).masked_fill(
            ~torch.tril(torch.ones(T, T)).bool(), float("-inf"))
        A = torch.softmax(s, dim=-1)
        return self.proj((A @ v).transpose(1, 2).reshape(B, T, d)), A[0]

    def forward(self, x, zero=None):
        out, _ = self.attention(x, zero)
        x = x + out
        return x + self.mlp(self.ln2(x))

class GPT(nn.Module):
    def __init__(self, vocab, context, d, n_layers, n_heads):
        super().__init__()
        self.context = context
        self.tok, self.pos = nn.Embedding(vocab, d), nn.Embedding(context, d)
        self.blocks = nn.ModuleList([Block(d, n_heads) for _ in range(n_layers)])
        self.ln_f, self.head = nn.LayerNorm(d), nn.Linear(d, vocab)

    def forward(self, idx, zero=None):
        x = self.tok(idx) + self.pos(torch.arange(idx.shape[1]))
        for layer, blk in enumerate(self.blocks):
            x = blk(x, zero[1] if zero is not None and zero[0] == layer else None)
        return self.head(self.ln_f(x))

torch.manual_seed(0)
model = GPT(len(vocab), context=24, d=32, n_layers=2, n_heads=4)
opt = torch.optim.AdamW(model.parameters(), lr=3e-3, weight_decay=0.01)

# 3. Train.
for step in range(201):
    x, y = get_batch(train_data)
    loss = nn.functional.cross_entropy(model(x).view(-1, len(vocab)), y.view(-1))
    opt.zero_grad(); loss.backward(); opt.step()
    if step % 50 == 0:
        with torch.no_grad():
            xv, yv = get_batch(val_data, batch=64)
            val = nn.functional.cross_entropy(model(xv).view(-1, len(vocab)),
                                              yv.view(-1)).item()
        print(f"step {step:>3}  training loss {loss.item():.3f}  "
              f"validation loss {val:.3f}")

# 4. The three measurements.
def encode(s):
    ids = [stoi[t]
           for t in re.findall(r"[A-Za-z]+|[0-9]+|[^\sA-Za-z0-9]|\n", s) if t in stoi]
    # the model reads at most 24 tokens, its context
    return torch.tensor([ids[:model.context]])

def entropy_map(sentence):
    ids = encode(sentence)
    with torch.no_grad():
        p = torch.softmax(model(ids)[0], dim=-1)
    H = -(p * (p + 1e-9).log()).sum(-1) / math.log(2)
    return [(vocab[int(t)], round(H[i].item(), 2), vocab[int(p[i].argmax())])
            for i, t in enumerate(ids[0])]

def attention_from_last(sentence, layer, head):
    ids = encode(sentence)
    with torch.no_grad():
        x = model.tok(ids) + model.pos(torch.arange(ids.shape[1]))
        for l, blk in enumerate(model.blocks):
            out, A = blk.attention(x)
            if l == layer:
                return [(vocab[int(t)], round(A[head, -1, j].item(), 2))
                        for j, t in enumerate(ids[0])]
            x = x + out
            x = x + blk.mlp(blk.ln2(x))

def ablation_table(sentences):
    def mean_loss(zero=None):
        total, count = 0.0, 0
        for s in sentences:
            ids = encode(s)
            if ids.shape[1] < 3:
                continue
            with torch.no_grad():
                logp = torch.log_softmax(model(ids[:, :-1], zero), dim=-1)
                nll = -logp[0, torch.arange(ids.shape[1] - 1), ids[0, 1:]]
            total += nll.sum().item(); count += nll.numel()
        return total / count
    base = mean_loss()
    rows = [(f"layer {l} head {h}", round(mean_loss((l, h)) - base, 3))
            for l in range(2) for h in range(4)]
    return base, sorted(rows, key=lambda r: r[1], reverse=True)

sentences = [s.strip() for s in text.split("\n") if s.strip()][:20]
print(entropy_map(sentences[0]))
print(attention_from_last(sentences[0], 1, 0))
base, table = ablation_table(sentences)
print(f"baseline {base:.3f}", table[:3])

# your code: run the three measurements on your own text, check each attention
# claim on a second prompt, and write your three findings as comments or prints below.
In the lesson, you build it here and the tutor can review it.
Quick check
Which of these is a finding in the sense this capstone asks for?
  1. "The attention heads are clearly doing something interesting in the second layer, as the printouts show"
  2. "Zeroing layer 1 head 0 raises the mean loss over 40 sentences from 0.458 to 0.959; no layer 0 head adds more than 0.037"
  3. "Head 2 attends to the product type", concluded from the one prompt the lesson used
  4. "The model learned the data well, as the training loss went down steadily from the first step"
Show the answer

"Zeroing layer 1 head 0 raises the mean loss over 40 sentences from 0.458 to 0.959; no layer 0 head adds more than 0.037"

A finding names the measurement, the numbers and the comparison. The others are impressions, an unquantified loss claim, and a one-prompt hypothesis.

Where this shows up in an engineer's week

On the job: these three measurements are the first things to run on a model that behaves oddly. Entropy tells you where it is guessing, attention tells you what it is reading, and an ablation tells you which part to blame. Keep the baseline loss next to every ablation, and measure on held-out text, not the text the model trained on.

In your own words
Before you write a finding about your own model, what two checks from this lesson should every claim pass?

Write a sentence or two, then compare it with a good answer in the lesson.

You can now

  • Map a model's certainty with the entropy of its next-token distribution at every position, and separate structure from content.
  • Read every head's attention from a position and test a hypothesis about what a head fetches on a second prompt.
  • Ablate heads one at a time, rank them by cost, and read the ranking with the compensation caveat.
  • Train the Phase 5 GPT on your own text and write up three findings with their evidence.

This closes Phase 5's main path. The optional "Math under the GPT" lessons, from Jacobians: derivatives with shapes on, derive its gradients by hand. Phase 6 takes the same model into post-training, adaptation and inference at scale.

Practice what you just read

  • 3 code examples you can edit and run
  • a practice exercise that checks your code
  • a 5-question quiz
  • 9 review cards for spaced repetition

Try a quiz question

The tiny GPT's next-token entropy is 0.01 bits after "is made" and 1.12 bits after "from". The best description is:

  1. The frame is memorized; the material is a real choice among several seen with tents
  2. The model is more accurate on short words such as 'is' than on long ones like 'silnylon'
  3. The model has forgotten the materials it was trained on and guesses at random
  4. Entropy rises with the position in the sentence, whatever the words are

Answer it at the end of the lesson to see if you're right, and why.

Before this lesson

Keep learning

Where this lesson leads.

575 lessons in thirteen phases. 176 are free without an account, and a free account opens the rest. See every lesson.

Phase 5 · about 36 hours of study

Transformers

Attention, GPT from scratch and pretraining 47 lessons 1 free
  1. Language modeling FreePredict the next token, and score it with cross-entropy.30 min
  2. Tokenization and BPEHow tokenizers are built from data.40 min
  3. A bigram modelThe simplest language model, built from counts.35 min
  4. Sampling strategiesTemperature, top-k and top-p: turning probabilities into text.35 min
  5. Beam search and constrained decodingWhy the likeliest word isn't the likeliest sentence, beam search with log probabilities, its taste for short and repetitive text, and forcing output into a grammar: what that guarantees and what it doesn't.40 min
  6. Neural language modelsWhy counting fails, and what embeddings fix.35 min
  7. Attention: the intuitionEach token decides which other tokens matter to it.30 min
  8. Queries, keys and valuesThe three projections behind attention.35 min
  9. Scaled dot-product attentionThe attention formula, line by line.35 min
  10. Causal maskingStop tokens from seeing the future.25 min
  11. Multi-head attentionSeveral attention patterns in parallel.40 min
  12. Positional encodingsTell the model where each token sits.30 min
  13. The transformer blockAttention, MLP, residuals and norms together.40 min
  14. The GPT architectureA stack of decoder blocks that predicts the next token.40 min
  15. Build a tiny GPTWrite and train a small GPT from scratch.45 min
  16. From your GPT to GPT-2 in Hugging FaceMatch your GPT's tensors to GPT-2's names and shapes, count its 124,439,808 parameters, load renamed weights into your own class, pad a batch and read a config.40 min
  17. Training a GPT wellWarmup and cosine decay, gradient clipping, a validation loss you can trust, saving and loading, and two training mistakes read off the loss curve.35 min
  18. Encoders and embedding modelsBERT-style models and where they fit.40 min
  19. Tokenization: text to integer IDsFollow "The cat sat" into the three integers a model receives.30 min
  20. Embeddings and positional wavesFrom three IDs to the 3 × 8 matrix that enters the first block.35 min
  21. Inside the attention matrixEvery entry of the 3 × 3 attention matrix, computed and steered.40 min
  22. The feed-forward layerExpand to four times d_model and back: shapes, parameters and what the layer stores.35 min
  23. The LM head and samplingFrom the last row to a probability for every token, and the next word.35 min
  24. Pretraining at scaleData, compute and what models learn from the internet.40 min
  25. Scaling lawsHow loss falls predictably with size, data and compute.40 min
  26. Inference and the KV cacheWhy generation is fast after the first token.40 min
  27. The block in today's open models OptionalRMSNorm, SwiGLU, rotary positions and grouped-query attention swapped into the block you built, with the parameter and cache counts before and after.35 min
  28. Long contextEfficient attention and million-token windows.35 min
  29. Mixture of expertsRoute each token to a few specialist sub-networks.35 min
  30. Capstone: diagnose your own GPTEntropy per position, a head's attention pattern and a head ablation on the tiny GPT, then the same three measurements on a GPT you train yourself.45 min
  31. Jacobians: derivatives with shapes OptionalThe derivative of a vector function is a matrix; the chain rule multiplies them, and backprop multiplies by them without building them.35 min
  32. Backprop through a linear layer OptionaldX = dY Wᵀ, dW = Xᵀ dY, db = column sums: derived by shape and by entry, checked on the tiny GPT's own projection.35 min
  33. The softmax Jacobian and p − y OptionalEvery output depends on every input: the full Jacobian, its cheap vector product, and why cross-entropy's gradient is p − y.35 min
  34. Backprop through attention OptionalFive formulas from the output back to queries, keys and values; why masked scores get zero gradient; what FlashAttention recomputes.40 min
  35. Multimodal transformersHow models see images and hear audio.35 min
  36. Object detection: boxes, overlap and suppressionEvery object and where it is: boxes in two formats, intersection over union, why one object gets many boxes, non-maximum suppression and its threshold, and grading with average precision.45 min
  37. Segmentation: a label for every pixelMasks, and how to grade them with IoU and Dice; why a pixel's label needs context and too much context blurs; the shrink-then-enlarge network with skip connections; and turning a mask into countable objects.45 min
  38. Build a vision transformerPatch embeddings, a class token, position embeddings and an unmasked block, trained in the page; then two failures that explain it: no positions means no layout, and few images means losing to a CNN a fraction of its size.45 min
  39. Image-text models in depth: retrieval, negatives and limitsHow to test a shared image-text space with recall at k, what the rest of the batch teaches, what hard and false negatives do, and why a similarity score can't see word order, counts or "no".45 min
  40. Self-supervised learning: labels from the data itselfPretrain an encoder with no labels by matching two views of each image, test it with a linear probe, and see what the augmentations remove, what collapse looks like, and how filling in masked parts differs.45 min
  41. Diffusion 1: turning data into noiseThe easy half of a diffusion model: one step of noise, why the data is shrunk as noise is added, the jump to any step with alpha bar, how to read a noise schedule, and the training examples a denoiser learns from.40 min
  42. Diffusion 2: learning to denoise, and samplingTrain a network to predict noise, then run it a hundred times from pure noise and watch a ring assemble: the sampling loop, why fresh noise goes back in, fewer steps and where they break, and a check that samples are new.45 min
  43. Diffusion 3: images, labels and guidanceFrom points to small images, a label the denoiser can be asked with, and classifier-free guidance: why a model that is told what to draw still often doesn't, how the guidance scale fixes it, and what it breaks when turned up too far.45 min
  44. Diffusion 4: latent diffusion and the whole pipelineWhy image generators diffuse an autoencoder's codes and not pixels: a latent diffusion model in miniature, what the autoencoder must be trained to do, the three parts of a text-to-image system, and where the cost of an image goes.40 min
  45. Sound as numbers: waveforms and spectrogramsA recording as an array of samples, why the sampling rate must be twice the highest pitch, spectrograms and the window trade-off, and the log-mel features a speech model is given.40 min
  46. Recognition and synthesis as a pipelineA whole speech recognizer for a made-up language: features, a guess per frame, collapsing repeats and blanks, a dictionary to repair mistakes, word error rate, and synthesis as the same stages run backward.40 min
  47. The voice agent loop and its latency budgetOne turn of a voice agent, timed: why every stage must stream, how to decide that someone has finished speaking, what to do when they interrupt, and why the slow turns matter more than the average.45 min

Never trained an AI? Teach a small one to write, right in your browser. It takes about 20 seconds and needs no account. Teach it to write