Index Ai Academy
Phase 5 · Math under the GPT

Backprop through attention

Five formulas from the output back to queries, keys and values; why masked scores get zero gradient; what FlashAttention recomputes.

About 40 minutes · Optional lesson: deeper material that no phase or certificate requires · A free account opens the whole lesson.
Already have an account? Sign in to open this lesson

What it covers10

Why it matters

Attention is a chain of pieces you can now differentiate: a matrix product QK⊤/dQK^{\top}/\sqrt{d}, a mask, a softmax on every row, and another matrix product with VV. Its backward pass is therefore a chain of known rules: the linear-layer formulas for the two products, the softmax vector-Jacobian product for each row, and nothing at all for the masked positions.

This lesson writes the five formulas, dVdV, dAdA, dSdS, dQdQ and dKdK, checks each against autograd on three tokens and on head 0 of the tiny GPT's first block, then carries the gradient into WQW_Q, WKW_K and WVW_V. You will see why masked scores get exactly zero gradient, why each row of the score gradient sums to zero, and why FlashAttention recomputes the attention matrix instead of storing it: at 8,192 tokens, 32 heads and 32 layers it is 274.88 GB in float32.

What you'll be able to do

  • Write the five gradients of one attention head, dV, dA, dS, dQ and dK, from the rules for linear layers and softmax
  • Explain why masked scores receive exactly zero gradient and why every row of dS sums to zero
  • Carry the gradient into W_Q, W_K and W_V and check it against autograd, including on the tiny GPT's own head
  • Say what FlashAttention recomputes in the backward pass and how much memory that saves
Quick check
The output of a head is O = AV, with A the T × T attention weights and V the T × d values. The gradient with respect to V is:
  1. dO Aᵀ, shape T × T: the output gradient spread over the attention pattern
  2. A dO, shape T × d: each value gets the output gradient of its own position
  3. Aᵀ dO, shape T × d: each value collects the output gradients of the tokens that attended to it
  4. dO Vᵀ, shape T × T: the gradient with respect to the weights, which V shares
Show the answer

Aᵀ dO, shape T × d: each value collects the output gradients of the tokens that attended to it

O = AV is a linear layer with A as the input and V as the weight, so dV = Aᵀ dO by the rule of Backprop through a linear layer: (T × T)(T × d) = T × d.

The idea: the forward pass, run backward

This rests on Attention as matrix math, the forward pass by hand, and on The softmax Jacobian and p − y, the one step that needs care.

The forward pass as four steps

One attention head, for TT tokens with head width dd: queries QQ, keys KK and values VV, each T×dT \times d.

  1. Scores: S=QK⊤/dS = QK^{\top}/\sqrt{d}, a T×TT \times T matrix, a matrix product.
  2. Mask: entries above the diagonal become −∞-\infty, so no token reads a later one.
  3. Weights: A=softmax(S)A = \mathrm{softmax}(S), row by row.
  4. Output: O=AVO = AV, T×dT \times d, another matrix product.

Every step is one you can now differentiate: the linear-layer rules for steps 1 and 4, the softmax vector-Jacobian product for step 3, and for step 2 nothing at all, as you will see. The next cell runs the forward pass on three tokens with hand-picked QQ, KK and VV, chosen so the attention matrix has readable numbers.

Worked example

Forward on three tokens

Hand-picked Q, K (the same matrix) and V with head width 4. The masked scores, the attention weights and the output.

import math
import torch

Q = torch.tensor([[1.0, 0.0, 1.0, 0.0], [0.0, 1.0, 0.0, 1.0], [1.0, 1.0, 0.0, 0.0]])
K = torch.tensor([[1.0, 0.0, 1.0, 0.0], [0.0, 1.0, 0.0, 1.0], [1.0, 1.0, 0.0, 0.0]])
V = torch.tensor([[1.0, 0.0], [0.0, 1.0], [0.5, 0.5]])
d = Q.shape[1]
mask = torch.tril(torch.ones(3, 3)).bool()
S = (Q @ K.T / math.sqrt(d)).masked_fill(~mask, float("-inf"))
A = torch.softmax(S, dim=-1)
O = A @ V
print("S:", S)
print("A:", A)
print("O:", O)
It runs right here in your browser, with nothing to install.

Token 0 can read only itself, so its row of A is one-hot and its output is its own value, [1, 0]. Token 1 scores 0 against token 0 and 1 against itself: the softmax of [0, 1] is [0.2689, 0.7311]. Token 2 scores 0.5, 0.5 and 1: weights [0.2741, 0.2741, 0.4519], and its output is that blend of the three values, which comes to [0.5, 0.5]. The −inf entries of S became exactly 0 in A.

Quick check
dV = Aᵀ dO on a two-token example. What does this print?
import torch
A = torch.tensor([[1.0, 0.0], [0.5, 0.5]])
dO = torch.tensor([[1.0, 2.0], [3.0, 4.0]])
print((A.T @ dO).tolist())
  1. [[2.5, 4.0], [1.5, 2.0]]
  2. [[2.5, 1.5], [4.0, 2.0]]
  3. [[1.0, 2.0], [2.0, 3.0]]
  4. [[4.0, 2.0], [2.5, 1.5]]
Show the answer

[[2.5, 4.0], [1.5, 2.0]]

Value 0 was read by token 0 with weight 1 and by token 1 with weight 0.5: 1·[1, 2] + 0.5·[3, 4] = [2.5, 4]. Value 1 was read only by token 1 with weight 0.5: [1.5, 2].

The five formulas

Backprop hands you dOdO, the gradient at the output, T×dT \times d. Walk the four steps in reverse.

Step 4, O=AVO = AV, is a linear layer with AA in the role of the input and VV in the role of the weight: dV=A⊤dOdV = A^{\top} dO and dA=dO V⊤dA = dO\,V^{\top}.

Step 3, the row softmax: each row of dSdS is the softmax vector-Jacobian product of that row, dS=A⊙(dA−rowsum(dA⊙A))dS = A \odot (dA - \mathrm{rowsum}(dA \odot A)), the formula of The softmax Jacobian and p − y applied to every row at once.

Step 2, the mask: where AA is 0, dSdS is 0, because the product above multiplies by AA. Nothing flows to the masked scores, and nothing needs to.

Step 1, S=QK⊤/dS = QK^{\top}/\sqrt{d}: for QQ it is a linear layer with K⊤/dK^{\top}/\sqrt{d} as the weight, so dQ=dS K/ddQ = dS\,K/\sqrt{d}; for KK the roles swap, dK=dS⊤Q/ddK = dS^{\top} Q/\sqrt{d}.

Five formulas, each with a shape you can check: dVdV, dQdQ and dKdK are T×dT \times d; dAdA and dSdS are T×TT \times T. The next cell computes all five by hand and prints autograd's answer under each.

Worked example

Backward by hand, autograd underneath

The same head with a chosen dO. Each hand formula, then the gradient autograd stored for the same tensor.

import math
import torch

Q = torch.tensor([[1.0, 0.0, 1.0, 0.0], [0.0, 1.0, 0.0, 1.0], [1.0, 1.0, 0.0, 0.0]],
                 requires_grad=True)
K = torch.tensor([[1.0, 0.0, 1.0, 0.0], [0.0, 1.0, 0.0, 1.0], [1.0, 1.0, 0.0, 0.0]],
                 requires_grad=True)
V = torch.tensor([[1.0, 0.0], [0.0, 1.0], [0.5, 0.5]], requires_grad=True)
d = Q.shape[1]
mask = torch.tril(torch.ones(3, 3)).bool()
S = (Q @ K.T / math.sqrt(d)).masked_fill(~mask, float("-inf"))
A = torch.softmax(S, dim=-1)
O = A @ V
S.retain_grad(); A.retain_grad()
# the gradient arriving at the output
dO = torch.tensor([[1.0, 0.0], [0.0, 1.0], [1.0, 1.0]])
(O * dO).sum().backward()

Ad, Kd, Qd, Vd = A.detach(), K.detach(), Q.detach(), V.detach()
dV = Ad.T @ dO
dA = dO @ Vd.T
dS = Ad * (dA - (dA * Ad).sum(-1, keepdim=True))
dQ = dS @ Kd / math.sqrt(d)
dK = dS.T @ Qd / math.sqrt(d)
for name, hand, auto in [("dV", dV, V.grad), ("dA", dA, A.grad), ("dS", dS, S.grad),
                         ("dQ", dQ, Q.grad), ("dK", dK, K.grad)]:
    print(f"{name} hand:", hand)
    print(f"{name} auto:", auto)
In the lesson you can change this and run it.

All five pairs agree. dV collects each value's share of the output gradient: value 2 was read only by token 2, with weight 0.4519, so it gets 0.4519 × [1, 1]. dA is dO Vᵀ. Then two rows of dS vanish. Row 0, because token 0's attention is one-hot, and a one-hot softmax has zero gradient: p ⊙ (g − g·p) is 0 when p picks out g's own entry. Row 2, because its upstream row of dA is uniform, [1, 1, 1], and a constant gradient across a softmax row cancels out. Only token 1's row carries gradient, ±0.1966, so only token 1's query moves (dQ row 1) and only keys 0 and 1 receive gradient (dK rows 0 and 1). The masked entries above the diagonal are exactly zero.

Quick check
Row 0 of dS is zero. Why?
  1. The mask removed token 0's scores, so nothing was computed for its row
  2. Token 0 attends only to itself, and a one-hot softmax row has zero gradient, whatever arrives
  3. Token 0 has no output gradient, since the first row of dO is [1, 0]
  4. Row 0 of dA is zero, so there was nothing to pass back through the softmax
Show the answer

Token 0 attends only to itself, and a one-hot softmax row has zero gradient, whatever arrives

p ⊙ (g − g·p) with p = [1, 0, 0] gives [g_0 − g_0, 0, 0] = 0. A softmax that has already saturated cannot be moved by its scores.

Try it now

The mask on a row of attention (illustrative scores). Remove it and put it back: a masked cell holds weight 0, and in the backward pass gradient 0, whatever the score behind it.

In your own words
Token 2's output came out as exactly [0.5, 0.5]. Explain why from the attention weights and values, and say what that means for how its gradient will be shared.

Write a sentence or two, then compare it with a good answer in the lesson.

In PyTorch: checked against autograd

Masks and zero-sum rows

Two properties of dSdS deserve their own look, because production code depends on them.

Masked scores get exactly zero gradient. The mask sets a score to −∞-\infty, the softmax turns it into a weight of exactly 0, and the vector-Jacobian product multiplies by that weight. So no gradient ever reaches a future position's score, which is what makes teacher forcing sound: the whole sequence is trained in one pass, every position predicting the next token, and no position's loss leaks into the queries and keys that would let it peek ahead.

Every row of dSdS sums to zero. Softmax ignores a constant added to a row of scores, so the loss cannot depend on a row's overall level, and the gradient has no component in that direction. In the printout above, row 1 is [−0.1966,0.1966,0][-0.1966, 0.1966, 0].

The next cell checks both on the tiny GPT itself: head 0 of the first block, on "Our Kestrel tent is made from", with the model's own queries, keys and values. The upstream gradient is random, which is all the formulas need.

Worked example

Head 0 of the tiny GPT's first block

The model's own q, k and v for head 0 on a six-token prompt, a random dO, and the hand formulas against autograd.

import math
import torch
from tinygpt import load_pretrained

model, tok = load_pretrained("tiny-gpt")
ids = torch.tensor([tok.encode("Our Kestrel tent is made from")])
T, blk, dh = ids.shape[1], model.blocks[0], 32 // model.blocks[0].h
with torch.no_grad():
    h = model.tok(ids) + model.pos(torch.arange(T))
    q_all, k_all, v_all = blk.qkv(blk.ln1(h)).split(32, dim=-1)
# head 0: the first 8 of 32 columns
q = q_all[0, :, :dh].detach().clone().requires_grad_(True)
k = k_all[0, :, :dh].detach().clone().requires_grad_(True)
v = v_all[0, :, :dh].detach().clone().requires_grad_(True)
mask = torch.tril(torch.ones(T, T)).bool()
A = torch.softmax((q @ k.T / math.sqrt(dh)).masked_fill(~mask, float("-inf")), dim=-1)
torch.manual_seed(0)
dO = torch.randn(T, dh)
((A @ v) * dO).sum().backward()

Ad = A.detach()
dA = dO @ v.detach().T
dS = Ad * (dA - (dA * Ad).sum(-1, keepdim=True))
dQ, dK, dV = (dS @ k.detach() / math.sqrt(dh), dS.T @ q.detach() / math.sqrt(dh),
              Ad.T @ dO)
print("tokens", T, " head width", dh)
print("attention from 'from':",
      [f"{tok.vocab[int(i)]} {Ad[-1, j].item():.2f}" for j, i in enumerate(ids[0])])
print("max |hand - autograd|: dQ", (dQ - q.grad).abs().max().item(), " dK",
      (dK - k.grad).abs().max().item(), " dV", (dV - v.grad).abs().max().item())
print("largest |dS| above the diagonal:", dS.masked_fill(mask, 0.0).abs().max().item())
print("row sums of dS:", [round(x, 6) for x in dS.sum(-1).tolist()])
print("|dQ| per token:", [round(x, 3) for x in (dQ ** 2).sum(-1).sqrt().tolist()])
print("|dK| per token:", [round(x, 3) for x in (dK ** 2).sum(-1).sqrt().tolist()])

import matplotlib.pyplot as plt
fig, ax = plt.subplots(figsize=(4.5, 3.8))
im = ax.imshow(dS.tolist(), cmap="RdBu", vmin=-0.3, vmax=0.3)
ax.set_xticks(range(T), [tok.vocab[int(i)] for i in ids[0]], rotation=60)
ax.set_yticks(range(T), [tok.vocab[int(i)] for i in ids[0]])
ax.set_title("dS: zero above the diagonal, rows sum to zero")
fig.colorbar(im, ax=ax)
In the lesson you can change this and run it.

The formulas hold on real numbers to 3e-08. "from" spreads its attention over the six tokens, most on "tent" (0.30) and "is" (0.23). Nothing above the diagonal in dS, and every row sums to zero. Two numbers show the structure the mask imposes: token 0's query gets no gradient at all (|dQ| = 0.0: its attention row is one-hot), and the last key gets almost none (|dK| = 0.021), because only one query, the last, is allowed to read it. W_Q and W_K are shared by every position, so they still train: in a causal model, the first position's query and the last position's key are where the least gradient flows back into them.

Quick check
A one-hot attention row and any upstream gradient. What does this print?
import torch
A = torch.tensor([[1.0, 0.0]])
dA = torch.tensor([[2.0, 7.0]])
print((A * (dA - (dA * A).sum(-1, keepdim=True))).tolist())
  1. [[2.0, 0.0]]
  2. [[0.0, 0.0]]
  3. [[-5.0, 5.0]]
  4. [[0.0, 7.0]]
Show the answer

[[0.0, 0.0]]

The weighted mean of dA under A = [1, 0] is 2, so dA − 2 = [0, 5], and multiplying by A = [1, 0] leaves [0, 0]. Saturated attention passes no gradient to its scores.

Quick check
Under the causal mask, which key receives the least gradient, and why?
  1. The middle token's key: it is read by exactly half of the queries
  2. The first token's key: every query reads it, so the contributions cancel out
  3. The last token's key: only the last query may read it, so one row of dS reaches it
  4. All keys receive the same gradient, since every row of dS sums to zero
Show the answer

The last token's key: only the last query may read it, so one row of dS reaches it

dK = dSᵀ Q/√d sums each key's column of dS over the queries allowed to read it. The last key has one reader; in the tiny GPT example its gradient size was 0.021 against 0.554 for the first key.

Into the weights, and what FlashAttention saves

QQ, KK and VV are not parameters; they come from the block's input XX through three linear layers, Q=XWQQ = XW_Q and so on, in the tiny GPT one 32 → 96 layer cut in three. The linear-layer rules finish the job: dWQ=X⊤dQdW_Q = X^{\top} dQ, dWK=X⊤dKdW_K = X^{\top} dK, dWV=X⊤dVdW_V = X^{\top} dV, and the input collects all three, dX=dQ WQ⊤+dK WK⊤+dV WV⊤dX = dQ\,W_Q^{\top} + dK\,W_K^{\top} + dV\,W_V^{\top}, because XX was used three times. The next cell checks the three weight gradients against autograd on an 8-wide input.

Then the cost. The backward pass needs AA twice, for dVdV and for dSdS. Storing it from the forward pass costs heads × T2T^2 numbers per layer: for the tiny GPT at 48 tokens, 18,432 floats, nothing. For a model with 32 heads and 32 layers at 2,048 tokens it is 4.29 billion floats, 17.18 GB in float32; at 8,192 tokens, 274.88 GB, more than any single GPU holds. FlashAttention keeps only each softmax row's maximum and sum from the forward pass and recomputes AA tile by tile during the backward pass: extra arithmetic, which GPUs have, instead of memory, which they do not. The last cell does that arithmetic.

Worked example

Into W_Q, W_K and W_V

Three tokens of width 8, three projection matrices, one causal head. Xᵀ times each of dQ, dK, dV against the weights' autograd gradients.

import math
import torch

torch.manual_seed(0)
X = torch.randn(3, 8)                                     # three tokens, 8 numbers each
WQ, WK, WV = (torch.randn(8, w, requires_grad=True) for w in (4, 4, 2))
Q, K, V = X @ WQ, X @ WK, X @ WV
for t in (Q, K, V): t.retain_grad()
mask = torch.tril(torch.ones(3, 3)).bool()
A = torch.softmax((Q @ K.T / math.sqrt(4)).masked_fill(~mask, float("-inf")), dim=-1)
dO = torch.tensor([[1.0, 0.0], [0.0, 1.0], [1.0, 1.0]])
((A @ V) * dO).sum().backward()
for name, W, dP in [("W_Q", WQ, Q.grad), ("W_K", WK, K.grad), ("W_V", WV, V.grad)]:
    print(f"{name}: shape {tuple(W.grad.shape)}, max |X^T dP - autograd| "
          f"= {(X.T @ dP - W.grad).abs().max().item():.1e}")
In the lesson you can change this and run it.

Three zeros. The attention backward delivered dQ, dK and dV at the projections' outputs; the linear-layer rule carries each to its weight with the same Xᵀ on the left. In the tiny GPT the three weights are one 96 × 32 matrix, so the three gradients sit side by side in one weight.grad, and the output projection and the feed-forward layer follow with the same rule. Every parameter of an attention block is reached by the formulas of these four lessons.

Drill

From the score gradient to queries and keys

Write scores_backward(dS, Q, K) for S = QKᵀ/√d, where d is the number of columns of Q. Return the tuple (dQ, dK): dQ = dS K/√d and dK = dSᵀ Q/√d.

import math
import torch

def scores_backward(dS, Q, K):
    return Q, K

Q = torch.tensor([[1.0, 0.0, 1.0, 0.0], [0.0, 1.0, 0.0, 1.0], [1.0, 1.0, 0.0, 0.0]])
dS = torch.tensor([[0.0, 0.0, 0.0], [-0.1966, 0.1966, 0.0], [0.0, 0.0, 0.0]])
dQ, dK = scores_backward(dS, Q, Q.clone())
print(dQ)
In the lesson, your code is checked as you write it.
Drill

The softmax backward for every row

Write softmax_rows_backward(A, dA): A holds softmax outputs row by row (each row sums to 1) and dA the gradient arriving at A. Return dS, the gradient with respect to the scores: for each row, A * (dA - (dA * A).sum()), done for all rows at once with sum(-1, keepdim=True).

import torch

def softmax_rows_backward(A, dA):
    return dA

A = torch.tensor([[1.0, 0.0], [0.5, 0.5]])
dA = torch.tensor([[2.0, 7.0], [1.0, 3.0]])
print(softmax_rows_backward(A, dA))
In the lesson, your code is checked as you write it.
In your own words
Under a causal mask, the first position's query and the last position's key send the least gradient back. W_Q and W_K are shared by every position, so what does that mean for training?

Write a sentence or two, then compare it with a good answer in the lesson.

In real models: what the backward pass costs

Everything above is cheap in arithmetic and expensive in memory. The map shows the node whose table is stored.

Worked example

The attention matrices' memory

heads × T² × layers numbers, for the tiny GPT and for two production-sized configurations.

for T, heads, layers in [(48, 4, 2), (2048, 32, 32), (8192, 32, 32)]:
    floats = heads * T * T * layers
    print(f"T={T} heads={heads} layers={layers}: {floats:,} floats, "
          f"{floats * 4 / 1e9:.2f} GB in float32, "
          f"{floats * 2 / 1e9:.2f} GB in bfloat16")
In the lesson you can change this and run it.

The attention matrices grow with the square of the sequence length and linearly with heads and layers. The tiny GPT's 18,432 floats are a rounding error. A production configuration at 2,048 tokens already needs 17.18 GB in float32 (8.59 GB in bfloat16) just to remember the attention weights for the backward pass; at 8,192 tokens, 274.88 GB. Recomputing them from the stored row statistics costs a second pass over the QKᵀ products and is what makes long-context training possible.

Quick check
From the causal-masking lesson: why must a language model not attend to later tokens during training?
  1. The positional encodings only carry information in one direction
  2. The attention weights would no longer sum to one across the row
  3. Later tokens have not been embedded yet when the earlier ones are processed
  4. It would read the token it is supposed to predict, and learn nothing useful
Show the answer

It would read the token it is supposed to predict, and learn nothing useful

With the answer in view, the easiest solution is to copy it, which does not work at generation time when the answer does not exist yet. The mask, and the zero gradient behind it, keep training honest.

Quick check
What does FlashAttention's backward pass trade, and for what?
  1. A shorter context window during the backward pass, for a faster forward pass overall
  2. Approximate attention weights in the backward pass, for skipping the softmax backward entirely
  3. Lower precision in the softmax, for an attention matrix that takes half the memory
  4. Extra arithmetic to recompute A tile by tile, for not storing heads × T² numbers per layer
Show the answer

Extra arithmetic to recompute A tile by tile, for not storing heads × T² numbers per layer

It keeps each row's maximum and sum, recomputes the weights when needed, and gets the exact same gradients with far less memory.

Where this shows up in an engineer's week

On the job: when a long-context run runs out of memory, the cause is the stored attention weights, which grow with the square of the sequence length. The fixes are in this lesson: a fused attention kernel that recomputes them, or checkpointing whole blocks. Both trade compute for memory, and both leave the gradients unchanged, so you can compare a run with and without them and expect the same loss.

Goes deeper in

In your own words
Storing the attention matrices for the backward pass costs heads × T² × layers numbers. Why does that grow faster than the model's parameters when the context gets longer, and what did the field do about it?

Write a sentence or two, then compare it with a good answer in the lesson.

You can now

  • Write dV = Aᵀ dO, dA = dO Vᵀ, dS = A ⊙ (dA − rowsum(dA ⊙ A)), dQ = dS K/√d and dK = dSᵀ Q/√d, and check them.
  • Explain why masked scores get exactly zero gradient and why every row of dS sums to zero.
  • Carry the gradient into W_Q, W_K and W_V with Xᵀ dQ and its siblings, and into X with the three terms summed.
  • Say what FlashAttention recomputes in the backward pass and how much memory it saves.

This closes the "Math under the GPT" track: every parameter of an attention block is now reached by rules you have derived and checked.

Practice what you just read

  • 5 code examples you can edit and run
  • a practice exercise that checks your code
  • a 5-question quiz
  • 10 review cards for spaced repetition

Try a quiz question

For O = AV, the gradient with respect to the values is:

  1. dV = Aᵀ dO: each value collects the output gradients of the tokens that attended to it
  2. dV = A dO: each value gets the output gradient at its own position and nothing else
  3. dV = dO Vᵀ: the output gradient spread across the values by their own entries
  4. dV = dO, since the values pass straight through to the output unchanged

Answer it at the end of the lesson to see if you're right, and why.

Before this lesson

Keep learning

Where this lesson leads.

575 lessons in thirteen phases. 176 are free without an account, and a free account opens the rest. See every lesson.

Phase 5 · about 36 hours of study

Transformers

Attention, GPT from scratch and pretraining 47 lessons 1 free
  1. Language modeling FreePredict the next token, and score it with cross-entropy.30 min
  2. Tokenization and BPEHow tokenizers are built from data.40 min
  3. A bigram modelThe simplest language model, built from counts.35 min
  4. Sampling strategiesTemperature, top-k and top-p: turning probabilities into text.35 min
  5. Beam search and constrained decodingWhy the likeliest word isn't the likeliest sentence, beam search with log probabilities, its taste for short and repetitive text, and forcing output into a grammar: what that guarantees and what it doesn't.40 min
  6. Neural language modelsWhy counting fails, and what embeddings fix.35 min
  7. Attention: the intuitionEach token decides which other tokens matter to it.30 min
  8. Queries, keys and valuesThe three projections behind attention.35 min
  9. Scaled dot-product attentionThe attention formula, line by line.35 min
  10. Causal maskingStop tokens from seeing the future.25 min
  11. Multi-head attentionSeveral attention patterns in parallel.40 min
  12. Positional encodingsTell the model where each token sits.30 min
  13. The transformer blockAttention, MLP, residuals and norms together.40 min
  14. The GPT architectureA stack of decoder blocks that predicts the next token.40 min
  15. Build a tiny GPTWrite and train a small GPT from scratch.45 min
  16. From your GPT to GPT-2 in Hugging FaceMatch your GPT's tensors to GPT-2's names and shapes, count its 124,439,808 parameters, load renamed weights into your own class, pad a batch and read a config.40 min
  17. Training a GPT wellWarmup and cosine decay, gradient clipping, a validation loss you can trust, saving and loading, and two training mistakes read off the loss curve.35 min
  18. Encoders and embedding modelsBERT-style models and where they fit.40 min
  19. Tokenization: text to integer IDsFollow "The cat sat" into the three integers a model receives.30 min
  20. Embeddings and positional wavesFrom three IDs to the 3 × 8 matrix that enters the first block.35 min
  21. Inside the attention matrixEvery entry of the 3 × 3 attention matrix, computed and steered.40 min
  22. The feed-forward layerExpand to four times d_model and back: shapes, parameters and what the layer stores.35 min
  23. The LM head and samplingFrom the last row to a probability for every token, and the next word.35 min
  24. Pretraining at scaleData, compute and what models learn from the internet.40 min
  25. Scaling lawsHow loss falls predictably with size, data and compute.40 min
  26. Inference and the KV cacheWhy generation is fast after the first token.40 min
  27. The block in today's open models OptionalRMSNorm, SwiGLU, rotary positions and grouped-query attention swapped into the block you built, with the parameter and cache counts before and after.35 min
  28. Long contextEfficient attention and million-token windows.35 min
  29. Mixture of expertsRoute each token to a few specialist sub-networks.35 min
  30. Capstone: diagnose your own GPTEntropy per position, a head's attention pattern and a head ablation on the tiny GPT, then the same three measurements on a GPT you train yourself.45 min
  31. Jacobians: derivatives with shapes OptionalThe derivative of a vector function is a matrix; the chain rule multiplies them, and backprop multiplies by them without building them.35 min
  32. Backprop through a linear layer OptionaldX = dY Wᵀ, dW = Xᵀ dY, db = column sums: derived by shape and by entry, checked on the tiny GPT's own projection.35 min
  33. The softmax Jacobian and p − y OptionalEvery output depends on every input: the full Jacobian, its cheap vector product, and why cross-entropy's gradient is p − y.35 min
  34. Backprop through attention OptionalFive formulas from the output back to queries, keys and values; why masked scores get zero gradient; what FlashAttention recomputes.40 min
  35. Multimodal transformersHow models see images and hear audio.35 min
  36. Object detection: boxes, overlap and suppressionEvery object and where it is: boxes in two formats, intersection over union, why one object gets many boxes, non-maximum suppression and its threshold, and grading with average precision.45 min
  37. Segmentation: a label for every pixelMasks, and how to grade them with IoU and Dice; why a pixel's label needs context and too much context blurs; the shrink-then-enlarge network with skip connections; and turning a mask into countable objects.45 min
  38. Build a vision transformerPatch embeddings, a class token, position embeddings and an unmasked block, trained in the page; then two failures that explain it: no positions means no layout, and few images means losing to a CNN a fraction of its size.45 min
  39. Image-text models in depth: retrieval, negatives and limitsHow to test a shared image-text space with recall at k, what the rest of the batch teaches, what hard and false negatives do, and why a similarity score can't see word order, counts or "no".45 min
  40. Self-supervised learning: labels from the data itselfPretrain an encoder with no labels by matching two views of each image, test it with a linear probe, and see what the augmentations remove, what collapse looks like, and how filling in masked parts differs.45 min
  41. Diffusion 1: turning data into noiseThe easy half of a diffusion model: one step of noise, why the data is shrunk as noise is added, the jump to any step with alpha bar, how to read a noise schedule, and the training examples a denoiser learns from.40 min
  42. Diffusion 2: learning to denoise, and samplingTrain a network to predict noise, then run it a hundred times from pure noise and watch a ring assemble: the sampling loop, why fresh noise goes back in, fewer steps and where they break, and a check that samples are new.45 min
  43. Diffusion 3: images, labels and guidanceFrom points to small images, a label the denoiser can be asked with, and classifier-free guidance: why a model that is told what to draw still often doesn't, how the guidance scale fixes it, and what it breaks when turned up too far.45 min
  44. Diffusion 4: latent diffusion and the whole pipelineWhy image generators diffuse an autoencoder's codes and not pixels: a latent diffusion model in miniature, what the autoencoder must be trained to do, the three parts of a text-to-image system, and where the cost of an image goes.40 min
  45. Sound as numbers: waveforms and spectrogramsA recording as an array of samples, why the sampling rate must be twice the highest pitch, spectrograms and the window trade-off, and the log-mel features a speech model is given.40 min
  46. Recognition and synthesis as a pipelineA whole speech recognizer for a made-up language: features, a guess per frame, collapsing repeats and blanks, a dictionary to repair mistakes, word error rate, and synthesis as the same stages run backward.40 min
  47. The voice agent loop and its latency budgetOne turn of a voice agent, timed: why every stage must stream, how to decide that someone has finished speaking, what to do when they interrupt, and why the slow turns matter more than the average.45 min

Never trained an AI? Teach a small one to write, right in your browser. It takes about 20 seconds and needs no account. Teach it to write