Backprop through attention
Five formulas from the output back to queries, keys and values; why masked scores get zero gradient; what FlashAttention recomputes.
What it covers10
Why it matters
Attention is a chain of pieces you can now differentiate: a matrix product , a mask, a softmax on every row, and another matrix product with . Its backward pass is therefore a chain of known rules: the linear-layer formulas for the two products, the softmax vector-Jacobian product for each row, and nothing at all for the masked positions.
This lesson writes the five formulas, , , , and , checks each against autograd on three tokens and on head 0 of the tiny GPT's first block, then carries the gradient into , and . You will see why masked scores get exactly zero gradient, why each row of the score gradient sums to zero, and why FlashAttention recomputes the attention matrix instead of storing it: at 8,192 tokens, 32 heads and 32 layers it is 274.88 GB in float32.
What you'll be able to do
- Write the five gradients of one attention head, dV, dA, dS, dQ and dK, from the rules for linear layers and softmax
- Explain why masked scores receive exactly zero gradient and why every row of dS sums to zero
- Carry the gradient into W_Q, W_K and W_V and check it against autograd, including on the tiny GPT's own head
- Say what FlashAttention recomputes in the backward pass and how much memory that saves
Show the answer
Aᵀ dO, shape T × d: each value collects the output gradients of the tokens that attended to it
O = AV is a linear layer with A as the input and V as the weight, so dV = Aᵀ dO by the rule of Backprop through a linear layer: (T × T)(T × d) = T × d.
The idea: the forward pass, run backward
This rests on Attention as matrix math, the forward pass by hand, and on The softmax Jacobian and p − y, the one step that needs care.
The forward pass as four steps
One attention head, for tokens with head width : queries , keys and values , each .
- Scores: , a matrix, a matrix product.
- Mask: entries above the diagonal become , so no token reads a later one.
- Weights: , row by row.
- Output: , , another matrix product.
Every step is one you can now differentiate: the linear-layer rules for steps 1 and 4, the softmax vector-Jacobian product for step 3, and for step 2 nothing at all, as you will see. The next cell runs the forward pass on three tokens with hand-picked , and , chosen so the attention matrix has readable numbers.
Forward on three tokens
Hand-picked Q, K (the same matrix) and V with head width 4. The masked scores, the attention weights and the output.
import math import torch Q = torch.tensor([[1.0, 0.0, 1.0, 0.0], [0.0, 1.0, 0.0, 1.0], [1.0, 1.0, 0.0, 0.0]]) K = torch.tensor([[1.0, 0.0, 1.0, 0.0], [0.0, 1.0, 0.0, 1.0], [1.0, 1.0, 0.0, 0.0]]) V = torch.tensor([[1.0, 0.0], [0.0, 1.0], [0.5, 0.5]]) d = Q.shape[1] mask = torch.tril(torch.ones(3, 3)).bool() S = (Q @ K.T / math.sqrt(d)).masked_fill(~mask, float("-inf")) A = torch.softmax(S, dim=-1) O = A @ V print("S:", S) print("A:", A) print("O:", O)
Token 0 can read only itself, so its row of A is one-hot and its output is its own value, [1, 0]. Token 1 scores 0 against token 0 and 1 against itself: the softmax of [0, 1] is [0.2689, 0.7311]. Token 2 scores 0.5, 0.5 and 1: weights [0.2741, 0.2741, 0.4519], and its output is that blend of the three values, which comes to [0.5, 0.5]. The −inf entries of S became exactly 0 in A.
import torch A = torch.tensor([[1.0, 0.0], [0.5, 0.5]]) dO = torch.tensor([[1.0, 2.0], [3.0, 4.0]]) print((A.T @ dO).tolist())
Show the answer
[[2.5, 4.0], [1.5, 2.0]]
Value 0 was read by token 0 with weight 1 and by token 1 with weight 0.5: 1·[1, 2] + 0.5·[3, 4] = [2.5, 4]. Value 1 was read only by token 1 with weight 0.5: [1.5, 2].
The five formulas
Backprop hands you , the gradient at the output, . Walk the four steps in reverse.
Step 4, , is a linear layer with in the role of the input and in the role of the weight: and .
Step 3, the row softmax: each row of is the softmax vector-Jacobian product of that row, , the formula of The softmax Jacobian and p − y applied to every row at once.
Step 2, the mask: where is 0, is 0, because the product above multiplies by . Nothing flows to the masked scores, and nothing needs to.
Step 1, : for it is a linear layer with as the weight, so ; for the roles swap, .
Five formulas, each with a shape you can check: , and are ; and are . The next cell computes all five by hand and prints autograd's answer under each.
Backward by hand, autograd underneath
The same head with a chosen dO. Each hand formula, then the gradient autograd stored for the same tensor.
import math import torch Q = torch.tensor([[1.0, 0.0, 1.0, 0.0], [0.0, 1.0, 0.0, 1.0], [1.0, 1.0, 0.0, 0.0]], requires_grad=True) K = torch.tensor([[1.0, 0.0, 1.0, 0.0], [0.0, 1.0, 0.0, 1.0], [1.0, 1.0, 0.0, 0.0]], requires_grad=True) V = torch.tensor([[1.0, 0.0], [0.0, 1.0], [0.5, 0.5]], requires_grad=True) d = Q.shape[1] mask = torch.tril(torch.ones(3, 3)).bool() S = (Q @ K.T / math.sqrt(d)).masked_fill(~mask, float("-inf")) A = torch.softmax(S, dim=-1) O = A @ V S.retain_grad(); A.retain_grad() # the gradient arriving at the output dO = torch.tensor([[1.0, 0.0], [0.0, 1.0], [1.0, 1.0]]) (O * dO).sum().backward() Ad, Kd, Qd, Vd = A.detach(), K.detach(), Q.detach(), V.detach() dV = Ad.T @ dO dA = dO @ Vd.T dS = Ad * (dA - (dA * Ad).sum(-1, keepdim=True)) dQ = dS @ Kd / math.sqrt(d) dK = dS.T @ Qd / math.sqrt(d) for name, hand, auto in [("dV", dV, V.grad), ("dA", dA, A.grad), ("dS", dS, S.grad), ("dQ", dQ, Q.grad), ("dK", dK, K.grad)]: print(f"{name} hand:", hand) print(f"{name} auto:", auto)
All five pairs agree. dV collects each value's share of the output gradient: value 2 was read only by token 2, with weight 0.4519, so it gets 0.4519 × [1, 1]. dA is dO Vᵀ. Then two rows of dS vanish. Row 0, because token 0's attention is one-hot, and a one-hot softmax has zero gradient: p ⊙ (g − g·p) is 0 when p picks out g's own entry. Row 2, because its upstream row of dA is uniform, [1, 1, 1], and a constant gradient across a softmax row cancels out. Only token 1's row carries gradient, ±0.1966, so only token 1's query moves (dQ row 1) and only keys 0 and 1 receive gradient (dK rows 0 and 1). The masked entries above the diagonal are exactly zero.
Show the answer
Token 0 attends only to itself, and a one-hot softmax row has zero gradient, whatever arrives
p ⊙ (g − g·p) with p = [1, 0, 0] gives [g_0 − g_0, 0, 0] = 0. A softmax that has already saturated cannot be moved by its scores.
The mask on a row of attention (illustrative scores). Remove it and put it back: a masked cell holds weight 0, and in the backward pass gradient 0, whatever the score behind it.
Write a sentence or two, then compare it with a good answer in the lesson.
In PyTorch: checked against autograd
Masks and zero-sum rows
Two properties of deserve their own look, because production code depends on them.
Masked scores get exactly zero gradient. The mask sets a score to , the softmax turns it into a weight of exactly 0, and the vector-Jacobian product multiplies by that weight. So no gradient ever reaches a future position's score, which is what makes teacher forcing sound: the whole sequence is trained in one pass, every position predicting the next token, and no position's loss leaks into the queries and keys that would let it peek ahead.
Every row of sums to zero. Softmax ignores a constant added to a row of scores, so the loss cannot depend on a row's overall level, and the gradient has no component in that direction. In the printout above, row 1 is .
The next cell checks both on the tiny GPT itself: head 0 of the first block, on "Our Kestrel tent is made from", with the model's own queries, keys and values. The upstream gradient is random, which is all the formulas need.
Head 0 of the tiny GPT's first block
The model's own q, k and v for head 0 on a six-token prompt, a random dO, and the hand formulas against autograd.
import math import torch from tinygpt import load_pretrained model, tok = load_pretrained("tiny-gpt") ids = torch.tensor([tok.encode("Our Kestrel tent is made from")]) T, blk, dh = ids.shape[1], model.blocks[0], 32 // model.blocks[0].h with torch.no_grad(): h = model.tok(ids) + model.pos(torch.arange(T)) q_all, k_all, v_all = blk.qkv(blk.ln1(h)).split(32, dim=-1) # head 0: the first 8 of 32 columns q = q_all[0, :, :dh].detach().clone().requires_grad_(True) k = k_all[0, :, :dh].detach().clone().requires_grad_(True) v = v_all[0, :, :dh].detach().clone().requires_grad_(True) mask = torch.tril(torch.ones(T, T)).bool() A = torch.softmax((q @ k.T / math.sqrt(dh)).masked_fill(~mask, float("-inf")), dim=-1) torch.manual_seed(0) dO = torch.randn(T, dh) ((A @ v) * dO).sum().backward() Ad = A.detach() dA = dO @ v.detach().T dS = Ad * (dA - (dA * Ad).sum(-1, keepdim=True)) dQ, dK, dV = (dS @ k.detach() / math.sqrt(dh), dS.T @ q.detach() / math.sqrt(dh), Ad.T @ dO) print("tokens", T, " head width", dh) print("attention from 'from':", [f"{tok.vocab[int(i)]} {Ad[-1, j].item():.2f}" for j, i in enumerate(ids[0])]) print("max |hand - autograd|: dQ", (dQ - q.grad).abs().max().item(), " dK", (dK - k.grad).abs().max().item(), " dV", (dV - v.grad).abs().max().item()) print("largest |dS| above the diagonal:", dS.masked_fill(mask, 0.0).abs().max().item()) print("row sums of dS:", [round(x, 6) for x in dS.sum(-1).tolist()]) print("|dQ| per token:", [round(x, 3) for x in (dQ ** 2).sum(-1).sqrt().tolist()]) print("|dK| per token:", [round(x, 3) for x in (dK ** 2).sum(-1).sqrt().tolist()]) import matplotlib.pyplot as plt fig, ax = plt.subplots(figsize=(4.5, 3.8)) im = ax.imshow(dS.tolist(), cmap="RdBu", vmin=-0.3, vmax=0.3) ax.set_xticks(range(T), [tok.vocab[int(i)] for i in ids[0]], rotation=60) ax.set_yticks(range(T), [tok.vocab[int(i)] for i in ids[0]]) ax.set_title("dS: zero above the diagonal, rows sum to zero") fig.colorbar(im, ax=ax)
The formulas hold on real numbers to 3e-08. "from" spreads its attention over the six tokens, most on "tent" (0.30) and "is" (0.23). Nothing above the diagonal in dS, and every row sums to zero. Two numbers show the structure the mask imposes: token 0's query gets no gradient at all (|dQ| = 0.0: its attention row is one-hot), and the last key gets almost none (|dK| = 0.021), because only one query, the last, is allowed to read it. W_Q and W_K are shared by every position, so they still train: in a causal model, the first position's query and the last position's key are where the least gradient flows back into them.
import torch A = torch.tensor([[1.0, 0.0]]) dA = torch.tensor([[2.0, 7.0]]) print((A * (dA - (dA * A).sum(-1, keepdim=True))).tolist())
Show the answer
[[0.0, 0.0]]
The weighted mean of dA under A = [1, 0] is 2, so dA − 2 = [0, 5], and multiplying by A = [1, 0] leaves [0, 0]. Saturated attention passes no gradient to its scores.
Show the answer
The last token's key: only the last query may read it, so one row of dS reaches it
dK = dSᵀ Q/√d sums each key's column of dS over the queries allowed to read it. The last key has one reader; in the tiny GPT example its gradient size was 0.021 against 0.554 for the first key.
Into the weights, and what FlashAttention saves
, and are not parameters; they come from the block's input through three linear layers, and so on, in the tiny GPT one 32 → 96 layer cut in three. The linear-layer rules finish the job: , , , and the input collects all three, , because was used three times. The next cell checks the three weight gradients against autograd on an 8-wide input.
Then the cost. The backward pass needs twice, for and for . Storing it from the forward pass costs heads × numbers per layer: for the tiny GPT at 48 tokens, 18,432 floats, nothing. For a model with 32 heads and 32 layers at 2,048 tokens it is 4.29 billion floats, 17.18 GB in float32; at 8,192 tokens, 274.88 GB, more than any single GPU holds. FlashAttention keeps only each softmax row's maximum and sum from the forward pass and recomputes tile by tile during the backward pass: extra arithmetic, which GPUs have, instead of memory, which they do not. The last cell does that arithmetic.
Into W_Q, W_K and W_V
Three tokens of width 8, three projection matrices, one causal head. Xᵀ times each of dQ, dK, dV against the weights' autograd gradients.
import math import torch torch.manual_seed(0) X = torch.randn(3, 8) # three tokens, 8 numbers each WQ, WK, WV = (torch.randn(8, w, requires_grad=True) for w in (4, 4, 2)) Q, K, V = X @ WQ, X @ WK, X @ WV for t in (Q, K, V): t.retain_grad() mask = torch.tril(torch.ones(3, 3)).bool() A = torch.softmax((Q @ K.T / math.sqrt(4)).masked_fill(~mask, float("-inf")), dim=-1) dO = torch.tensor([[1.0, 0.0], [0.0, 1.0], [1.0, 1.0]]) ((A @ V) * dO).sum().backward() for name, W, dP in [("W_Q", WQ, Q.grad), ("W_K", WK, K.grad), ("W_V", WV, V.grad)]: print(f"{name}: shape {tuple(W.grad.shape)}, max |X^T dP - autograd| " f"= {(X.T @ dP - W.grad).abs().max().item():.1e}")
Three zeros. The attention backward delivered dQ, dK and dV at the projections' outputs; the linear-layer rule carries each to its weight with the same Xᵀ on the left. In the tiny GPT the three weights are one 96 × 32 matrix, so the three gradients sit side by side in one weight.grad, and the output projection and the feed-forward layer follow with the same rule. Every parameter of an attention block is reached by the formulas of these four lessons.
From the score gradient to queries and keys
Write scores_backward(dS, Q, K) for S = QKᵀ/√d, where d is the number of columns of Q. Return the tuple (dQ, dK): dQ = dS K/√d and dK = dSᵀ Q/√d.
import math import torch def scores_backward(dS, Q, K): return Q, K Q = torch.tensor([[1.0, 0.0, 1.0, 0.0], [0.0, 1.0, 0.0, 1.0], [1.0, 1.0, 0.0, 0.0]]) dS = torch.tensor([[0.0, 0.0, 0.0], [-0.1966, 0.1966, 0.0], [0.0, 0.0, 0.0]]) dQ, dK = scores_backward(dS, Q, Q.clone()) print(dQ)
The softmax backward for every row
Write softmax_rows_backward(A, dA): A holds softmax outputs row by row (each row sums to 1) and dA the gradient arriving at A. Return dS, the gradient with respect to the scores: for each row, A * (dA - (dA * A).sum()), done for all rows at once with sum(-1, keepdim=True).
import torch def softmax_rows_backward(A, dA): return dA A = torch.tensor([[1.0, 0.0], [0.5, 0.5]]) dA = torch.tensor([[2.0, 7.0], [1.0, 3.0]]) print(softmax_rows_backward(A, dA))
Write a sentence or two, then compare it with a good answer in the lesson.
In real models: what the backward pass costs
Everything above is cheap in arithmetic and expensive in memory. The map shows the node whose table is stored.
The attention matrices' memory
heads × T² × layers numbers, for the tiny GPT and for two production-sized configurations.
for T, heads, layers in [(48, 4, 2), (2048, 32, 32), (8192, 32, 32)]: floats = heads * T * T * layers print(f"T={T} heads={heads} layers={layers}: {floats:,} floats, " f"{floats * 4 / 1e9:.2f} GB in float32, " f"{floats * 2 / 1e9:.2f} GB in bfloat16")
The attention matrices grow with the square of the sequence length and linearly with heads and layers. The tiny GPT's 18,432 floats are a rounding error. A production configuration at 2,048 tokens already needs 17.18 GB in float32 (8.59 GB in bfloat16) just to remember the attention weights for the backward pass; at 8,192 tokens, 274.88 GB. Recomputing them from the stored row statistics costs a second pass over the QKᵀ products and is what makes long-context training possible.
Show the answer
It would read the token it is supposed to predict, and learn nothing useful
With the answer in view, the easiest solution is to copy it, which does not work at generation time when the answer does not exist yet. The mask, and the zero gradient behind it, keep training honest.
Show the answer
Extra arithmetic to recompute A tile by tile, for not storing heads × T² numbers per layer
It keeps each row's maximum and sum, recomputes the weights when needed, and gets the exact same gradients with far less memory.
Where this shows up in an engineer's week
On the job: when a long-context run runs out of memory, the cause is the stored attention weights, which grow with the square of the sequence length. The fixes are in this lesson: a fused attention kernel that recomputes them, or checkpointing whole blocks. Both trade compute for memory, and both leave the gradients unchanged, so you can compare a run with and without them and expect the same loss.
Goes deeper in
- Inference and the KV cache: the inference-time cousin, which stores keys and values to avoid recomputing them.
- Long context: why the T squared table is the limit on context length.
- Backprop through a linear layer: the linear-layer rules this lesson reuses at the end.
Write a sentence or two, then compare it with a good answer in the lesson.
You can now
- Write dV = Aᵀ dO, dA = dO Vᵀ, dS = A ⊙ (dA − rowsum(dA ⊙ A)), dQ = dS K/√d and dK = dSᵀ Q/√d, and check them.
- Explain why masked scores get exactly zero gradient and why every row of dS sums to zero.
- Carry the gradient into W_Q, W_K and W_V with Xᵀ dQ and its siblings, and into X with the three terms summed.
- Say what FlashAttention recomputes in the backward pass and how much memory it saves.
This closes the "Math under the GPT" track: every parameter of an attention block is now reached by rules you have derived and checked.
Practice what you just read
- 5 code examples you can edit and run
- a practice exercise that checks your code
- a 5-question quiz
- 10 review cards for spaced repetition
Try a quiz question
For O = AV, the gradient with respect to the values is:
Before this lesson
Where this lesson leads.
575 lessons in thirteen phases. 176 are free without an account, and a free account opens the rest. See every lesson.
Phase 5 · about 36 hours of study
Transformers
Attention, GPT from scratch and pretraining
47 lessons 1 free
- Language modeling FreePredict the next token, and score it with cross-entropy.30 min
- Tokenization and BPEHow tokenizers are built from data.40 min
- A bigram modelThe simplest language model, built from counts.35 min
- Sampling strategiesTemperature, top-k and top-p: turning probabilities into text.35 min
- Beam search and constrained decodingWhy the likeliest word isn't the likeliest sentence, beam search with log probabilities, its taste for short and repetitive text, and forcing output into a grammar: what that guarantees and what it doesn't.40 min
- Neural language modelsWhy counting fails, and what embeddings fix.35 min
- Attention: the intuitionEach token decides which other tokens matter to it.30 min
- Queries, keys and valuesThe three projections behind attention.35 min
- Scaled dot-product attentionThe attention formula, line by line.35 min
- Causal maskingStop tokens from seeing the future.25 min
- Multi-head attentionSeveral attention patterns in parallel.40 min
- Positional encodingsTell the model where each token sits.30 min
- The transformer blockAttention, MLP, residuals and norms together.40 min
- The GPT architectureA stack of decoder blocks that predicts the next token.40 min
- Build a tiny GPTWrite and train a small GPT from scratch.45 min
- From your GPT to GPT-2 in Hugging FaceMatch your GPT's tensors to GPT-2's names and shapes, count its 124,439,808 parameters, load renamed weights into your own class, pad a batch and read a config.40 min
- Training a GPT wellWarmup and cosine decay, gradient clipping, a validation loss you can trust, saving and loading, and two training mistakes read off the loss curve.35 min
- Encoders and embedding modelsBERT-style models and where they fit.40 min
- Tokenization: text to integer IDsFollow "The cat sat" into the three integers a model receives.30 min
- Embeddings and positional wavesFrom three IDs to the 3 × 8 matrix that enters the first block.35 min
- Inside the attention matrixEvery entry of the 3 × 3 attention matrix, computed and steered.40 min
- The feed-forward layerExpand to four times d_model and back: shapes, parameters and what the layer stores.35 min
- The LM head and samplingFrom the last row to a probability for every token, and the next word.35 min
- Pretraining at scaleData, compute and what models learn from the internet.40 min
- Scaling lawsHow loss falls predictably with size, data and compute.40 min
- Inference and the KV cacheWhy generation is fast after the first token.40 min
- The block in today's open models OptionalRMSNorm, SwiGLU, rotary positions and grouped-query attention swapped into the block you built, with the parameter and cache counts before and after.35 min
- Long contextEfficient attention and million-token windows.35 min
- Mixture of expertsRoute each token to a few specialist sub-networks.35 min
- Capstone: diagnose your own GPTEntropy per position, a head's attention pattern and a head ablation on the tiny GPT, then the same three measurements on a GPT you train yourself.45 min
- Jacobians: derivatives with shapes OptionalThe derivative of a vector function is a matrix; the chain rule multiplies them, and backprop multiplies by them without building them.35 min
- Backprop through a linear layer OptionaldX = dY Wᵀ, dW = Xᵀ dY, db = column sums: derived by shape and by entry, checked on the tiny GPT's own projection.35 min
- The softmax Jacobian and p − y OptionalEvery output depends on every input: the full Jacobian, its cheap vector product, and why cross-entropy's gradient is p − y.35 min
- Backprop through attention OptionalFive formulas from the output back to queries, keys and values; why masked scores get zero gradient; what FlashAttention recomputes.40 min
- Multimodal transformersHow models see images and hear audio.35 min
- Object detection: boxes, overlap and suppressionEvery object and where it is: boxes in two formats, intersection over union, why one object gets many boxes, non-maximum suppression and its threshold, and grading with average precision.45 min
- Segmentation: a label for every pixelMasks, and how to grade them with IoU and Dice; why a pixel's label needs context and too much context blurs; the shrink-then-enlarge network with skip connections; and turning a mask into countable objects.45 min
- Build a vision transformerPatch embeddings, a class token, position embeddings and an unmasked block, trained in the page; then two failures that explain it: no positions means no layout, and few images means losing to a CNN a fraction of its size.45 min
- Image-text models in depth: retrieval, negatives and limitsHow to test a shared image-text space with recall at k, what the rest of the batch teaches, what hard and false negatives do, and why a similarity score can't see word order, counts or "no".45 min
- Self-supervised learning: labels from the data itselfPretrain an encoder with no labels by matching two views of each image, test it with a linear probe, and see what the augmentations remove, what collapse looks like, and how filling in masked parts differs.45 min
- Diffusion 1: turning data into noiseThe easy half of a diffusion model: one step of noise, why the data is shrunk as noise is added, the jump to any step with alpha bar, how to read a noise schedule, and the training examples a denoiser learns from.40 min
- Diffusion 2: learning to denoise, and samplingTrain a network to predict noise, then run it a hundred times from pure noise and watch a ring assemble: the sampling loop, why fresh noise goes back in, fewer steps and where they break, and a check that samples are new.45 min
- Diffusion 3: images, labels and guidanceFrom points to small images, a label the denoiser can be asked with, and classifier-free guidance: why a model that is told what to draw still often doesn't, how the guidance scale fixes it, and what it breaks when turned up too far.45 min
- Diffusion 4: latent diffusion and the whole pipelineWhy image generators diffuse an autoencoder's codes and not pixels: a latent diffusion model in miniature, what the autoencoder must be trained to do, the three parts of a text-to-image system, and where the cost of an image goes.40 min
- Sound as numbers: waveforms and spectrogramsA recording as an array of samples, why the sampling rate must be twice the highest pitch, spectrograms and the window trade-off, and the log-mel features a speech model is given.40 min
- Recognition and synthesis as a pipelineA whole speech recognizer for a made-up language: features, a guess per frame, collapsing repeats and blanks, a dictionary to repair mistakes, word error rate, and synthesis as the same stages run backward.40 min
- The voice agent loop and its latency budgetOne turn of a voice agent, timed: why every stage must stream, how to decide that someone has finished speaking, what to do when they interrupt, and why the slow turns matter more than the average.45 min
Never trained an AI? Teach a small one to write, right in your browser. It takes about 20 seconds and needs no account. Teach it to write