The softmax Jacobian and p − y
Every output depends on every input: the full Jacobian, its cheap vector product, and why cross-entropy's gradient is p − y.
What it covers11
Why it matters
Softmax sits at two places in a GPT: the attention weights in every head and the output distribution over the vocabulary. Its derivative is unlike the elementwise functions: every output depends on every input, so its Jacobian is a full matrix with a simple formula, . Put cross-entropy on top and it collapses to the cleanest gradient in deep learning, : the probabilities minus the one-hot truth.
This lesson derives both, checks them against the numbers and against autograd, and reads the result off the tiny GPT itself: on "Our Kestrel tent is made from", the gradient at the right token is −0.475 and at the leading wrong one +0.463, and the 195 entries sum to zero. You will also see what temperature does to the gradient, and why it is a sampling setting, not a training one.
What you'll be able to do
- Derive the softmax Jacobian, ∂p_i/∂z_j = p_i(δ_ij − p_j), and check it numerically
- Compute the softmax vector-Jacobian product without forming the matrix
- Show that cross-entropy on top of softmax has gradient p − y, and read it off the tiny GPT's logits
- Explain what temperature does to the gradient and why training keeps T = 1
Show the answer
They all fall, because the probabilities must still sum to 1
The shared denominator grows, so every other probability shrinks. That coupling is why softmax's Jacobian is full rather than diagonal.
The idea: every output depends on every input
This rests on Softmax: scores into probabilities, the function itself, and on Jacobians: derivatives with shapes, the table of slopes.
A full Jacobian with a simple formula
Softmax turns logits into probabilities . The denominator is shared, so every depends on every : raise one logit and its probability rises while all the others fall. The Jacobian is a full matrix, not a diagonal one, but its entries have one formula:
where is 1 when and 0 otherwise. On the diagonal that is , the slope of a probability with respect to its own logit; off the diagonal it is , what loses when gains. In matrix form, .
The derivation is the quotient rule. For , the numerator's derivative is and so is the denominator's, giving . For , the numerator does not move and the denominator moves by , giving . Every row sums to zero, because adding the same constant to every logit leaves the probabilities unchanged. The next cell builds from the formula for and compares it with the numerical estimate.
Formula against numerical estimate
diag(p) − ppᵀ for z = [2, 0, 1], the numerical Jacobian beside it, and the row sums.
import torch z = torch.tensor([2.0, 0.0, 1.0]) p = torch.softmax(z, dim=0) print("p:", p) J = torch.eye(3) * p - p.view(3, 1) @ p.view(1, 3) # diag(p) - p p^T print("formula J:", J) def jacobian(f, x, h=1e-3): y = f(x) Jn = torch.zeros(y.shape[0], x.shape[0]) for j in range(x.shape[0]): e = torch.zeros_like(x); e[j] = h Jn[:, j] = (f(x + e) - f(x - e)) / (2 * h) return Jn print("numerical:", jacobian(lambda v: torch.softmax(v, dim=0), z)) print("row sums:", J.sum(1))
p = [0.665, 0.090, 0.245]. The diagonal entry J[0][0] = 0.665 × (1 − 0.665) = 0.2227; the off-diagonal J[0][1] = −0.665 × 0.090 = −0.0599. The matrix is symmetric, because −p_i p_j does not care about order. The numerical estimate agrees to four decimals, and each row sums to zero within float32 rounding (the entries of order 1e-08 are zeros): push all three logits up together and nothing happens to p.
import torch p = torch.softmax(torch.tensor([2.0, 0.0, 1.0]), dim=0) J = torch.eye(3) * p - p.view(3, 1) @ p.view(1, 3) print(round(J.sum(1)[0].item(), 3))
Show the answer
0.0
Row i sums to p_i(1 − p_i) − p_i Σ_{j≠i} p_j = p_i − p_i Σ_j p_j = p_i − p_i = 0. Shifting every logit by the same amount changes nothing.
The vector-Jacobian product
Backprop does not need , only for the gradient arriving from above. Because is symmetric, , and the formula collapses:
the upstream gradient minus its -weighted average, times . Three vector operations on numbers and no matrix. For the attention weights in a head, where is the sequence length, that is the difference between a backward pass that fits in memory and one that does not.
The next cell checks the shortcut three ways: autograd, the formula, and the explicit from the matrix built above.
Three ways to the same gradient
Autograd, the vector formula p ⊙ (g − g·p), and the explicit Jᵀg, for an upstream gradient g = [1, 0, −1].
import torch z = torch.tensor([2.0, 0.0, 1.0], requires_grad=True) p = torch.softmax(z, dim=0) g = torch.tensor([1.0, 0.0, -1.0]) # the gradient arriving at p (p * g).sum().backward() print("autograd: ", z.grad) pd = p.detach() print("p * (g - g.p):", pd * (g - (g * pd).sum())) J = torch.eye(3) * pd - pd.view(3, 1) @ pd.view(1, 3) print("J^T g: ", J.T @ g)
g·p = 0.665 − 0.245 = 0.420, so the centered gradient is [0.580, −0.420, −1.420], and multiplying by p gives [0.3855, −0.0379, −0.3476]. Autograd and the explicit matrix product agree. Notice the middle entry: its upstream gradient was zero, yet its logit gets a negative gradient, because raising logit 1 would steal probability from the other two, and the loss cares about those. Full Jacobian, full coupling.
Cross-entropy collapses it to p − y
The loss on one token is , where is the correct index. Its gradient with respect to is at position and zero elsewhere. Feed that into the softmax Jacobian:
So , with the one-hot vector of the correct answer. The same result comes faster from : the derivative is , and the minus sign of the loss flips it.
Read it as instructions to the logits: raise the correct one by , the probability still missing, and lower every other one by exactly its own probability. The entries sum to zero, so the gradient never changes the overall level of the logits, only their differences. The next two cells check it on the three-logit example and then on the tiny GPT's real logits for "Our Kestrel tent is made from", where the answer is silnylon.
Cross-entropy gives p − y
Loss −log p_2 on z = [2, 0, 1]. Autograd's gradient against p minus the one-hot target, and the sum of the entries.
import torch z = torch.tensor([2.0, 0.0, 1.0], requires_grad=True) y = 2 # the correct index loss = -torch.log_softmax(z, dim=0)[y] loss.backward() onehot = torch.zeros(3); onehot[y] = 1.0 p = torch.softmax(z.detach(), dim=0) print("loss:", round(loss.item(), 4)) print("autograd:", z.grad) print("p - y: ", p - onehot) print("sum of the gradient:", z.grad.sum().item())
The correct logit, index 2, had probability 0.2447, so its gradient is 0.2447 − 1 = −0.7553: a descent step raises it. The other two get +0.6652 and +0.0900, their own probabilities: they fall, the confident wrong one fastest. The loss is −ln 0.2447 = 1.4076. The sum of the three entries is −1.2e-07, zero in float32.
import torch p = torch.softmax(torch.tensor([0.0, 0.0]), dim=0) print((p - torch.tensor([1.0, 0.0])).tolist())
Show the answer
[-0.5, 0.5]
p = [0.5, 0.5]; subtracting the one-hot [1, 0] gives [−0.5, 0.5]: raise the correct logit by the half that is missing, lower the other by its half.
Cross-entropy for the three leading tokens (illustrative logits). The loss is −log of the correct bar; each logit's gradient is its bar's height, minus 1 for the correct one.
Write a sentence or two, then compare it with a good answer in the lesson.
In PyTorch: the gradient on a trained model
Temperature, and why it belongs to sampling
Divide the logits by a temperature before the softmax and the chain rule adds one factor: , where is the softmax at that temperature. A low temperature sharpens and divides by a small number: at the gradient at the correct logit in the three-logit example is −1.7654, more than twice the −0.7553 at . A high temperature flattens and shrinks it: −0.3464 at .
That is why temperature is a sampling knob and not a training one. Training at gives the clean ; changing during training would rescale every gradient, a disguised learning-rate change. At generation time there is no gradient, and temperature reshapes the distribution exactly as the sampling lesson described. The next cell prints the gradient at three temperatures with the formula beside it.
Temperature scales the gradient
Loss −log softmax(z/T)[2] at T = 0.5, 1 and 2: autograd against (p_T − y)/T.
import torch z = torch.tensor([2.0, 0.0, 1.0]) onehot = torch.tensor([0.0, 0.0, 1.0]) for T in [0.5, 1.0, 2.0]: zg = z.clone().requires_grad_(True) loss = -torch.log_softmax(zg / T, dim=0)[2] loss.backward() pT = torch.softmax(z / T, dim=0) print("T", T, " autograd", zg.grad, " (p_T - y) / T", (pT - onehot) / T)
At T = 0.5 the distribution sharpens to p_T = [0.867, 0.016, 0.117] and the division by 0.5 doubles everything: the correct logit's gradient is −1.7654. At T = 2 the distribution flattens and the division halves: −0.3464. Same logits, same answer, gradients that differ by a factor of five between the extremes. Hold T at 1 for training.
Show the answer
−log of the probability the model gave the correct answer
With a one-hot target only one term of −Σ y_i log p_i survives: −log p_y. Its gradient with respect to the logits is p − y.
Show the answer
It changes the logits' differences, never their common level, which softmax ignores anyway
A zero-sum gradient has no component along the all-ones direction, the direction softmax cannot see. The update spends itself entirely on differences between logits.
The softmax vector-Jacobian product
Write softmax_vjp(p, g): p is a 1-D tensor of probabilities (a softmax output) and g the gradient arriving at p. Return the gradient with respect to the logits, p * (g - (g * p).sum()), without building the Jacobian.
import torch def softmax_vjp(p, g): return g p = torch.softmax(torch.tensor([2.0, 0.0, 1.0]), dim=0) print(softmax_vjp(p, torch.tensor([1.0, 0.0, -1.0])))
p − y as a function
Write cross_entropy_backward(z, y): z is a 1-D tensor of logits and y the correct index. Return the gradient of −log softmax(z)[y] with respect to z, which is softmax(z) minus the one-hot vector of y.
import torch def cross_entropy_backward(z, y): return z print(cross_entropy_backward(torch.tensor([2.0, 0.0, 1.0]), 2))
The softmax Jacobian as a picture
diag(p) minus p p-transpose for six logits, as a heat map. The diagonal is positive, the rest negative, and each row adds to zero.
import torch import matplotlib.pyplot as plt z = torch.tensor([2.0, 0.0, 1.0, -1.0, 0.5, 1.5]) p = torch.softmax(z, dim=0) J = torch.diag(p) - p.view(-1, 1) @ p.view(1, -1) fig, ax = plt.subplots(figsize=(4.5, 3.8)) im = ax.imshow(J.tolist(), cmap="RdBu", vmin=-0.25, vmax=0.25) ax.set_xlabel("input logit") ax.set_ylabel("output probability") ax.set_title("Softmax Jacobian for six logits") fig.colorbar(im, ax=ax) print("largest diagonal entry:", round(float(J.diag().max()), 3)) print("row sums:", [round(float(r.sum()), 3) for r in J])
The strongest diagonal entry is 0.244, for the token with the largest probability, 0.42: 0.42 x 0.58. Every row sums to 0 (the printed -0.0 is a rounding of a tiny negative), so the picture is balanced: red on the diagonal, blue off it.
On the tiny GPT's logits
The last position's 195 logits for "Our Kestrel tent is made from", the loss against silnylon, and the gradient autograd leaves at the logits.
import torch from tinygpt import load_pretrained model, tok = load_pretrained("tiny-gpt") ids = torch.tensor([tok.encode("Our Kestrel tent is made from")]) target = tok.token_id("silnylon") logits = model(ids)[0, -1] # the last position's 195 logits logits.retain_grad() # keep the gradient at this non-leaf tensor loss = -torch.log_softmax(logits, dim=-1)[target] loss.backward() p = torch.softmax(logits.detach(), dim=-1) print("loss", round(loss.item(), 3), " p(silnylon)", round(p[target].item(), 3)) top = torch.topk(p, 3) for prob, i in zip(top.values, top.indices): print(f"{tok.vocab[int(i)]:10s} p {prob.item():.3f} " f" gradient {logits.grad[int(i)].item():+.3f}") onehot = torch.zeros(len(tok.vocab)); onehot[target] = 1.0 print("max |gradient - (p - y)|:", (logits.grad - (p - onehot)).abs().max().item()) print("sum over the vocabulary:", round(logits.grad.sum().item(), 6))
The model gives silnylon 0.525 and recycled 0.463: nearly a coin flip between two materials, with the right one ahead. The gradient at silnylon is 0.525 − 1 = −0.475, at recycled +0.463, at everything else its own small probability (waxed: 0.004). One training step on this example pushes the two apart. Autograd's 195 numbers match p − y to 6e-08, and they sum to zero. This vector, computed at every position of every training sequence, is the entire signal that trains a language model.
Show the answer
silnylon rises, recycled falls by almost as much, waxed barely moves
Descent moves each logit against its gradient: −0.475 means silnylon rises; +0.463 means recycled falls; waxed's +0.004 is a nudge.
Write a sentence or two, then compare it with a good answer in the lesson.
In real models: why the loss takes logits
In a training loop the softmax and the loss are one call, and the gradient at the logits is p minus y at every position.
Where this shows up in an engineer's week
On the job: apply the softmax and then the log yourself and you will hit a NaN the first time a probability underflows to 0. Pass logits to the fused loss. And when a loss will not go down, look at p minus y on a batch: if the correct token's entry is already near 0, the model is confident and right, and the problem is elsewhere.
Goes deeper in
- Language modeling: the loss this gradient comes from, and what a loss of 2.1 means.
- Backprop through attention: the softmax Jacobian again, applied to each row of attention.
- Why temperature changes the answer: the setting that divides the logits, and why it is for sampling.
Write a sentence or two, then compare it with a good answer in the lesson.
You can now
- Derive the softmax Jacobian, p_i(δ_ij − p_j), and check it numerically.
- Compute the softmax vector-Jacobian product, p ⊙ (g − g·p), without the matrix.
- Show that cross-entropy on softmax has gradient p − y, and read it off the tiny GPT's logits.
- Explain what temperature does to the gradient and why training keeps T = 1.
Next: backprop through attention, where every row of the attention matrix is a softmax.
Practice what you just read
- 6 code examples you can edit and run
- a practice exercise that checks your code
- a 5-question quiz
- 10 review cards for spaced repetition
Try a quiz question
The entry ∂p_i/∂z_j of the softmax Jacobian for i ≠ j is:
Before this lesson
The math behind this
Go deeper, if you like
- Backprop through attention Optional
Where this lesson leads.
575 lessons in thirteen phases. 176 are free without an account, and a free account opens the rest. See every lesson.
Phase 5 · about 36 hours of study
Transformers
Attention, GPT from scratch and pretraining
47 lessons 1 free
- Language modeling FreePredict the next token, and score it with cross-entropy.30 min
- Tokenization and BPEHow tokenizers are built from data.40 min
- A bigram modelThe simplest language model, built from counts.35 min
- Sampling strategiesTemperature, top-k and top-p: turning probabilities into text.35 min
- Beam search and constrained decodingWhy the likeliest word isn't the likeliest sentence, beam search with log probabilities, its taste for short and repetitive text, and forcing output into a grammar: what that guarantees and what it doesn't.40 min
- Neural language modelsWhy counting fails, and what embeddings fix.35 min
- Attention: the intuitionEach token decides which other tokens matter to it.30 min
- Queries, keys and valuesThe three projections behind attention.35 min
- Scaled dot-product attentionThe attention formula, line by line.35 min
- Causal maskingStop tokens from seeing the future.25 min
- Multi-head attentionSeveral attention patterns in parallel.40 min
- Positional encodingsTell the model where each token sits.30 min
- The transformer blockAttention, MLP, residuals and norms together.40 min
- The GPT architectureA stack of decoder blocks that predicts the next token.40 min
- Build a tiny GPTWrite and train a small GPT from scratch.45 min
- From your GPT to GPT-2 in Hugging FaceMatch your GPT's tensors to GPT-2's names and shapes, count its 124,439,808 parameters, load renamed weights into your own class, pad a batch and read a config.40 min
- Training a GPT wellWarmup and cosine decay, gradient clipping, a validation loss you can trust, saving and loading, and two training mistakes read off the loss curve.35 min
- Encoders and embedding modelsBERT-style models and where they fit.40 min
- Tokenization: text to integer IDsFollow "The cat sat" into the three integers a model receives.30 min
- Embeddings and positional wavesFrom three IDs to the 3 × 8 matrix that enters the first block.35 min
- Inside the attention matrixEvery entry of the 3 × 3 attention matrix, computed and steered.40 min
- The feed-forward layerExpand to four times d_model and back: shapes, parameters and what the layer stores.35 min
- The LM head and samplingFrom the last row to a probability for every token, and the next word.35 min
- Pretraining at scaleData, compute and what models learn from the internet.40 min
- Scaling lawsHow loss falls predictably with size, data and compute.40 min
- Inference and the KV cacheWhy generation is fast after the first token.40 min
- The block in today's open models OptionalRMSNorm, SwiGLU, rotary positions and grouped-query attention swapped into the block you built, with the parameter and cache counts before and after.35 min
- Long contextEfficient attention and million-token windows.35 min
- Mixture of expertsRoute each token to a few specialist sub-networks.35 min
- Capstone: diagnose your own GPTEntropy per position, a head's attention pattern and a head ablation on the tiny GPT, then the same three measurements on a GPT you train yourself.45 min
- Jacobians: derivatives with shapes OptionalThe derivative of a vector function is a matrix; the chain rule multiplies them, and backprop multiplies by them without building them.35 min
- Backprop through a linear layer OptionaldX = dY Wᵀ, dW = Xᵀ dY, db = column sums: derived by shape and by entry, checked on the tiny GPT's own projection.35 min
- The softmax Jacobian and p − y OptionalEvery output depends on every input: the full Jacobian, its cheap vector product, and why cross-entropy's gradient is p − y.35 min
- Backprop through attention OptionalFive formulas from the output back to queries, keys and values; why masked scores get zero gradient; what FlashAttention recomputes.40 min
- Multimodal transformersHow models see images and hear audio.35 min
- Object detection: boxes, overlap and suppressionEvery object and where it is: boxes in two formats, intersection over union, why one object gets many boxes, non-maximum suppression and its threshold, and grading with average precision.45 min
- Segmentation: a label for every pixelMasks, and how to grade them with IoU and Dice; why a pixel's label needs context and too much context blurs; the shrink-then-enlarge network with skip connections; and turning a mask into countable objects.45 min
- Build a vision transformerPatch embeddings, a class token, position embeddings and an unmasked block, trained in the page; then two failures that explain it: no positions means no layout, and few images means losing to a CNN a fraction of its size.45 min
- Image-text models in depth: retrieval, negatives and limitsHow to test a shared image-text space with recall at k, what the rest of the batch teaches, what hard and false negatives do, and why a similarity score can't see word order, counts or "no".45 min
- Self-supervised learning: labels from the data itselfPretrain an encoder with no labels by matching two views of each image, test it with a linear probe, and see what the augmentations remove, what collapse looks like, and how filling in masked parts differs.45 min
- Diffusion 1: turning data into noiseThe easy half of a diffusion model: one step of noise, why the data is shrunk as noise is added, the jump to any step with alpha bar, how to read a noise schedule, and the training examples a denoiser learns from.40 min
- Diffusion 2: learning to denoise, and samplingTrain a network to predict noise, then run it a hundred times from pure noise and watch a ring assemble: the sampling loop, why fresh noise goes back in, fewer steps and where they break, and a check that samples are new.45 min
- Diffusion 3: images, labels and guidanceFrom points to small images, a label the denoiser can be asked with, and classifier-free guidance: why a model that is told what to draw still often doesn't, how the guidance scale fixes it, and what it breaks when turned up too far.45 min
- Diffusion 4: latent diffusion and the whole pipelineWhy image generators diffuse an autoencoder's codes and not pixels: a latent diffusion model in miniature, what the autoencoder must be trained to do, the three parts of a text-to-image system, and where the cost of an image goes.40 min
- Sound as numbers: waveforms and spectrogramsA recording as an array of samples, why the sampling rate must be twice the highest pitch, spectrograms and the window trade-off, and the log-mel features a speech model is given.40 min
- Recognition and synthesis as a pipelineA whole speech recognizer for a made-up language: features, a guess per frame, collapsing repeats and blanks, a dictionary to repair mistakes, word error rate, and synthesis as the same stages run backward.40 min
- The voice agent loop and its latency budgetOne turn of a voice agent, timed: why every stage must stream, how to decide that someone has finished speaking, what to do when they interrupt, and why the slow turns matter more than the average.45 min
Never trained an AI? Teach a small one to write, right in your browser. It takes about 20 seconds and needs no account. Teach it to write