Jacobians: derivatives with shapes
The derivative of a vector function is a matrix; the chain rule multiplies them, and backprop multiplies by them without building them.
What it covers11
Why it matters
Phase 2 taught the chain rule for numbers: multiply the slopes. A GPT is made of vectors and matrices, and when a function takes a vector in and gives a vector out, its derivative is a table of slopes, the Jacobian. The chain rule still holds, but "multiply" now means matrix multiplication and the shapes have to fit.
Every gradient formula in the other "Math under the GPT" lessons is a Jacobian argument with the shapes worked out: the linear layer's in Backprop through a linear layer, the softmax's in The softmax Jacobian and p − y, and Backprop through attention. This lesson gives you the tool: what a Jacobian is, how to measure one, why backprop multiplies by it without ever building it, and why that makes a derivative with fourteen million entries cost a few hundred thousand operations.
What you'll be able to do
- Write the Jacobian of a vector function, with its shape, and estimate it numerically
- Apply the chain rule to vector functions as a product of Jacobians in the right order
- Explain why backprop computes vector-Jacobian products and never builds a Jacobian
- Recognize diagonal Jacobians in elementwise functions and the full one in softmax
Show the answer
Six, as a 2 × 3 matrix: one row per output, one column per input
Every output can move when any input moves: 2 outputs × 3 inputs = 6 slopes. Arranged by output row and input column, they are the Jacobian.
The idea: a table of slopes
This rests on Jacobians and Hessians, where the table of slopes was built by hand, and on The chain rule with vectors, the rule that multiplies them.
From one slope to a table of slopes
A function from a number to a number has one derivative. A function from a vector of numbers to a vector of numbers has of them: how much each output moves when each input moves. Arranged with one row per output and one column per input, they form the Jacobian , with .
Two cases you already know are Jacobians in disguise. A scalar function of a vector, like a loss, has a Jacobian with one row: the gradient from the gradients lesson, written across. A linear map has Jacobian itself: output is , so its slope with respect to is , at every .
You can estimate any Jacobian numerically the way the gradients lesson checked a gradient: nudge one input up and down by a small , take the change in every output divided by , and you have one column. Do it once per input. The next cell does exactly that for with a 2 × 3 matrix, so the answer is known before the code runs.
A numerical Jacobian
One column per input, by central differences, for a linear map whose Jacobian we know.
Setup (runs before every cell) · 9 lines
The lesson runs this once, before its first cell, and every cell, drill and example starts with what it defines. A cell that changes a setup object changes it for the cells after it, until the page reloads.
import torch def jacobian(f, x, h=1e-3): y = f(x) J = torch.zeros(y.shape[0], x.shape[0]) for j in range(x.shape[0]): e = torch.zeros_like(x); e[j] = h J[:, j] = (f(x + e) - f(x - e)) / (2 * h) return J
import torch W = torch.tensor([[1.0, 2.0, 0.0], [0.0, -1.0, 3.0]]) x = torch.tensor([1.0, 1.0, 1.0]) def jacobian(f, x, h=1e-3): y = f(x) J = torch.zeros(y.shape[0], x.shape[0]) # one row per output, one column per input for j in range(x.shape[0]): e = torch.zeros_like(x); e[j] = h # column j: every output's slope along input j J[:, j] = (f(x + e) - f(x - e)) / (2 * h) return J print(jacobian(lambda v: W @ v, x)) print(W)
Two rows, three columns, one entry per (output, input) pair, and the estimate matches W to three decimals. The 0.9999 and 3.0001 in place of 1 and 3 are float32 rounding in the differences, not a property of the function. Read it by rows: output 0 depends on inputs 0 and 1 with slopes 1 and 2 and ignores input 2; output 1 ignores input 0. The Jacobian of a linear map is the matrix, at every point.
import torch W = torch.tensor([[1.0, 2.0, 0.0], [0.0, -1.0, 3.0]]) def jacobian(f, x, h=1e-3): y = f(x) J = torch.zeros(y.shape[0], x.shape[0]) for j in range(x.shape[0]): e = torch.zeros_like(x); e[j] = h J[:, j] = (f(x + e) - f(x - e)) / (2 * h) return J print(jacobian(lambda v: W @ v, torch.tensor([1.0, 1.0, 1.0])).shape)
Show the answer
torch.Size([2, 3])
Rows follow the outputs (2), columns follow the inputs (3). The Jacobian has the shape of W, outputs × inputs.
Each Step nudges one input and writes one column. The result is W itself, because the map is linear.
Write a sentence or two, then compare it with a good answer in the lesson.
In PyTorch: Jacobians you can run
The chain rule becomes matrix multiplication
With numbers, the chain rule for multiplies two slopes. With vectors, each slope is a Jacobian and the multiplication is a matrix product:
The order is fixed: the outer function's Jacobian on the left. The shapes enforce it. If maps inputs to values and maps those to outputs, then is , is , and their product is the Jacobian of the whole function. Any other arrangement does not multiply.
Elementwise functions, the activations, have the simplest Jacobians of all. If , output depends only on input , so the Jacobian is diagonal: on the diagonal and zero elsewhere. The next cell takes of the linear map from before, builds that diagonal matrix , and checks that equals the Jacobian measured numerically.
The chain rule as a matrix product
tanh after the linear map: the diagonal Jacobian of tanh times W, against the numerical estimate.
Uses the setup above, 9 lines.
import torch W = torch.tensor([[1.0, 2.0, 0.0], [0.0, -1.0, 3.0]]) x = torch.tensor([1.0, 1.0, 1.0]) h = W @ x # the inner function's output: [3, 2] D = torch.eye(2) * (1 - torch.tanh(h) ** 2) # the diagonal Jacobian of tanh at h print("numerical:", jacobian(lambda v: torch.tanh(W @ v), x)) print("D @ W: ", D @ W) print("D:", D)
The inner map sends x to h = [3, 2], where tanh is nearly flat: 1 − tanh²(3) = 0.0099 and 1 − tanh²(2) = 0.0707, the two diagonal entries of D. Multiplying D into W scales each row of W by its output's slope: row 0 becomes 0.0099 × [1, 2, 0] = [0.0099, 0.0197, 0], row 1 becomes 0.0707 × [0, −1, 3] = [0, −0.0707, 0.2120]. The numerical Jacobian agrees to four decimals. Chain rule, shapes and all: (2 × 2)(2 × 3) = 2 × 3.
Show the answer
2 × 5, the product of a 2 × 3 and a 3 × 5 matrix
Outer Jacobian (2 × 3) on the left, inner (3 × 5) on the right; the shared 3 disappears and 2 × 5 remains: outputs × inputs of the whole function.
Write the numerical Jacobian
Write jacobian(f, x, h=1e-3): f maps a 1-D tensor to a 1-D tensor. Return a tensor of shape (outputs, inputs) whose column j is the central difference (f(x + h·e_j) − f(x − h·e_j)) / (2h).
Uses the setup above, 9 lines.
import torch def jacobian(f, x, h=1e-3): return torch.zeros(0, 0) W = torch.tensor([[1.0, 2.0, 0.0], [0.0, -1.0, 3.0]]) print(jacobian(lambda v: W @ v, torch.tensor([1.0, 1.0, 1.0])))
Backprop never builds the Jacobian
Training needs the gradient of one number, the loss, with respect to everything. By the chain rule that is : the Jacobian's transpose times the upstream gradient, a vector-Jacobian product. The result has the shape of , as a gradient always has the shape of the thing it is for.
Backprop computes that product without ever forming . The autograd engine from the autograd lesson does it one node at a time: the line that adds the other input's value times the output's gradient into a factor's gradient is a vector-Jacobian product with a 1 × 1 Jacobian, the local slope, times the gradient arriving from above. Every backward function in that class is the same move.
The saving is not cosmetic. The tiny GPT's output layer turns 48 hidden vectors of 32 numbers into 48 rows of 195 logits. The Jacobian of logits with respect to hidden states has 9,360 × 1,536 entries, over fourteen million. The vector-Jacobian product needs one multiply-add per weight per position, 299,520 of them, 48 times fewer, and the ratio grows with the sequence length. The next two cells show the product agreeing with autograd on the tanh example, then the count.
The vector-Jacobian product is what autograd computes
A loss whose gradient at y is a chosen vector g. Autograd's x.grad against Jᵀg from the numerical Jacobian.
Uses the setup above, 9 lines.
import torch W = torch.tensor([[1.0, 2.0, 0.0], [0.0, -1.0, 3.0]]) x = torch.tensor([1.0, 1.0, 1.0]) g = torch.tensor([1.0, -2.0]) # the gradient arriving from above, one number per output xg = torch.tensor([1.0, 1.0, 1.0], requires_grad=True) y = torch.tanh(W @ xg) (y * g).sum().backward() # a loss whose gradient with respect to y is exactly g print("autograd x.grad:", xg.grad) print("J^T g: ", jacobian(lambda v: torch.tanh(W @ v), x).T @ g)
Multiplying y by a fixed g and summing makes a loss whose gradient with respect to y is g, so autograd's x.grad is the vector-Jacobian product. It matches Jᵀg entry for entry: 0.0099 = 1 × 0.0099, 0.1610 = 1 × 0.0197 − 2 × (−0.0707), −0.4239 = −2 × 0.2120, with the numerical 0.1611 a rounding away. Autograd never computed the 2 × 3 matrix: it pushed the two upstream numbers back through tanh (multiply by the slopes) and then through W (multiply by Wᵀ).
Jacobian sizes in the tiny GPT
The output layer: 48 positions, 32 hidden numbers each, 195 logits each. The full Jacobian against the product backprop computes.
Uses the setup above, 9 lines.
T, d, V = 48, 32, 195 # tokens, hidden width, vocabulary of the tiny GPT logits, hidden = T * V, T * d print("logits:", logits, " hidden:", hidden) print("full Jacobian entries:", logits * hidden) print("vector-Jacobian multiply-adds:", logits * d) print(f"ratio: {logits * hidden / (logits * d):.0f}")
Every logit depends on the hidden numbers at its own position only, so most of the 14,376,960 Jacobian entries would be zeros, and the useful ones are the 6,240 weights of the output layer. The product needs each weight once per position: 9,360 × 32 = 299,520 multiply-adds. The ratio is the sequence length, 48, because the full Jacobian pretends every position could affect every other. A production model's vocabulary and context are each hundreds of times larger, which makes the full matrix unthinkable and the product routine.
import torch W = torch.tensor([[1.0, 2.0, 0.0], [0.0, -1.0, 3.0]]) g = torch.tensor([1.0, 1.0]) print((W.T @ g).tolist())
Show the answer
[1.0, 1.0, 3.0]
Wᵀ is 3 × 2; times a vector of two it gives three numbers, the column sums of W: 1 + 0, 2 − 1, 0 + 3. A gradient has the shape of its variable, here the three inputs.
Show the answer
f′(g(x)) · g′(x): the outer slope at the inner value, times the inner slope
The outer slope is taken where the outer function actually runs, at g(x). With vectors, the same rule reads J_f(g(x)) · J_g(x).
Elementwise, diagonal, cheap
Put the two ideas together for the activations. An elementwise function has a diagonal Jacobian, so its vector-Jacobian product is an elementwise multiply: . No matrix is built and nothing is summed; the backward pass through ReLU, GELU or tanh costs the same as the forward pass, one operation per number. The next cell measures ReLU's Jacobian at : a diagonal of 1, 0, 1, because ReLU passes positive inputs with slope 1 and blocks negative ones with slope 0.
Softmax is the exception, and the reason those other lessons exist. Every output of softmax depends on every input through the shared denominator, so its Jacobian is a full matrix, and so is the Jacobian of attention, which contains a softmax. Those two take a lesson each: The softmax Jacobian and p − y and Backprop through attention. The linear layer, whose Jacobian is the weight matrix, has its own lesson.
An elementwise function's Jacobian is diagonal
ReLU at [1, −2, 3], measured numerically.
Uses the setup above, 9 lines.
import torch print(jacobian(torch.relu, torch.tensor([1.0, -2.0, 3.0])))
Diagonal, as every elementwise function's Jacobian is: output i moves only with input i. The slopes are 1 for the positive inputs and 0 for −2, where ReLU is flat; the 0.9999 is the finite-difference estimate of 1, as before. Backprop through this layer multiplies the upstream gradient by [1, 0, 1], which is why a unit whose input is negative passes no gradient: its diagonal entry is 0.
Show the answer
[1.0, 0.4]: each entry times its own slope
A diagonal Jacobian makes the vector-Jacobian product an elementwise multiply: 2 × 0.5 and 4 × 0.1.
Multiply a chain of Jacobians
Write chain_jacobians(Js): Js lists the Jacobians of a chain of functions from the innermost to the outermost. Return the Jacobian of the whole chain (the outermost on the left). Raise ValueError if two neighbors cannot be multiplied.
Uses the setup above, 9 lines.
import torch def chain_jacobians(Js): return Js[0] J_g = torch.ones(3, 5) # inner: 5 inputs to 3 values J_f = torch.ones(2, 3) # outer: 3 values to 2 outputs print(chain_jacobians([J_g, J_f]).shape)
Two Jacobians as pictures: diagonal and full
The numerical Jacobian of tanh (elementwise) and of softmax (every output depends on every input), drawn as heat maps.
Uses the setup above, 9 lines.
import matplotlib.pyplot as plt x = torch.tensor([0.5, -1.0, 2.0, 0.0]) J_tanh = jacobian(torch.tanh, x) J_soft = jacobian(lambda v: torch.softmax(v, dim=0), x) fig, axes = plt.subplots(1, 2, figsize=(6.5, 3)) for ax, J, name in zip(axes, (J_tanh, J_soft), ("tanh", "softmax")): im = ax.imshow(J.tolist(), cmap="RdBu", vmin=-1, vmax=1) ax.set_title(name) ax.set_xlabel("input") ax.set_ylabel("output") off = J_tanh - torch.diag(torch.diag(J_tanh)) print("tanh off-diagonal is zero:", bool((off.abs() < 1e-3).all())) print("softmax rows sum to about 0:", [round(float(r.sum()), 3) for r in J_soft])
The tanh picture is a diagonal: output i moves only with input i. The softmax picture is full, with positive entries on the diagonal and negative ones off it, because raising one logit takes probability from all the others. Each row of the softmax Jacobian sums to about 0, since the probabilities always add to 1.
Write a sentence or two, then compare it with a good answer in the lesson.
In real models: why autograd works this way
An engineer meets Jacobians as shapes and memory, not as tables. The map shows where the gradient starts.
Where this shows up in an engineer's week
On the job: the Jacobian view explains three everyday facts. A gradient has the shape of the tensor it belongs to, since it is a vector-Jacobian product with one number per input. A model's backward pass costs about twice its forward pass, since each layer does two such products. And an out-of-memory error in training is nearly always stored activations, not Jacobians, which are never built.
Goes deeper in
- Backprop through a linear layer: the Jacobian of a linear layer, worked out in full.
- The softmax Jacobian and p − y: the one full Jacobian in a GPT, and why cross-entropy tames it.
- Build an autograd engine: how the library records the graph that these products run over.
Write a sentence or two, then compare it with a good answer in the lesson.
You can now
- Write the Jacobian of a vector function, with its shape, and estimate it numerically.
- Apply the chain rule to vector functions as a product of Jacobians, outer on the left.
- Explain why backprop computes vector-Jacobian products and never builds a Jacobian, with the tiny GPT's numbers.
- Recognize diagonal Jacobians in elementwise functions and the full one in softmax.
Next: the three formulas for a linear layer, derived by shape and by entry.
Practice what you just read
- 6 code examples you can edit and run
- a practice exercise that checks your code
- a 5-question quiz
- 10 review cards for spaced repetition
Try a quiz question
A function maps 4 inputs to 6 outputs. Its Jacobian has shape:
Before this lesson
- The chain rule with vectors
- Gradients in matrix form, and how to check them
- The linear layer as algebra
- Build an autograd engine
The math behind this
- The linear layer as algebra
- The chain rule with vectors
- Gradients in matrix form, and how to check them
Go deeper, if you like
- Backprop through a linear layer Optional
- The softmax Jacobian and p − y Optional
Where this lesson leads.
575 lessons in thirteen phases. 176 are free without an account, and a free account opens the rest. See every lesson.
Phase 5 · about 36 hours of study
Transformers
Attention, GPT from scratch and pretraining
47 lessons 1 free
- Language modeling FreePredict the next token, and score it with cross-entropy.30 min
- Tokenization and BPEHow tokenizers are built from data.40 min
- A bigram modelThe simplest language model, built from counts.35 min
- Sampling strategiesTemperature, top-k and top-p: turning probabilities into text.35 min
- Beam search and constrained decodingWhy the likeliest word isn't the likeliest sentence, beam search with log probabilities, its taste for short and repetitive text, and forcing output into a grammar: what that guarantees and what it doesn't.40 min
- Neural language modelsWhy counting fails, and what embeddings fix.35 min
- Attention: the intuitionEach token decides which other tokens matter to it.30 min
- Queries, keys and valuesThe three projections behind attention.35 min
- Scaled dot-product attentionThe attention formula, line by line.35 min
- Causal maskingStop tokens from seeing the future.25 min
- Multi-head attentionSeveral attention patterns in parallel.40 min
- Positional encodingsTell the model where each token sits.30 min
- The transformer blockAttention, MLP, residuals and norms together.40 min
- The GPT architectureA stack of decoder blocks that predicts the next token.40 min
- Build a tiny GPTWrite and train a small GPT from scratch.45 min
- From your GPT to GPT-2 in Hugging FaceMatch your GPT's tensors to GPT-2's names and shapes, count its 124,439,808 parameters, load renamed weights into your own class, pad a batch and read a config.40 min
- Training a GPT wellWarmup and cosine decay, gradient clipping, a validation loss you can trust, saving and loading, and two training mistakes read off the loss curve.35 min
- Encoders and embedding modelsBERT-style models and where they fit.40 min
- Tokenization: text to integer IDsFollow "The cat sat" into the three integers a model receives.30 min
- Embeddings and positional wavesFrom three IDs to the 3 × 8 matrix that enters the first block.35 min
- Inside the attention matrixEvery entry of the 3 × 3 attention matrix, computed and steered.40 min
- The feed-forward layerExpand to four times d_model and back: shapes, parameters and what the layer stores.35 min
- The LM head and samplingFrom the last row to a probability for every token, and the next word.35 min
- Pretraining at scaleData, compute and what models learn from the internet.40 min
- Scaling lawsHow loss falls predictably with size, data and compute.40 min
- Inference and the KV cacheWhy generation is fast after the first token.40 min
- The block in today's open models OptionalRMSNorm, SwiGLU, rotary positions and grouped-query attention swapped into the block you built, with the parameter and cache counts before and after.35 min
- Long contextEfficient attention and million-token windows.35 min
- Mixture of expertsRoute each token to a few specialist sub-networks.35 min
- Capstone: diagnose your own GPTEntropy per position, a head's attention pattern and a head ablation on the tiny GPT, then the same three measurements on a GPT you train yourself.45 min
- Jacobians: derivatives with shapes OptionalThe derivative of a vector function is a matrix; the chain rule multiplies them, and backprop multiplies by them without building them.35 min
- Backprop through a linear layer OptionaldX = dY Wᵀ, dW = Xᵀ dY, db = column sums: derived by shape and by entry, checked on the tiny GPT's own projection.35 min
- The softmax Jacobian and p − y OptionalEvery output depends on every input: the full Jacobian, its cheap vector product, and why cross-entropy's gradient is p − y.35 min
- Backprop through attention OptionalFive formulas from the output back to queries, keys and values; why masked scores get zero gradient; what FlashAttention recomputes.40 min
- Multimodal transformersHow models see images and hear audio.35 min
- Object detection: boxes, overlap and suppressionEvery object and where it is: boxes in two formats, intersection over union, why one object gets many boxes, non-maximum suppression and its threshold, and grading with average precision.45 min
- Segmentation: a label for every pixelMasks, and how to grade them with IoU and Dice; why a pixel's label needs context and too much context blurs; the shrink-then-enlarge network with skip connections; and turning a mask into countable objects.45 min
- Build a vision transformerPatch embeddings, a class token, position embeddings and an unmasked block, trained in the page; then two failures that explain it: no positions means no layout, and few images means losing to a CNN a fraction of its size.45 min
- Image-text models in depth: retrieval, negatives and limitsHow to test a shared image-text space with recall at k, what the rest of the batch teaches, what hard and false negatives do, and why a similarity score can't see word order, counts or "no".45 min
- Self-supervised learning: labels from the data itselfPretrain an encoder with no labels by matching two views of each image, test it with a linear probe, and see what the augmentations remove, what collapse looks like, and how filling in masked parts differs.45 min
- Diffusion 1: turning data into noiseThe easy half of a diffusion model: one step of noise, why the data is shrunk as noise is added, the jump to any step with alpha bar, how to read a noise schedule, and the training examples a denoiser learns from.40 min
- Diffusion 2: learning to denoise, and samplingTrain a network to predict noise, then run it a hundred times from pure noise and watch a ring assemble: the sampling loop, why fresh noise goes back in, fewer steps and where they break, and a check that samples are new.45 min
- Diffusion 3: images, labels and guidanceFrom points to small images, a label the denoiser can be asked with, and classifier-free guidance: why a model that is told what to draw still often doesn't, how the guidance scale fixes it, and what it breaks when turned up too far.45 min
- Diffusion 4: latent diffusion and the whole pipelineWhy image generators diffuse an autoencoder's codes and not pixels: a latent diffusion model in miniature, what the autoencoder must be trained to do, the three parts of a text-to-image system, and where the cost of an image goes.40 min
- Sound as numbers: waveforms and spectrogramsA recording as an array of samples, why the sampling rate must be twice the highest pitch, spectrograms and the window trade-off, and the log-mel features a speech model is given.40 min
- Recognition and synthesis as a pipelineA whole speech recognizer for a made-up language: features, a guess per frame, collapsing repeats and blanks, a dictionary to repair mistakes, word error rate, and synthesis as the same stages run backward.40 min
- The voice agent loop and its latency budgetOne turn of a voice agent, timed: why every stage must stream, how to decide that someone has finished speaking, what to do when they interrupt, and why the slow turns matter more than the average.45 min
Never trained an AI? Teach a small one to write, right in your browser. It takes about 20 seconds and needs no account. Teach it to write