So far each derivative has been a single slope, or a gradient: one slope per input, for one output. Layers in a network have many outputs too: 784 pixels in, 128 numbers out. The Jacobian collects every slope of every output with respect to every input into one matrix. And to understand the shape of a loss, flat or steep, bowl or saddle, you need the slopes of the slopes: the Hessian. These two words appear constantly in machine-learning papers. This lesson builds both by hand and by nudging, connects the Jacobian of a layer to its weight matrix, checks it on a PyTorch layer, and uses the Hessian to explain why saddle points, not bad valleys, are what slow deep networks down.
What you'll be able to do
Build the Jacobian of a function with vector inputs and outputs: one row per output, one column per input
See that the Jacobian of a layer z = Wx is W itself, check it on PyTorch's nn.Linear by nudges and by .grad, and see that an element-wise function's Jacobian is diagonal
Build the Hessian, the table of second partials, and use it to tell a bowl, a dome and a saddle apart
Read the words advanced papers use: Jacobian shapes of layers, and saddle points in training
Quick check
Have a guess before you start. A function takes 2 numbers in and gives 3 numbers out. How many slopes are there, one for each output and each input?
5
6
2
3
Show the answer
6
Each of the 3 outputs has a slope with respect to each of the 2 inputs: 3 × 2 = 6.
The math: all the slopes in a grid
Purpose. Derivatives when inputs and outputs are both vectors, and curvature in many directions.
For a function f from n inputs to m outputs, the Jacobian J is an m × n matrix: row i holds the gradient of output i, and column j says how every output responds to input j.
Jij=∂xj∂fi
In plain words: one row per output, one column per input, each entry one slope.
For a linear layer z = Wx, output i is Σⱼ Wᵢⱼxⱼ, whose slope with respect to xⱼ is just Wᵢⱼ. So the Jacobian of a layer is its weight matrix: nudge input j, and the outputs move by column j of W.
Quick check
From lesson 55: what is the shape of W x when W is 3 × 2?
1 number out
2 numbers out, from 3 in
6 numbers out
3 numbers out, from 2 in
Show the answer
3 numbers out, from 2 in
W x has one entry per row of W: 3 outputs from 2 inputs. Its Jacobian is W, 3 × 2.
The picture. Step through the Jacobian of a 2 → 3 layer. The drawing is the Jacobian itself, a grid of slopes: one row per output, z₁ to z₃, and one column per input, x₁ and x₂. It starts as question marks, slopes still to measure. Each step nudges one input, and because one nudge moves every output at once, a whole column fills in, and it is a column of W. Darker cells are bigger slopes. The code lane's PyTorch tab writes the same nudge for a PyTorch layer.
Try it now
Then add a third number to each line of matrix W, 4 on the first line, 0 on the second and 2 on the third, and press Use these numbers: a third input, so the grid gains a third column.
Element-wise functions. An activation like relu acts on each entry separately: output i depends only on input i. So every slope off the diagonal is 0, and the Jacobian is diagonal. For relu at z = (2, −1, 0.5), it's diag(1, 0, 1): 1 where the unit is on, 0 where it's off. Backpropagation multiplies by such Jacobians without ever building them, which is what lesson 81's da * (z > 0) did.
The Hessian. For a function with one output and several inputs, the Hessian H is the table of second partials, the slopes of the slopes:
Hij=∂xi∂xj∂2ff=x2+3y2⇒H=(2006)
In plain words: the curvature in each direction, plus how the slope in one direction changes as you move in another. The order doesn't matter, so H is symmetric (lesson 57).
Quick check
What is the Hessian of f = x² + xy + y²?
[[1, 1], [1, 1]]
[[2, 1], [0, 2]]
[[2, 1], [1, 2]]
[[2, 0], [0, 2]]
Show the answer
[[2, 1], [1, 2]]
∂²f/∂x² = 2, ∂²f/∂y² = 2, and the cross term ∂²f/∂x∂y = 1, in both off-diagonal places.
Bowl, dome, saddle. At a flat point, the Hessian's eigenvalues (lesson 62) say what shape the surface has:
all positive: a bowl, curving up in every direction: a minimum. x² + 3y² has eigenvalues 2 and 6.
all negative: a dome: a maximum. −x² − y² has −2 and −2.
mixed signs: a saddle, up in some directions and down in others, like a mountain pass. x² − y² has 2 and −2.
A saddle has zero slope but isn't a minimum: there's always a way further down.
The picture. Each frame below draws one surface from above, as level curves, like the contour lines on a hiking map. From the flat point at (0, 0), two short paths run along x and along y, each labeled with the surface's curvature in that direction: the Hessian's eigenvalues, since these Hessians are diagonal.
In your own words
In your own words: why is the Jacobian of a linear layer just its weight matrix?
Write a sentence or two, then compare it with a good answer in the lesson.
In Python: a numerical Jacobian
Purpose. Measure every slope of a vector function by nudging one input at a time.
import numpy as np
defjacobian(f, x):
h = 0.000001
fx = f(x)
J = np.zeros((len(fx), len(x)))
for j inrange(len(x)):
up = x.copy()
down = x.copy()
up[j] = up[j] + h
down[j] = down[j] - h
J[:, j] = (f(up) - f(down)) / (2 * h)
return J
W = np.array([[1.0, 2.0], [0.0, -1.0], [3.0, 1.0]])
deflayer(x):
return W @ x
print(np.round(jacobian(layer, np.array([0.5, 2.0])), 4).tolist())
print(np.round(jacobian(np.tanh, np.array([0.0, 1.0])), 4).tolist())
It runs right here in your browser, with nothing to install.
Each pass fills one column, J[:, j], with how all outputs moved when input j moved. The layer's Jacobian is W exactly. tanh acts entry by entry, so its Jacobian is diagonal: slope 1 at 0 and 0.42 at 1.
Code flow. Step through the same loop on the math part's layer, one line at a time, with W written out in place of f. The nudge here is h = 0.5, which keeps the boxes readable. A layer is a straight-line function, so a nudge of any size measures its slopes exactly. Watch up and down differ from x in place j only, and J gain one column per pass.
A named mistake: rows and columns swapped. It's easy to fill J[j], a row, instead of J[:, j], a column. One nudge gives one number per output, but a row of J has one place per input. When the two counts differ, NumPy stops with an error, because the outputs don't fit in a row. When they're equal, nothing stops you, and J quietly comes out as the transpose, with the right shape. So check one slope you know by hand: row i, column j must be output i's slope for input j.
Hessians numerically. A Hessian is the Jacobian of the gradient: nudge each input and see how the whole gradient changes. For functions whose second partials are simple, working them out by hand is easier, and the eigenvalues then come from np.linalg.eigvalsh, the version of lesson 62's eigh that returns only eigenvalues.
Worked example
Worked example: classify four surfaces
Write down the Hessians of the picture's three surfaces and of x² + xy + y², and classify each flat point from its eigenvalues.
import numpy as np
hessians = [("bowl", np.array([[2.0, 0.0], [0.0, 6.0]])),
("dome", np.array([[-2.0, 0.0], [0.0, -2.0]])),
("saddle", np.array([[2.0, 0.0], [0.0, -2.0]])),
("tilted bowl", np.array([[2.0, 1.0], [1.0, 2.0]]))]
for name, H in hessians:
ev = np.linalg.eigvalsh(H)
if np.all(ev > 0):
shape = "minimum"elif np.all(ev < 0):
shape = "maximum"else:
shape = "saddle"print(name, np.round(ev, 3).tolist(), shape)
In the lesson you can change this and run it.
x² + 3y² has eigenvalues 2 and 6, both positive: a minimum. −x² − y² has −2 and −2, both negative: a maximum. x² − y² has −2 and 2: a saddle. x² + xy + y² has off-diagonal entries, so you can't read the curvatures straight off the diagonal, but its eigenvalues, 1 and 3, are both positive: a bowl turned at an angle. np.all checks that every eigenvalue passes the test.
Quick check
What does this print?
import numpy as np
W = np.array([[2.0, 0.0, 1.0], [1.0, 1.0, 0.0]])
x = np.array([1.0, 1.0, 1.0])
print((W @ x).shape, W.shape)
(3,) (2, 3)
(2,) (3, 2)
(2, 3) (2, 3)
(2,) (2, 3)
Show the answer
(2,) (2, 3)
3 inputs give 2 outputs, and the Jacobian, W, is 2 × 3: one row per output, one column per input.
Quick check
What does this print?
import numpy as np
print(np.round(np.linalg.eigvalsh(np.array([[3.0, 1.0], [1.0, 3.0]])), 3).tolist())
[3.0, 3.0]
[2.0, 4.0]
[0.0, 6.0]
[4.0, 2.0]
Show the answer
[2.0, 4.0]
Another tilted bowl, like x² + xy + y² above: its curvatures are 3 − 1 = 2 and 3 + 1 = 4, sorted smallest first. Both are positive, so a minimum.
Drill
A Jacobian, column by column
Fix jacobian so it fills columns, not rows. Then set J to the Jacobian of f(x) = (x₀x₁, x₀ + 3x₁, x₁²) at (2, 1), rounded to 4 places, as a list of lists.
import numpy as np
defjacobian(f, x):
h = 0.000001
fx = f(x)
J = np.zeros((len(fx), len(x)))
for j inrange(len(x)):
up = x.copy()
down = x.copy()
up[j] = up[j] + h
down[j] = down[j] - h
J[j] = (f(up) - f(down)) / (2 * h)
return J
deff(x):
return np.array([x[0] * x[1], x[0] + 3 * x[1], x[1] ** 2])
J = []
print(J)
In the lesson, your code is checked as you write it.
In your own words
For the drill's f, 2 inputs and 3 outputs, filling J[j] instead of J[:, j] stops with an error. Why does the same slip give no error for a function with 2 inputs and 2 outputs, and how would you catch it there?
Write a sentence or two, then compare it with a good answer in the lesson.
In AI and ML: the words in the papers
Purpose. Read Jacobian shapes and saddle points the way papers use them, and check a real layer's Jacobian in PyTorch.
Backpropagation multiplies Jacobians. The chain rule for vector functions multiplies Jacobians: for x → z → a, ∂a/∂x = (∂a/∂z)(∂z/∂x). A network's backward pass multiplies a gradient by each layer's Jacobian in turn, but never writes one out as a table. A layer's Jacobian is its W, so its step back is one @ with Wᵀ. Writing out relu's would waste memory: for 4,096 units it has 4,096 × 4,096 entries, about 16.8 million, and every one off the diagonal is 0. So backprop keeps only the diagonal, one 0 or 1 per unit.
∂x∂a=activationdiag(relu′(z))layerW
In plain words: a layer followed by relu has Jacobian "relu's 0s and 1s on the diagonal, times W".
A real layer's Jacobian. The trail club's sign reader ends with a layer from 4 features of a sign to 3 scores, one each for arrow, tent and trail. Nudge each feature of one sign, and watch the scores move:
import numpy as np
W = np.array([[1.5, -0.5, 0.25, 2.0], [-1.0, 0.75, 1.0, -0.5], [0.5, 1.25, -1.5, 0.25]])
b = np.array([0.5, -0.25, 0.0])
x = np.array([2.0, 0.5, 1.0, 0.0])
print((W @ x + b).tolist())
h = 0.25
J = np.zeros((3, 4))
for j inrange(4):
e = np.zeros(4)
e[j] = h
J[:, j] = ((W @ (x + e) + b) - (W @ (x - e) + b)) / (2 * h)
print(J.tolist())
print(np.array_equal(J, W))
The sign scores 3.5 for arrow, the highest. The Jacobian is 3 × 4, one row per score and one column per feature, and it is W, number for number. The bias drops out, since it's added to both the nudged-up and the nudged-down scores. The nudge is 0.25 here: the layer is a straight line in every direction, so any nudge measures its slopes exactly, and a quarter keeps every number exact in binary.
In PyTorch. The same layer is nn.Linear(4, 3), loaded with W and b as in lesson 50. The nudges run under torch.no_grad(), because they are measurements, not training. The nudge's size matters more here. PyTorch works in float32 by default (lesson 12), whose rounding is too coarse for the Python part's nudge of 0.000001: a nudge that small moves the scores by little more than their own rounding, and the slopes come out wrong. A quarter is still exact.
import torch
import torch.nn as nn
layer = nn.Linear(4, 3)
with torch.no_grad():
layer.weight.copy_(torch.tensor([[1.5, -0.5, 0.25, 2.0], [-1.0, 0.75, 1.0, -0.5], [0.5, 1.25, -1.5, 0.25]]))
layer.bias.copy_(torch.tensor([0.5, -0.25, 0.0]))
x = torch.tensor([2.0, 0.5, 1.0, 0.0])
print(layer(x).tolist())
h = 0.25
J = torch.zeros(3, 4)
with torch.no_grad():
for j inrange(4):
e = torch.zeros(4)
e[j] = h
J[:, j] = (layer(x + e) - layer(x - e)) / (2 * h)
print(J.tolist())
print(torch.equal(J, layer.weight))
It prints the same scores and the same Jacobian as the NumPy cell, and torch.equal confirms that J is layer.weight, every entry. PyTorch stores a layer's weight as (outputs, inputs), 3 × 4, exactly the Jacobian's shape. Autograd (lesson 82) finds the same slopes without nudging: backward() from one score fills x.grad with that score's row of the Jacobian.
Quick check
The same layer, with x now recording its slopes. What does this print?
import torch
import torch.nn as nn
layer = nn.Linear(4, 3)
with torch.no_grad():
layer.weight.copy_(torch.tensor([[1.5, -0.5, 0.25, 2.0], [-1.0, 0.75, 1.0, -0.5], [0.5, 1.25, -1.5, 0.25]]))
layer.bias.copy_(torch.tensor([0.5, -0.25, 0.0]))
x = torch.tensor([2.0, 0.5, 1.0, 0.0], requires_grad=True)
layer(x)[2].backward()
print(x.grad.tolist())
[0.25, 1.0, -1.5]
[0.125]
[1.5, -0.5, 0.25, 2.0]
[0.5, 1.25, -1.5, 0.25]
Show the answer
[0.5, 1.25, -1.5, 0.25]
layer(x)[2] is the trail score. backward() from it fills x.grad with its slope for each of the 4 features: row 3 of the Jacobian, which is row 3 of W. That's how autograd builds a Jacobian when one is needed: one backward pass per output, one row each.
Saddles slow training. In very high dimensions, a flat point with a Hessian of mixed signs, a saddle, is far more common than a true minimum: with millions of directions, it's unlikely that every one curves up. Near a saddle the gradient is tiny, so training seems stuck, though there's a way down. Momentum (lesson 80) and noise from small batches help carry it through, which is part of why they work so well in practice.
Quick check
A layer maps 64 inputs to 10 outputs and is followed by relu. What is the shape of the combined Jacobian ∂a/∂x?
10 × 10
64 × 64
10 × 64
64 × 10
Show the answer
10 × 64
diag(relu′) is 10 × 10 and W is 10 × 64, so their product is 10 × 64: outputs × inputs.
Drill
Shapes through a network
A network maps 784 → 256 → 64 → 10, with relu after the first two layers. Set shapes to the list of Jacobian shapes (as tuples) of the three layers, and full to the shape of the whole network's Jacobian, from input to output.
31Formulas with fractions: clear them firstFreeClearing denominators, cross-multiplying, a letter in the bottom, and reciprocal formulas like 1/R = 1/R₁ + 1/R₂.35 min
37Inverse functions, lines and scalingFreeUndoing a function by making x the subject, domains that allow it, line forms, and unscaling predictions.40 min
38Sequences and sums: find n, a, d and rFreeEvery letter of the arithmetic and geometric formulas as the subject, and an average's window from its decay.40 min
40Review: rearrange anything (algebra)FreeBroken rearrangements to fix, mixed practice from the whole algebra block, a strategy chart, and its formula sheet.35 min
41Coordinates and distanceFreePoints as (x, y), Pythagoras, distance in any number of coordinates, and nearest neighbors.42 min
42Angles, degrees and radiansFreeMeasuring a turn, converting units, arcs, and the angle between two arrows.36 min
43Sine and cosineFreeAcross and up: right triangles, the unit circle, and directions and rotations.42 min
44Waves: height, speed and shiftFreeAmplitude, period and phase, sampling a wave, and position codes in transformers.37 min
68Vector equations: rearranging with arrowsFreeSolving for a vector or a scale, recovering a gradient from two weight snapshots, and an angle from a dot product.40 min
87Counting: choices, orders and groupsFreeThe counting principle, factorials, ordered picks and groups, and why attention's cost grows with the square of the length.43 min
88Probability: from counts to chancesFreeEqually likely outcomes, the complement, distributions, seeded simulation, and a next-word model.43 min
89Combining events: and, or, independentFreeThe or rule, independence, probability trees, drawing with and without replacement, and chains of steps.42 min
100Maximum likelihoodFreeLikelihoods as products, log-likelihoods as sums, grid search, underflow, and the negative log-likelihood loss.45 min
101Linear regression: the best lineFreeResiduals, least squares by formula, three roads to one line, the matrix form, and train versus test error.45 min
109One neuron, start to finishFreeWeighted sum, activation, loss, slopes and update, by hand and in code, and what one straight line can and can't split.45 min
111Training a small networkFreeThe backward pass in matrix form, gradient checks, a full training loop, and reading loss curves.45 min
112The same network in PyTorchFreenn.Sequential with the NumPy weights loaded, the same loss and slopes, then SGD and mini-batches.45 min
113Attention as matrix mathFreeScaled dot-product scores, the causal mask, a row-by-row softmax, and blending value vectors.45 min
114Residual connections and layer normFreeAdding the input back and why its slope keeps learning alive, layer norm by hand, and add and norm.40 min
115A transformer block, start to finishFreeA sentence through embeddings, attention, add and norm, a small network and next-word scores, every shape named.45 min
117Final test: transpose any formulaFreeSlips to fix from every topic, mixed practice of every kind of rearranging, the strategy chart and the full sheet.45 min
Learn the math behind AI from arithmetic up: numbers, algebra, vectors and matrices, calculus, probability and statistics, each shown as numbers moving and checked in Python, ending with a network you build and train from the formulas.
See the course →
Never trained an AI? Teach a small one to write, right in your browser. It takes about 20 seconds and needs no account. Teach it to write