Jacobians and Hessians
Why this matters
So far each derivative has been a single slope, or a gradient: one slope per input, for one output. Layers in a network have many outputs too: 784 pixels in, 128 numbers out. The Jacobian collects every slope of every output with respect to every input into one matrix. And to understand the shape of a loss, flat or steep, bowl or saddle, you need the slopes of the slopes: the Hessian. These two words appear constantly in machine-learning papers. This lesson builds both by hand and by nudging, connects the Jacobian of a layer to its weight matrix, checks it on a PyTorch layer, and uses the Hessian to explain why saddle points, not bad valleys, are what slow deep networks down.
- Build the Jacobian of a function with vector inputs and outputs: one row per output, one column per input
- See that the Jacobian of a layer z = Wx is W itself, check it on PyTorch's nn.Linear by nudges and by .grad, and see that an element-wise function's Jacobian is diagonal
- Build the Hessian, the table of second partials, and use it to tell a bowl, a dome and a saddle apart
- Read the words advanced papers use: Jacobian shapes of layers, and saddle points in training
The math: all the slopes in a grid
Purpose. Derivatives when inputs and outputs are both vectors, and curvature in many directions.
For a function f from n inputs to m outputs, the Jacobian J is an m × n matrix: row i holds the gradient of output i, and column j says how every output responds to input j.
In plain words: one row per output, one column per input, each entry one slope.
For a linear layer z = Wx, output i is Σⱼ Wᵢⱼxⱼ, whose slope with respect to xⱼ is just Wᵢⱼ. So the Jacobian of a layer is its weight matrix: nudge input j, and the outputs move by column j of W.