Index Ai Academy
Foundation · Foundation mathCalculus

Jacobians and Hessians

About 42 minLooking ahead: some prerequisites aren't done yet

Why this matters

So far each derivative has been a single slope, or a gradient: one slope per input, for one output. Layers in a network have many outputs too: 784 pixels in, 128 numbers out. The Jacobian collects every slope of every output with respect to every input into one matrix. And to understand the shape of a loss, flat or steep, bowl or saddle, you need the slopes of the slopes: the Hessian. These two words appear constantly in machine-learning papers. This lesson builds both by hand and by nudging, connects the Jacobian of a layer to its weight matrix, checks it on a PyTorch layer, and uses the Hessian to explain why saddle points, not bad valleys, are what slow deep networks down.

  • Build the Jacobian of a function with vector inputs and outputs: one row per output, one column per input
  • See that the Jacobian of a layer z = Wx is W itself, check it on PyTorch's nn.Linear by nudges and by .grad, and see that an element-wise function's Jacobian is diagonal
  • Build the Hessian, the table of second partials, and use it to tell a bowl, a dome and a saddle apart
  • Read the words advanced papers use: Jacobian shapes of layers, and saddle points in training
Quick check
Have a guess before you start. A function takes 2 numbers in and gives 3 numbers out. How many slopes are there, one for each output and each input?

The math: all the slopes in a grid

Purpose. Derivatives when inputs and outputs are both vectors, and curvature in many directions.

For a function f from n inputs to m outputs, the Jacobian J is an m × n matrix: row i holds the gradient of output i, and column j says how every output responds to input j.

Jij=∂fi∂xjJ_{ij} = \frac{\partial f_i}{\partial x_j}

In plain words: one row per output, one column per input, each entry one slope.

For a linear layer z = Wx, output i is Σⱼ Wᵢⱼxⱼ, whose slope with respect to xⱼ is just Wᵢⱼ. So the Jacobian of a layer is its weight matrix: nudge input j, and the outputs move by column j of W.