Index Ai Academy
Foundation · Foundation mathCalculus

The gradient

About 42 minLooking ahead: some prerequisites aren't done yet

Why this matters

Lesson 77 gave one slope per input. Put them side by side and they make a vector, the gradient, and that vector has a meaning of its own: it points in the direction the function climbs fastest, and its length says how steep that climb is. Turn it around and it points straight downhill. That's the direction gradient descent (lesson 79) moves a model's weights, every step of every training run. This lesson builds the gradient from partials, measures it numerically for any number of inputs, and computes it for a line's loss, where it turns out to be a dot product you met in lesson 50. Then PyTorch's loss.backward() finds the same gradient for you.

  • Collect all the partial derivatives into one vector, the gradient ∇f
  • Read the gradient as the direction of steepest climb, its length as the steepness, and its opposite as steepest descent
  • Write numerical_gradient(f, point) for any number of inputs, and check it against the hand formula
  • Compute the gradient of a line's mean squared error, with ∂L/∂w as a dot product of errors and inputs, and get the same gradient from PyTorch's loss.backward()
Quick check
Have a guess before you start. On a hillside, the ground rises 2 m per step east and 6 m per step north. Which way is the climb steepest?

The math: all the slopes as one arrow

Purpose. All the slopes at once, as a direction.

The gradient of f is the vector of its partial derivatives, written ∇f ("del f" or "grad f"):

∇f=(∂f∂x,∂f∂y)\nabla f = \left( \frac{\partial f}{\partial x}, \frac{\partial f}{\partial y} \right)

In plain words: one slope per input, stacked into an arrow.

For f = x² + 3y² at (1, 2), the partials are 2 and 12, so ∇f = (2, 12). Three facts make it useful:

  • it points in the direction where f rises fastest;
  • its length is that fastest rate: √(2² + 12²) ≈ 12.17;
  • its opposite, −∇f = (−2, −12), points where f falls fastest.