Index Ai Academy
Foundation · Foundation mathCalculus

Automatic differentiation: how PyTorch finds every slope

About 40 minLooking ahead: some prerequisites aren't done yet

Why this matters

Lesson 81 found every slope of a small network by hand: a forward pass that kept each value, then a backward pass that multiplied along each path and added across paths. PyTorch does that bookkeeping for you. It's called automatic differentiation, or autograd, and the idea behind it is a tape: as your code runs, each step is written down with what its slope will need, and backward() reads the tape from the end. This lesson builds that tape in plain Python, then matches it with PyTorch, slope for slope. Knowing what gets recorded explains zero_grad(), no_grad() and the commonest autograd error messages.

  • Record a calculation as a tape: one entry per step, each keeping its inputs and its local slopes
  • Walk the tape backward to get every slope, multiplying along a path and adding across paths
  • Use requires_grad, backward() and .grad, and match PyTorch's slopes to the ones found by hand
  • Explain why slopes add up unless they're cleared, what no_grad() turns off, and what detach() is for
Quick check
Have a guess before you start. You walk into the woods taking five turns, and write each turn in a notebook as you go. To walk back out, how do you use the notebook?

The math: record every step, then read the record backward

Purpose. Every slope of a calculation, from a record of how it ran.

A tape is a list of the steps a calculation ran, in the order they ran. Each step adds one entry, and an entry keeps three things: the value the step made, the inputs it used, and its local slope for each input, worked out right then, while the input values are at hand. Take the graph from lesson 81, x = t², y = c·t and z = x·y, at t = 2 and c = 3. Running it writes three entries:

entrymadefromlocal slopes
#1x = 2² = 4tdx/dt = 2t = 4
#2y = 3 × 2 = 6c, t∂y/∂c = t = 2, ∂y/∂t = c = 3
#3z = 4 × 6 = 24x, y∂z/∂x = y = 6, ∂z/∂y = x = 4

In plain words: the forward pass fills the tape, so every local slope is ready before the backward pass starts.