Index Ai Academy
Foundation · Foundation mathCalculus

The chain rule with many paths: backpropagation by hand

About 45 minLooking ahead: some prerequisites aren't done yet

Why this matters

Lesson 74's chain rule handled a single chain: one link after another. A real network isn't a single chain. One weight's effect splits, reaches the loss along several routes, and the routes join up again. Backpropagation is the chain rule extended to that: work backward from the loss, multiply local slopes along each path, and where paths meet, add. Done in the right order, every slope in the network comes out of one backward pass, reusing the work already done, instead of one expensive nudge per weight. This lesson does it all by hand on a small graph, then on a network with two inputs, two hidden units and one output: nine slopes, one update, a lower loss, and PyTorch agreeing with every slope.

  • Draw a calculation as a graph of small steps, and run it forward
  • Apply the two-clause rule: multiply along a path, add across paths
  • Write a backward pass line by line, and check every slope against a numerical one
  • Find all nine slopes of a 2-2-1 network by hand, check them against PyTorch, take one update, and see the loss fall
Quick check
Have a guess before you start. A town's income comes from tourists and from a factory, and both depend on the weather. To see how the weather affects income, what do you do with the two routes?

The math: multiply along a path, add across paths

Purpose. The slope of the loss for every weight, in one backward pass.

Write a calculation as a graph: each node is a small step, and an arrow says one value feeds another. Take z, which depends on x and y, which both depend on t:

x=t2,y=3t,z=x⋅yx = t^2, \quad y = 3t, \quad z = x \cdot y

To find dz/dt, follow every path from t to z. Along each, multiply the local slopes; then add the paths:

dzdt=∂z∂xdxdt+∂z∂ydydt=y⋅2t+x⋅3\frac{dz}{dt} = \frac{\partial z}{\partial x}\frac{dx}{dt} + \frac{\partial z}{\partial y}\frac{dy}{dt} = y \cdot 2t + x \cdot 3

In plain words: multiply along a path, add across paths. At t = 2: x = 4, y = 6, so dz/dt = 6 × 4 + 4 × 3 = 36.