The chain rule with many paths: backpropagation by hand
Why this matters
Lesson 74's chain rule handled a single chain: one link after another. A real network isn't a single chain. One weight's effect splits, reaches the loss along several routes, and the routes join up again. Backpropagation is the chain rule extended to that: work backward from the loss, multiply local slopes along each path, and where paths meet, add. Done in the right order, every slope in the network comes out of one backward pass, reusing the work already done, instead of one expensive nudge per weight. This lesson does it all by hand on a small graph, then on a network with two inputs, two hidden units and one output: nine slopes, one update, a lower loss, and PyTorch agreeing with every slope.
- Draw a calculation as a graph of small steps, and run it forward
- Apply the two-clause rule: multiply along a path, add across paths
- Write a backward pass line by line, and check every slope against a numerical one
- Find all nine slopes of a 2-2-1 network by hand, check them against PyTorch, take one update, and see the loss fall
The math: multiply along a path, add across paths
Purpose. The slope of the loss for every weight, in one backward pass.
Write a calculation as a graph: each node is a small step, and an arrow says one value feeds another. Take z, which depends on x and y, which both depend on t:
To find dz/dt, follow every path from t to z. Along each, multiply the local slopes; then add the paths:
In plain words: multiply along a path, add across paths. At t = 2: x = 4, y = 6, so dz/dt = 6 × 4 + 4 × 3 = 36.