Backpropagation, step by step

A network can have billions of weights, and training needs to know how each one affects the loss. Backpropagation gets all of those answers in one backward pass. It's the chain rule from calculus, applied from the output back to the start, and it's small enough to follow by hand.

What you just saw

The forward pass computes each value and remembers it. The backward pass starts with the loss and works toward the inputs. At each step it multiplies by the local slope, the amount that step's output changes when its input does. By the end, each weight has a number saying how much the loss would change if that weight moved. Training then nudges each weight the other way.

Why deep networks were hard to train

Those multiplications add up. If each layer's slope is below 1, the blame shrinks at every layer on its way back, and early layers barely learn. If it's above 1, the blame grows out of control.

One fix

A skip connection adds a path whose slope is exactly 1, so blame has a clear way back. It's one of the main reasons very deep networks can be trained.