Local derivative
How one small step in a computation graph changes its output when its input moves a little. Backpropagation multiplies these along the path.
What it means
A computation graph breaks a calculation into small steps. A local derivative answers one question about one step: if I nudge this input a little, how much does this step's output move? "Local" means it looks only at that step's own inputs, not at the whole network, which is what makes it easy to work out.
Backpropagation gets the gradient of the whole network by multiplying local derivatives along the path from the output back to each weight. That's the chain rule.
An example in Python
# y = (a * b) + c, computed in two small steps def forward(a, b, c): d = a * b return d, d + c a, b, c = 2.0, -3.0, 10.0 d, y = forward(a, b, c) # Each step's local derivative only looks at its own inputs. dd_da = b # for d = a * b, nudging a moves d by b dy_dd = 1.0 # for y = d + c, nudging d moves y by 1 print("backpropagation multiplies along the path: dy/da =", dy_dd * dd_da) # Check it by really nudging a. nudge = 0.001 _, y2 = forward(a + nudge, b, c) print("by nudging a:", round((y2 - y) / nudge, 6))
backpropagation multiplies along the path: dy/da = -3.0 by nudging a: -3.0
Two small facts multiply to give the answer, and really nudging a confirms it. A network with millions of weights is the same idea: every step only has to know its own local derivative. To step through this visually, try the backpropagation explainer.
Try it now
Forward computes the values. Backward multiplies the local slopes, one step at a time.
Learn it properly
Computation graphs · Every calculation as a graph of small operations. About 30 minutes.
Other terms from the same lesson
- computation graph: A map of every calculation a model did, which lets the computer trace each result back to its inputs.
- topological order: An order for the steps of a graph where every step comes after the steps it depends on.
Never trained an AI?
Teach a small one to write, right in your browser. It takes about 20 seconds and needs no account.