Phase 2 · Math, when you need it · Calculus

The chain rule

Derivatives of functions inside functions: the heart of backprop.

About 30 minutes · A free account opens the whole lesson

Create a free account Try part of it here

Already have an account? Sign in to open this lesson

Why it matters

A neural network is a long chain of small steps. Each layer feeds the next one. At the end sits the loss: a score of how wrong the model was. Training means adjusting the model's inner numbers, its weights, to shrink that loss. So you need to know how each weight affects the loss, however far back in the chain it sits. The chain rule tells you: multiply the rates of change along the path from the weight to the loss. Backpropagation, the method that trains every modern model, is the chain rule organized so that each step is worked out only once.

What you'll be able to do

Differentiate a function inside a function by multiplying rates
Follow a computation graph forward (values) and backward (derivatives)
Compute the gradient of a single neuron's loss by hand
Explain backpropagation as the chain rule applied from the loss backwards

Rates multiply along a chain

Picture an ad campaign as a chain with three links. Spending money buys clicks. Some clicks turn into conversions (a sale or a sign-up). Each conversion brings in revenue.

Say each extra dollar buys 2 clicks. Each click yields 0.05 conversions, which is one conversion in every 20 clicks. Each conversion is worth $60. How much revenue does one extra dollar bring in?

Follow the dollar down the chain. It buys 2 clicks. Those make 2 × 0.05 = 0.1 conversions. Those are worth 0.1 × 60 = $6. So each extra dollar of spend is worth $2 \times 0.05 \times 60 = \$6$ of revenue.

Rates along a chain multiply. That one idea is the chain rule.

Quick check

Spend → clicks at 3 clicks per dollar, clicks → signups at 0.1 per click. What's the rate of signups per dollar?

0.03
0.3
3.1
30

Show the answer

0.3

Rates along a chain multiply: 3 × 0.1 = 0.3. Adding them (3.1) is a common mistake.

The chain rule

In code, a function inside a function looks like f(g(x)). First g runs on $x$ . Then f runs on whatever g gave back.

Give that middle value a name: $u = g(x)$ . The final answer is then $y = f(u)$ . Now there are three rates, each written like a fraction:

$\frac{du}{dx}$ is how fast the middle value $u$ changes as $x$ changes;
$\frac{dy}{du}$ is how fast the answer $y$ changes as $u$ changes;
$\frac{dy}{dx}$ is the one you want: how fast $y$ changes as $x$ changes.

Each one is a slope, the same idea as $f'(x)$ from the last two lessons. The chain rule says:

\frac{dy}{dx} = \frac{dy}{du} \cdot \frac{du}{dx} \qquad \text{where } u = g(x)

In plain words: the overall rate is the outside rate times the inside rate. It's the ad chain again: dollars to clicks, then clicks to revenue.

Here's a worked example: $y = (3x + 1)^2$ , at $x = 2$ .

The inside step is $u = 3x + 1$ . Each extra 1 in $x$ adds 3 to $u$ , so $du/dx = 3$ . At $x = 2$ , $u = 7$ .
The outside step is $y = u^2$ . The power rule gives $dy/du = 2u$ . At $u = 7$ , that's 14.
Multiply the two rates: $14 \times 3 = 42$ .

The same steps work for every $x$ at once:

\frac{dy}{dx} = 2u \cdot 3 = 6(3x + 1)

At $x = 2$ , that's $6 \times 7 = 42$ again. Python checks it with the nudge method. Here rule is the chain-rule answer. numeric moves $x$ a millionth up and down and measures the change.

f = lambda x: (3 * x + 1) ** 2
rule = lambda x: 6 * (3 * x + 1)
x = 2.0
numeric = (f(x + 1e-6) - f(x - 1e-6)) / 2e-6
print(rule(x), round(numeric, 4))

It runs right here in your browser, with nothing to install.

Both print 42.0.

Watch out: work out the outside slope at the inside value. Here $2u$ means $2 \times 7$ , not $2 \times 2$ .

Go deeper Why the rates multiply

Nudge $x$ by a tiny amount $h$ . The middle value $u$ moves by about $\frac{du}{dx} \cdot h$ . That move in $u$ is itself a nudge, so $y$ moves by about $\frac{dy}{du}$ times it:

\Delta y \approx \frac{dy}{du} \cdot \frac{du}{dx} \cdot h

Here $\Delta y$ (delta y) means "the change in y". Divide by $h$ to get the slope. As $h$ shrinks toward 0, the "about" becomes exact, and the chain rule is what's left.

Quick check

For y = (4x − 1)² at x = 1, what does this print?

inside_rate = 4          # u = 4x - 1
u = 4 * 1 - 1            # the inside value at x = 1
outside_rate = 2 * u     # slope of u squared
print(outside_rate * inside_rate)

Show the answer

At x = 1, u = 3. The outside rate is 2 × 3 = 6, the inside rate is 4, and the chain rule multiplies them: 24.

Models are chains too

A machine-learning model is a chain like the ad campaign. It turns an input into a prediction. Then a loss scores how wrong that prediction was: 0 is perfect, and bigger is worse. Training adjusts the model's weights, the numbers it can tune, to make the loss smaller.

The smallest possible model has one weight, $w$ . It multiplies the input $x$ by $w$ to make a prediction. The right answer is called the target, $y$ . The loss is the miss, squared: $(\text{prediction} - y)^2$ . Squaring stops the loss from going negative and punishes big misses more.

You walk this chain in two directions:

The forward pass goes from input to loss. It computes the values: prediction, then loss.
The backward pass goes from loss to weight. It computes the rates: first how the loss responds to the prediction, then how the prediction responds to $w$ .

Worked example

Worked example: a one-weight model

Input x = 2, target y = 3, weight w = 1. The model predicts 1 × 2 = 2. It misses by −1, so the loss is (−1)² = 1. Find how the loss changes as w changes, using the chain rule. Then check it with nudges.

x, y = 2.0, 3.0               # input and target
w = 1.0                       # the weight we can adjust

pred = w * x                  # forward: the prediction
loss = (pred - y) ** 2        # forward: the loss

dloss_dpred = 2 * (pred - y)  # backward: outside rate
dpred_dw = x                  # backward: inside rate
dloss_dw = dloss_dpred * dpred_dw

h = 1e-6
nudge = (((w + h) * x - y) ** 2 - ((w - h) * x - y) ** 2) / (2 * h)
print(pred, loss)
print(dloss_dpred, dpred_dw, dloss_dw, round(nudge, 4))

In the lesson you can change this and run it.

The outside rate is the slope of "miss squared", which is 2 × miss: 2 × (−1) = −2. The inside rate is how the prediction moves with w. Raising w by 1 raises w × x by x, which is 2. Multiply: −2 × 2 = −4, and the nudge agrees. The minus sign is the useful part: raising w lowers the loss. Sure enough, w = 1.5 predicts exactly 3, and the loss drops to 0.

Forward, then backward: one neuron

A neuron, the building block of a neural network, makes this chain one step longer. After multiplying by $w$ , it adds a second adjustable number, the bias $b$ . Then it passes the result through the sigmoid $\sigma$ from earlier lessons, which is the S-shaped curve that squashes any number into the range 0 to 1.

In machine learning (the subject of Phase 3), chains like this are drawn as a computation graph: a row of small steps, each taking inputs and producing an output. With input $x$ and target $y$ , the neuron's graph has three steps:

z = wx + b \quad\rightarrow\quad a = \sigma(z) \quad\rightarrow\quad L = (a - y)^2

In plain words: $z$ is the input times the weight, plus the bias. $a$ , the activation, is $z$ squashed by the sigmoid. $L$ is the loss: the miss between $a$ and the target, squared.

Try it with small numbers: input $x = 2$ , target $y = 1$ , weight $w = 0.5$ , bias $b = -0.5$ . The forward pass fills in the values from left to right:

$z = 0.5 \times 2 - 0.5 = 0.5$
$a = \sigma(0.5) \approx 0.6225$
$L = (0.6225 - 1)^2 \approx 0.1425$

Quick check

With an activation of 0.6225 and a target of 1, what is the squared loss, to 3 places?

print(round((0.6225 - 1) ** 2, 3))

0.623
0.378
1.0
0.143

Show the answer

0.143

The miss is about −0.38, and squaring it gives about 0.143. That's the forward pass finished.

The backward pass goes right to left. This chain has two numbers you can adjust, $w$ and $b$ , so you want a rate for each. These rates are written with a curly d, $\partial$ , said "partial". $\partial L / \partial w$ means how fast $L$ changes as $w$ changes, with everything else held still. Partial derivatives get their own lesson next. For now, read each one as a slope.

Each backward step reuses the one before it:

Loss to activation. As in the one-weight model, the slope is 2 × miss: $2 \times (0.6225 - 1) \approx -0.7551$ .
Activation to z. The last lesson gave the sigmoid's slope as $\sigma(z)(1 - \sigma(z))$ . Since $a = \sigma(z)$ , that's $a(1 - a)$ : $0.6225 \times 0.3775 \approx 0.2350$ . Multiply: $-0.7551 \times 0.2350 \approx -0.1774$ .
z to w. In $z = wx + b$ , raising $w$ by 1 raises $z$ by $x$ . So multiply by $x = 2$ : about $-0.3549$ .
z to b. Raising $b$ by 1 raises $z$ by 1. So multiply by 1: about $-0.1774$ .

Here are the same four steps in symbols:

\frac{\partial L}{\partial a} = 2(a - y) \qquad \frac{\partial L}{\partial z} = \frac{\partial L}{\partial a} \cdot \sigma(z)(1 - \sigma(z)) \qquad \frac{\partial L}{\partial w} = \frac{\partial L}{\partial z} \cdot x \qquad \frac{\partial L}{\partial b} = \frac{\partial L}{\partial z} \cdot 1

In plain words: start at the loss. At each step back, multiply by that step's own slope. When you reach a weight, you have its rate.

The stepper below runs the same numbers in Python.

Try it now

Step through: the forward lines fill in z, a and L; the backward lines multiply local slopes from the loss back to w and b.

Reading the result

The code prints 0.1425 -0.3549: the loss, then $\partial L / \partial w$ .

That slope is negative. So increasing $w$ would lower the loss. Training therefore nudges $w$ up a little. The next lessons turn that habit into a method called gradient descent.

Every weight in a network gets its slope the same way. Start at the loss, walk backward, and multiply the local slopes.

Quick check

For some weight, ∂L/∂w = 0.3. To lower the loss, which way should training nudge w?

Set w to 0.3
Leave it alone: 0.3 is close to 0
Up: a positive slope always means go up
Down: a positive slope means raising w raises the loss

Show the answer

Down: a positive slope means raising w raises the loss

A positive slope means the loss rises as w rises. So lowering w lowers the loss.

Why go backward?

A deep network has many weights feeding into the same later steps. Think of streams that join a river, which then flows to the sea. Once you know how the river reaches the sea, every stream that joins it can reuse that answer. Nobody has to trace the whole way again.

Going backward from the loss, each in-between rate (like $\partial L / \partial z$ ) is worked out once. Then every weight that feeds into it reuses it. So one backward pass costs about the same as one forward pass, no matter how many weights there are. This is backpropagation, often shortened to backprop.

It also explains the vanishing-gradient problem from the last lesson. In a deep stack of sigmoids, the backward pass multiplies many slopes of at most 0.25. Five of them multiply to about 0.001. So the slopes for the early layers shrink toward zero, and those layers barely learn.

For later: frameworks like PyTorch, which is the library Phase 4 uses, build the computation graph as your code runs and call loss.backward() to run the backward pass for you. You'll write one by hand in Phase 4 (and Baby AI already uses one).

Quick check

Five sigmoid layers each pass back their steepest possible slope, 0.25. What does this print?

slope = 1.0
for layer in range(5):
    slope = slope * 0.25
print(round(slope, 4))

0.001
1.25
0.05
0.25

Show the answer

0.001

The backward pass multiplies the slopes: 0.25 five times is about 0.001. Even at its best, the signal shrinks fast. That's the vanishing gradient.

Practice what you just read

2 code examples you can edit and run
a practice exercise that checks your code
a 5-question quiz
4 review cards for spaced repetition

Create a free account

Try a quiz question

What is the derivative of (2x + 5)³?

6(2x + 5)²
3(2x + 5)²
2(2x + 5)³
6x²

Answer it at the end of the lesson to see if you're right, and why.

Before this lesson

Derivative rules