Backpropagation

Rule 2: ReLU

On the way forward, ReLU kept positive numbers and turned the rest into 0. On the way back, it acts as a gate:

Here $[Z > 0]$ is a table of 1s and 0s. It has a 1 wherever $Z$ was positive and a 0 everywhere else. The symbol $\odot$ means multiply matching entries, one pair at a time. This is called element-wise multiplication.

In plain words: where ReLU's input was positive, the gradient passes straight through. Where its input was 0 or negative, the gradient is blocked and becomes 0. A gate that was shut on the way forward is shut on the way back too.

Say three values went into ReLU: 3, −2 and 0.5. The gradients arriving from above are 0.4, 0.9 and −0.6. The first and third inputs were positive, so their gradients pass: 0.4 and −0.6. The second was negative, so its gradient becomes 0.

import numpy as np
Z = np.array([3.0, -2.0, 0.5])     # what went into the ReLU on the way forward
dA = np.array([0.4, 0.9, -0.6])    # gradient arriving from the layer above
dZ = dA * (Z > 0)                  # pass it where Z was positive, block it elsewhere
print(dZ)

It prints [ 0.4 0. -0.6]. In code, (Z > 0) gives True or False for each entry, and NumPy treats those as 1 and 0.

Rule 3: the linear layer

The linear layer $Z = XW + \mathbf{b}$ has three things that need a gradient: the weights, the biases and the inputs.

$X^\top$ is $X$ transposed, with rows and columns swapped. You met transposes in Phase 2. $\sum_{\text{rows}}$ means add up down the rows, giving one total per column.

In plain words: a weight's gradient is its input times the gradient arriving at its neuron. Add that up over every example in the batch. This is the multiply rule from the last lesson: a weight's slope is the other input. Every example uses the same weight, so the shares add up. A bias is added straight on, so its slope is 1. Its gradient is the arriving gradient, added up over the examples. Last, each input feeds every neuron. So its gradient adds up, over the neurons, the arriving gradient times the weight that links them.

Here is a tiny case. There are two examples, two inputs and one neuron.

• Example 1 has inputs 2 and 3. Example 2 has inputs 1 and −1.
• The weights are 0.5 on input 1 and −1 on input 2.
• The gradient arriving at the neuron is 0.5 for example 1 and 2.0 for example 2.

Work out each gradient:

• Weight on input 1: example 1 gives 2 × 0.5 = 1. Example 2 gives 1 × 2.0 = 2. The total is 3.
• Weight on input 2: 3 × 0.5 = 1.5, plus −1 × 2.0 = −2. The total is −0.5.
• Bias: 0.5 + 2.0 = 2.5.
• Inputs of example 1: 0.5 × 0.5 = 0.25 and 0.5 × −1 = −0.5.
• Inputs of example 2: 2.0 × 0.5 = 1 and 2.0 × −1 = −2.

import numpy as np
X = np.array([[2.0, 3.0],          # example 1: inputs 2 and 3
              [1.0, -1.0]])        # example 2: inputs 1 and -1
W = np.array([[0.5],               # weight on input 1
              [-1.0]])             # weight on input 2
dZ = np.array([[0.5],              # gradient arriving for example 1
               [2.0]])             # gradient arriving for example 2

dW = X.T @ dZ                      # input times gradient, added over the examples
db = dZ.sum(0)                     # one bias, so add up the gradients
dX = dZ @ W.T                      # what each input passes further back
print("dW:", dW.ravel())
print("db:", db)
print("dX:", dX.tolist())

The code prints the same numbers: dW is 3 and −0.5, db is 2.5, and dX is [[0.25, −0.5], [1.0, −2.0]]. Matrix multiplication does all the "multiply, then add up over the examples" work in one line.

Why does $dX$ matter? In a deeper network, this layer's input is the output of the layer before. So $dX$ becomes the $dA$ for that earlier layer, and the backward pass carries on. In the very first layer, $X$ is the data. You can't change the data, so you don't need its gradient there.

The rules side by side

In the cross-entropy loss, $y_i$ is the true class of example $i$. So $P_{i, y_i}$ is the probability that example $i$ gave its true class.

One habit will save you hours. Every gradient has the same shape as the thing it's the gradient of. $dW$ is shaped like $W$, and $d\mathbf{b}$ like $\mathbf{b}$. That gives you a quick test for any formula you write.

Backprop by hand, checked

Now put the three rules to work on a real network. It's a 4 → 16 → 3 network on the Iris flowers from the forward pass lesson: 4 measurements in, 16 hidden neurons, and 3 species out. The code runs forward, then backward. Then it tests three weights with the nudge method, also called finite differences: nudge a weight up and down a tiny bit, and see how the loss moves.

import numpy as np
from sklearn.datasets import load_iris

data = load_iris()
X = (data.data - data.data.mean(0)) / data.data.std(0)
y = data.target
N = len(y)
Y = np.eye(3)[y]                                   # one-hot targets, shape (150, 3)

rng = np.random.default_rng(0)
W1, b1 = rng.normal(0, 0.5, (4, 16)), np.zeros(16)
W2, b2 = rng.normal(0, 0.5, (16, 3)), np.zeros(3)

def forward(W1, b1, W2, b2):
    Z1 = X @ W1 + b1
    A1 = np.maximum(0, Z1)
    Z2 = A1 @ W2 + b2
    P = np.exp(Z2 - Z2.max(1, keepdims=True))
    P /= P.sum(1, keepdims=True)
    loss = -np.mean(np.log(P[np.arange(N), y]))
    return loss, (Z1, A1, P)

loss, (Z1, A1, P) = forward(W1, b1, W2, b2)

# backward: the table above, from the loss to the first layer
dZ2 = (P - Y) / N
dW2, db2 = A1.T @ dZ2, dZ2.sum(0)
dA1 = dZ2 @ W2.T
dZ1 = dA1 * (Z1 > 0)
dW1, db1 = X.T @ dZ1, dZ1.sum(0)
print("loss:", round(loss, 4), "  gradient shapes:", dW1.shape, db1.shape, dW2.shape, db2.shape)

# finite-difference check on three individual weights
for name, W, i, grad in [("W1", W1, (0, 3), dW1), ("W1", W1, (2, 7), dW1), ("W2", W2, (5, 1), dW2)]:
    old = W[i]
    W[i] = old + 1e-5; up = forward(W1, b1, W2, b2)[0]
    W[i] = old - 1e-5; down = forward(W1, b1, W2, b2)[0]
    W[i] = old
    print(f"{name}{i}: backprop {grad[i]: .6f}   numerical {(up - down) / 2e-5: .6f}")

Read the backward lines from top to bottom. Each one is a rule from the table:

• dZ2 = (P - Y) / N is rule 1, softmax with cross-entropy.
• dW2, db2 and dA1 are rule 3, the linear layer, for the second layer.
• dZ1 = dA1 * (Z1 > 0) is rule 2, the ReLU gate.
• dW1 and db1 are rule 3 again, for the first layer.

The first line of output shows a loss of 2.8774 and four gradient shapes. Each one matches its parameter: (4, 16), (16,), (16, 3) and (3,). Then come the three nudge tests. Backprop and the nudge agree to six decimal places. For example, both give −0.392407 for the first weight.

Now use the gradients. Gradient descent subtracts a small step from every weight: 0.5 times its gradient. Then it runs forward and backward again. The code below repeats that over and over. It prints a progress line every 50 steps, from step 0 to step 300.

import numpy as np
from sklearn.datasets import load_iris

data = load_iris()
X = (data.data - data.data.mean(0)) / data.data.std(0)
y = data.target
N, Y = len(y), np.eye(3)[data.target]
rng = np.random.default_rng(0)
W1, b1 = rng.normal(0, 0.5, (4, 16)), np.zeros(16)
W2, b2 = rng.normal(0, 0.5, (16, 3)), np.zeros(3)

for step in range(301):
    Z1 = X @ W1 + b1; A1 = np.maximum(0, Z1); Z2 = A1 @ W2 + b2
    P = np.exp(Z2 - Z2.max(1, keepdims=True)); P /= P.sum(1, keepdims=True)
    loss = -np.mean(np.log(P[np.arange(N), y]))
    dZ2 = (P - Y) / N
    dW2, db2 = A1.T @ dZ2, dZ2.sum(0)
    dZ1 = (dZ2 @ W2.T) * (Z1 > 0)
    dW1, db1 = X.T @ dZ1, dZ1.sum(0)
    for param, grad in [(W1, dW1), (b1, db1), (W2, dW2), (b2, db2)]:
        param -= 0.5 * grad                           # gradient descent, in place
    if step % 50 == 0:
        print(f"step {step:>3}  loss {loss:.4f}  accuracy {np.mean(P.argmax(1) == y):.2f}")

The loss falls from 2.8774 to about 0.04. Accuracy climbs from 0.09 to 0.99. That accuracy is measured on the same 150 flowers the network trained on, so it's flattering. Even so, every step used nothing but the three rules above.

w, grad = 1.0, -0.2
w -= 0.5 * grad
print(w)

import math

def forward(w):                            # w: the weight from x1 into hidden neuron 1
    x1, x2 = 1.0, 2.0
    z1 = w * x1 + 0.25 * x2 + 0.0
    z2 = -1.0 * x1 + 0.5 * x2 + 0.5
    h1, h2 = max(0.0, z1), max(0.0, z2)
    o1, o2 = 1.0 * h1 - 1.0 * h2, -0.5 * h1 + 3.0 * h2
    e1, e2 = math.exp(o1), math.exp(o2)
    p1, p2 = e1 / (e1 + e2), e2 / (e1 + e2)
    return -math.log(p2), (x1, z1, p1, p2)       # the true class is 2

loss, (x1, z1, p1, p2) = forward(0.5)

# backward, one rule at a time
do1, do2 = p1 - 0, p2 - 1                  # softmax + cross-entropy: P - Y
dh1 = do1 * 1.0 + do2 * -0.5               # h1 feeds both logits, so the paths add
dh2 = do1 * -1.0 + do2 * 3.0
dz1 = dh1 * (z1 > 0)                       # ReLU: pass it on, since z1 was positive
dw = dz1 * x1                              # multiply rule: the other input, x1

nudge = (forward(0.5 + 1e-6)[0] - forward(0.5 - 1e-6)[0]) / 2e-6
print(round(loss, 3), round(do1, 3), round(do2, 3))
print(round(dh1, 3), round(dh2, 3))
print(round(dw, 3), round(nudge, 3))

Why this trick makes deep learning possible

There's a slower way to get gradients: the nudge method. Nudge one weight, rerun the whole network, and see how the loss moves. That costs at least one extra forward pass per weight. The check above used two per weight, one nudge up and one down.

The Iris network has 131 weights and biases. Nudging each one up and down, as the check did, takes 262 forward passes for one training step. A model with a billion weights would need billions of forward passes for a single step.

Backpropagation gets every gradient from one backward pass. That pass costs about twice a forward pass. So all the gradients together cost about as much as three forward passes. That holds whether the model has ten weights or a trillion. It's like a shop tracing a bad month back through its records once, instead of rerunning the month once for each member of staff.

The general method is called reverse-mode automatic differentiation. It's why training large networks is possible at all. The nudge method still has one job: checking gradient code, as you did above.

Where it goes wrong

• Shapes. If $dW$ isn't shaped like $W$, a transpose is missing.
• The bias gradient. Add it up over the batch. $d\mathbf{b}$ has one value per neuron, not one per example.
• Stale values. The ReLU gate needs $Z$ from this forward pass, not an older one.
• Scaling. If the loss is an average over $N$ examples, every gradient carries a $1/N$.

In the next lesson, you'll write a small program that does this bookkeeping for you. After that, PyTorch takes over. Either way, these are exactly the rules the software applies.