Lesson 110 ran a network forward with weights someone had already chosen. This lesson chooses them. Training is the loop of lesson 79 with lesson 110's network inside it: forward pass, loss, backward pass, update, repeated. The only new piece is the backward pass written for whole matrices at once, and even that is old: it's lesson 81's "multiply along the path" with the matrix products of lesson 56, and its first line is lesson 105's p − y. Each line can be checked two ways, by its shape and by a numerical slope, so nothing has to be taken on trust. At the end the network learns to read trail signs from 24 examples and is judged on 12 it never saw, and you'll be able to read the loss curve of a real training run and say whether its learning rate is right.
What you'll be able to do
Write the backward pass in matrix form, dZ₂ = P − Y down to dW₁ = Xᵀ dZ₁, and justify each line by its shape
Run the full loop, forward, loss, backward, update, and check the slopes against a numerical estimate and PyTorch's loss.backward()
Train a two-layer network on trail signs to high accuracy on held-back signs
Read a training run: a learning rate too low, too high and about right, and training versus held-back accuracy
Quick check
Have a guess before you start. In lesson 105, softmax followed by cross-entropy had a very simple slope with respect to the scores. What was it?
1 − p
y − p²
log p
p − y
Show the answer
p − y
That one line, done for every row of a batch, is the first line of the backward pass: dZ₂ = P − Y.
The math: the backward pass in matrix form
Purpose. Get the loss's slope for every weight in the network, for a whole batch, in five lines.
The forward pass of lesson 110 was Z₁ = XW₁ + b₁, A₁ = relu(Z₁), Z₂ = A₁W₂ + b₂, P = softmax(Z₂), and the loss averaged −log of each right-class chance. The backward pass runs the same steps in reverse, each line one link of the chain rule:
Z₂ = A₁W₂: each weight's slope sums input × push over the batch
3
dA₁ = dZ₂ W₂ᵀ
the push travels back through the same weights, turned around
4
dZ₁ = dA₁, set to 0 where Z₁ ≤ 0
relu passes the push only where the unit was on
5
dW₁ = Xᵀ dZ₁, db₁ = column sums of dZ₁
the same rule as line 2, one layer down
Every "d" matrix has exactly the shape of the matrix it's the slope of: that alone fixes where each transpose goes.
Quick check
From lesson 81: a value feeds two later steps, which send back slopes 0.3 and −0.1. What is its slope?
0.2
−0.03
0.4
0.3
Show the answer
0.2
Across paths, slopes add: 0.3 + (−0.1) = 0.2. The matrix product dZ₂ W₂ᵀ does exactly this adding for every hidden unit.
The picture. Here are the five lines on a network small enough to draw: one sign with three features, two hidden units and two classes. Each column of circles is one matrix, entry by entry, with values inside and slopes (∂) below. Press Next to send the slopes back one line at a time, and watch the bottom hidden unit.
Try it now
Shapes decide the transposes. Now a whole batch: 8 signs, 2 features, 4 hidden units and 3 classes. This picture draws the network as boxes, with each matrix's shape on the arrow it travels along: forward along the top, back along the bottom. Each weight's slope lands on its own layer's box. Watch it come out the same shape as the weight.
In plain words: start from "predicted minus true"; each weight matrix's slope is its layer's input, transposed, times the push arriving at its output; the push passes back through a layer by its weights turned around, and through relu only where the unit was on.
One entry by hand. dW₂'s entry (j, c), hidden unit j to class c, is Σᵢ A₁[i, j] × dZ₂[i, c]: the sum over the batch of "how active unit j was" times "how much class c's score should move". In the picture the sum had one term. Over a batch, a unit that was off for every example gets a row of zeros, like the picture's bottom unit. ⊙ means multiply entry by entry, * in NumPy.
Quick check
W₁ is 2 × 4. What shape must dW₁ be, and which product gives it?
2 × 4, from Xᵀ dZ₁
2 × 4, from X dZ₁ᵀ
4 × 2, from dZ₁ᵀ X
n × 4, from dZ₁ alone
Show the answer
2 × 4, from Xᵀ dZ₁
A slope has its weight's shape. Xᵀ dZ₁ is (2 × n)(n × 4) = 2 × 4, summing over the batch.
In your own words
In your own words: why does every slope matrix have exactly the shape of its weight matrix?
Write a sentence or two, then compare it with a good answer in the lesson.
In Python: the full loop, checked
Purpose. Train the network, and prove the backward pass right before trusting it.
It runs right here in your browser, with nothing to install.
Thirty-six signs are made around three typical feature profiles, one per class; 24 train the network and 12 are held back. train runs the loop 300 times from the same starting weights, keeping the loss every 50 steps, then prints that list and the training and held-back accuracy. At rate 0.5 the loss falls from 1.044 to 0.006 and every sign is read correctly. The other two rates go wrong in two different ways, which the last part of the lesson reads.
Code flow. Here is the same loop on the math part's one sign, run three times with no biases. Step through it: the first pass fills in every number from the picture, the last line moves both weight matrices, and the right class's chance, P's second entry, climbs on each pass. With one sign there is only one row, so the softmax divides by a plain .sum(), and with scores this small it can skip subtracting the largest score first.
Quick check
What does this print? (The first backward line for a batch of 2.)
import numpy as np
P = np.array([[0.7, 0.2, 0.1], [0.1, 0.6, 0.3]])
Y = np.array([[1.0, 0.0, 0.0], [0.0, 0.0, 1.0]])
print(((P - Y) / 2).round(2).tolist())
[[-0.3, 0.2, 0.1], [0.1, 0.6, -0.7]]
[[0.35, 0.1, 0.05], [0.05, 0.3, 0.15]]
[[-0.15, 0.1, 0.05], [0.05, 0.3, -0.35]]
[[0.15, -0.1, -0.05], [-0.05, -0.3, 0.35]]
Show the answer
[[-0.15, 0.1, 0.05], [0.05, 0.3, -0.35]]
P − Y pushes each right class down by its missing chance and every other class by its stolen one; ÷ 2 averages over the batch.
A named mistake: a backward pass that's almost right. A missing transpose usually crashes, but a missing "÷ n", a forgotten relu mask or a sign error runs and trains, just badly. The guard is a gradient check: nudge one weight by a tiny h both ways, measure the loss's change, and compare with the backward pass's slope (lesson 81). At the starting weights of the runs above, the slope for W₁[0, 0] comes out 0.02854243 both ways.
Worked example
Worked example: a gradient check on one weight
Compare the backward pass's slope for W₁[0, 0] with a central difference, at the starting weights.
Both numbers are 0.02854243. The backward pass got its slope from five matrix lines, the central difference by brute force from two forward passes. They share no code past the forward pass, so their agreement is strong evidence the backward pass is right; libraries run checks like this on every new layer type. Copies matter: nudging W1 itself would move the point every later slope is measured at (lesson 78).
dW₂[0, 0] = 2 × (−0.5) + 1 × 0.1 = −0.9: summed over both examples. Unit 1 was off for the first example, so only the second counts: 3 × 0.1 = 0.3.
Drill
Write the backward pass
Fill in backward so it returns dW1, db1, dW2 and db2 for the network, using the five lines from the table. The test checks every shape and compares one slope with a gradient check.
import numpy as np
defforward(X, W1, b1, W2, b2):
Z1 = X @ W1 + b1
A1 = np.maximum(Z1, 0)
Z2 = A1 @ W2 + b2
E = np.exp(Z2 - Z2.max(axis=1, keepdims=True))
return Z1, A1, E / E.sum(axis=1, keepdims=True)
defbackward(X, Y, W1, b1, W2, b2):
Z1, A1, P = forward(X, W1, b1, W2, b2)
n = len(X)
dW1 = np.zeros_like(W1)
db1 = np.zeros_like(b1)
dW2 = np.zeros_like(W2)
db2 = np.zeros_like(b2)
return dW1, db1, dW2, db2
r = np.random.default_rng(3)
X = r.normal(0, 1, (6, 2))
Y = np.zeros((6, 3))
for i inrange(6):
Y[i, i % 3] = 1.0
W1, b1, W2, b2 = r.normal(0, 0.5, (2, 4)), np.zeros(4), r.normal(0, 0.5, (4, 3)), np.zeros(3)
dW1, db1, dW2, db2 = backward(X, Y, W1, b1, W2, b2)
print(dW1.shape, db1.shape, dW2.shape, db2.shape)
In the lesson, your code is checked as you write it.
Plot the three runs. A loss curve is the first thing anyone checks in a training run. This cell types in the losses the first cell printed and draws one line per learning rate: label= names each line, and plt.legend() shows the names.
On your plot, which run is still falling steadily at step 250, so that more steps would help it most?
rate 0.5
None: all three have flattened
rate 0.01
rate 5
Show the answer
rate 0.01
Rate 0.01 drops about 0.03 every 50 steps and hasn't flattened: it needs more steps or a bigger rate. That's what too low looks like.
In your own words
The gradient check agreed to eight decimal places. What kinds of mistakes would it have caught, and is there anything it couldn't catch?
Write a sentence or two, then compare it with a good answer in the lesson.
In AI and ML: reading a training run
Purpose. Diagnose a training run from its loss curve and its two accuracies.
The loop you just wrote is the whole of training, so it lights almost the whole map. Only the parts for models of words (vectors, attention, sampling) and for the size of a run (memory, cost) stay dark.
Your plot holds three runs of the same network, data and starting weights. Too low (0.01) creeps: still falling at the end, it needs more steps or a bigger rate. About right (0.5) drops fast, then flattens near 0. Too high (5) jumps past the bottom on the first steps, and here something worse happens: every hidden unit switched off for every sign, relu blocked every slope (lesson 109's dead neurons), and the loss froze at 1.099, which is log 3, the loss of guessing a third for every class. A loss stuck at exactly log(classes) is the signature of a network that has stopped learning.
Quick check
A run's training loss stays at 0.693 from step 20 onward, on a two-class problem. What's the likely story?
The learning rate is too small
It has found the best possible weights
It stopped: 0.693 is log 2, a 50/50 guess
It overfits
Show the answer
It stopped: 0.693 is log 2, a 50/50 guess
log(number of classes) is the loss of equal guesses. Frozen there usually means dead units, often from a learning rate that was too high.
Live training. The playground runs this same loop, forward, loss, backward, update, many times a second on points of two kinds, and draws the training and test loss as it learns. Its hidden units use tanh, an S-shaped bend like the sigmoid, unless you pick ReLU. It starts at a learning rate of 0.01. With random weights it guesses about 50/50, so its loss starts near log 2; what matters is whether it moves.
Train versus held back. In the first cell both accuracies matched, because the three kinds of sign are well separated. On harder data, training accuracy rising while held-back accuracy stalls or falls is overfitting (lesson 101). Leave a playground run going at 0.5 to see it start: the training loss keeps falling while the test loss levels off, then starts to climb.
In PyTorch. Real training code almost never writes the five lines. As in lesson 81, loss.backward() works out every slope by the chain rule and stores each in the .grad of every tensor made with requires_grad=True. This cell does the five lines by hand on the first cell's starting network, then lets PyTorch do them.
import numpy as np
import torch
import torch.nn.functional as F
rng = np.random.default_rng(7)
centers = np.array([[0.9, 0.2, 0.1], [0.2, 0.9, 0.3], [0.1, 0.3, 0.9]])
X = np.zeros((24, 3))
y = np.zeros(24, dtype=int)
Y = np.zeros((24, 3))
for i inrange(24):
X[i] = centers[i % 3] + rng.normal(0, 0.15, 3)
y[i] = i % 3
Y[i, i % 3] = 1.0
r = np.random.default_rng(1)
W1, b1 = r.normal(0, 0.5, (3, 4)), np.zeros(4)
W2, b2 = r.normal(0, 0.5, (4, 3)), np.zeros(3)
# the forward pass and the five lines, by hand
Z1 = X @ W1 + b1
A1 = np.maximum(Z1, 0)
Z2 = A1 @ W2 + b2
E = np.exp(Z2 - Z2.max(axis=1, keepdims=True))
dZ2 = (E / E.sum(axis=1, keepdims=True) - Y) / 24
dW2, db2 = A1.T @ dZ2, dZ2.sum(axis=0)
dZ1 = (dZ2 @ W2.T) * (Z1 > 0)
dW1, db1 = X.T @ dZ1, dZ1.sum(axis=0)
# the same network in PyTorch, and its backward pass
tW1, tb1 = torch.tensor(W1, requires_grad=True), torch.tensor(b1, requires_grad=True)
tW2, tb2 = torch.tensor(W2, requires_grad=True), torch.tensor(b2, requires_grad=True)
scores = torch.relu(torch.tensor(X) @ tW1 + tb1) @ tW2 + tb2
loss = F.cross_entropy(scores, torch.tensor(y)) # softmax and the average -log p in one call
loss.backward()
print("loss", round(loss.item(), 3), "slope of W1[0, 0]", round(tW1.grad[0, 0].item(), 6))
for name, mine, auto in [("dW1", dW1, tW1), ("db1", db1, tb1), ("dW2", dW2, tW2), ("db2", db2, tb2)]:
print(name, "largest gap:", round(float(np.abs(mine - auto.grad.numpy()).max()), 6))
It prints the loss 1.044, the first cell's loss at step 0, and W₁[0, 0]'s slope 0.028542, the gradient check's number to six places. Every largest gap is 0.0 to six places: loss.backward() is your five lines.
Drill
Train to 90% on held-back signs
Use your forward and backward lines in a training loop on the sign data (seed 7 for the data, seed 1 for the weights). Choose a learning rate and a number of steps so that test_acc, the accuracy on the 12 held-back signs, is at least 0.9, and record final_loss, the training loss after the last step.
31Formulas with fractions: clear them firstFreeClearing denominators, cross-multiplying, a letter in the bottom, and reciprocal formulas like 1/R = 1/R₁ + 1/R₂.35 min
37Inverse functions, lines and scalingFreeUndoing a function by making x the subject, domains that allow it, line forms, and unscaling predictions.40 min
38Sequences and sums: find n, a, d and rFreeEvery letter of the arithmetic and geometric formulas as the subject, and an average's window from its decay.40 min
40Review: rearrange anything (algebra)FreeBroken rearrangements to fix, mixed practice from the whole algebra block, a strategy chart, and its formula sheet.35 min
41Coordinates and distanceFreePoints as (x, y), Pythagoras, distance in any number of coordinates, and nearest neighbors.42 min
42Angles, degrees and radiansFreeMeasuring a turn, converting units, arcs, and the angle between two arrows.36 min
43Sine and cosineFreeAcross and up: right triangles, the unit circle, and directions and rotations.42 min
44Waves: height, speed and shiftFreeAmplitude, period and phase, sampling a wave, and position codes in transformers.37 min
68Vector equations: rearranging with arrowsFreeSolving for a vector or a scale, recovering a gradient from two weight snapshots, and an angle from a dot product.40 min
87Counting: choices, orders and groupsFreeThe counting principle, factorials, ordered picks and groups, and why attention's cost grows with the square of the length.43 min
88Probability: from counts to chancesFreeEqually likely outcomes, the complement, distributions, seeded simulation, and a next-word model.43 min
89Combining events: and, or, independentFreeThe or rule, independence, probability trees, drawing with and without replacement, and chains of steps.42 min
100Maximum likelihoodFreeLikelihoods as products, log-likelihoods as sums, grid search, underflow, and the negative log-likelihood loss.45 min
101Linear regression: the best lineFreeResiduals, least squares by formula, three roads to one line, the matrix form, and train versus test error.45 min
109One neuron, start to finishFreeWeighted sum, activation, loss, slopes and update, by hand and in code, and what one straight line can and can't split.45 min
111Training a small networkFreeThe backward pass in matrix form, gradient checks, a full training loop, and reading loss curves.45 min
112The same network in PyTorchFreenn.Sequential with the NumPy weights loaded, the same loss and slopes, then SGD and mini-batches.45 min
113Attention as matrix mathFreeScaled dot-product scores, the causal mask, a row-by-row softmax, and blending value vectors.45 min
114Residual connections and layer normFreeAdding the input back and why its slope keeps learning alive, layer norm by hand, and add and norm.40 min
115A transformer block, start to finishFreeA sentence through embeddings, attention, add and norm, a small network and next-word scores, every shape named.45 min
117Final test: transpose any formulaFreeSlips to fix from every topic, mixed practice of every kind of rearranging, the strategy chart and the full sheet.45 min
Learn the math behind AI from arithmetic up: numbers, algebra, vectors and matrices, calculus, probability and statistics, each shown as numbers moving and checked in Python, ending with a network you build and train from the formulas.
See the course →
Never trained an AI? Teach a small one to write, right in your browser. It takes about 20 seconds and needs no account. Teach it to write