Training a small network
Why this matters
Lesson 110 ran a network forward with weights someone had already chosen. This lesson chooses them. Training is the loop of lesson 79 with lesson 110's network inside it: forward pass, loss, backward pass, update, repeated. The only new piece is the backward pass written for whole matrices at once, and even that is old: it's lesson 81's "multiply along the path" with the matrix products of lesson 56, and its first line is lesson 105's p − y. Each line can be checked two ways, by its shape and by a numerical slope, so nothing has to be taken on trust. At the end the network learns to read trail signs from 24 examples and is judged on 12 it never saw, and you'll be able to read the loss curve of a real training run and say whether its learning rate is right.
- Write the backward pass in matrix form, dZ₂ = P − Y down to dW₁ = Xᵀ dZ₁, and justify each line by its shape
- Run the full loop, forward, loss, backward, update, and check the slopes against a numerical estimate and PyTorch's loss.backward()
- Train a two-layer network on trail signs to high accuracy on held-back signs
- Read a training run: a learning rate too low, too high and about right, and training versus held-back accuracy
The math: the backward pass in matrix form
Purpose. Get the loss's slope for every weight in the network, for a whole batch, in five lines.
The forward pass of lesson 110 was Z₁ = XW₁ + b₁, A₁ = relu(Z₁), Z₂ = A₁W₂ + b₂, P = softmax(Z₂), and the loss averaged −log of each right-class chance. The backward pass runs the same steps in reverse, each line one link of the chain rule:
| line | formula | why |
|---|---|---|
| 1 | dZ₂ = (P − Y) ÷ n | lesson 105's slope, averaged over the batch |
| 2 | dW₂ = A₁ᵀ dZ₂, db₂ = column sums of dZ₂ | Z₂ = A₁W₂: each weight's slope sums input × push over the batch |
| 3 | dA₁ = dZ₂ W₂ᵀ | the push travels back through the same weights, turned around |
| 4 | dZ₁ = dA₁, set to 0 where Z₁ ≤ 0 | relu passes the push only where the unit was on |
| 5 | dW₁ = Xᵀ dZ₁, db₁ = column sums of dZ₁ | the same rule as line 2, one layer down |
Every "d" matrix has exactly the shape of the matrix it's the slope of: that alone fixes where each transpose goes.