The same network in PyTorch
Why this matters
Lessons 109 to 111 built a neural network from formulas and trained it, every line written by hand. Engineers write the same network in about ten lines of PyTorch. This lesson puts those ten lines beside your NumPy and shows they are the same thing, number for number: from the same starting weights they give the same loss, the same slopes and the same training run. Then it trains the network the way real models are trained, a few examples at a time, and reads the noisier loss curve that comes out.
- Build the two-layer network as nn.Sequential, and name the formula behind each layer and each call
- Load the NumPy weights in nn.Linear's (out, in) layout, and get the same loss and the same slopes
- Train it with optim.SGD beside the NumPy loop, step for step, and catch a missing zero_grad() by its numbers
- Train it on mini-batches from a DataLoader, and read the noisier loss curve that gives
The math: every line is a formula you know
Purpose. Read each line of a PyTorch training loop as the formula it computes.
Here is lesson 111's network and its whole training loop, written the way engineers write it:
model = nn.Sequential(nn.Linear(3, 4), nn.ReLU(), nn.Linear(4, 3)) opt = torch.optim.SGD(model.parameters(), lr=0.5) for step in range(300): opt.zero_grad() loss = F.cross_entropy(model(X), y) loss.backward() opt.step()
Nothing in it is new math. nn.Sequential runs the layers it's given in order, each one's output feeding the next, so model(X) is three functions composed (lesson 20). Each piece is a formula you've worked by hand:
| PyTorch | the formula it computes | from |
|---|---|---|
nn.Linear(3, 4) | Z₁ = XWᵀ + b, its weight W stored 4 × 3: your W₁ turned around | lesson 56 |
nn.ReLU() | A₁ = max(0, Z₁), entry by entry | lesson 110 |
nn.Linear(4, 3) | the scores, Z₂ = A₁Wᵀ + b, W stored 3 × 4 | lesson 110 |
F.cross_entropy(scores, y) | softmax of each row, then the mean of −log of each right class's chance | lesson 104, lesson 105 |
loss.backward() | the five lines, dZ₂ = (P − Y) ÷ n back to dW₁, each slope stored in its weight's .grad | lesson 111 |
opt.step() | w ← w − η·g, for every weight and bias | lesson 79 |
opt.zero_grad() | every .grad emptied, because backward() adds into it | lesson 82 |