Lessons 109 to 111 built a neural network from formulas and trained it, every line written by hand. Engineers write the same network in about ten lines of PyTorch. This lesson puts those ten lines beside your NumPy and shows they are the same thing, number for number: from the same starting weights they give the same loss, the same slopes and the same training run. Then it trains the network the way real models are trained, a few examples at a time, and reads the noisier loss curve that comes out.
What you'll be able to do
Build the two-layer network as nn.Sequential, and name the formula behind each layer and each call
Load the NumPy weights in nn.Linear's (out, in) layout, and get the same loss and the same slopes
Train it with optim.SGD beside the NumPy loop, step for step, and catch a missing zero_grad() by its numbers
Train it on mini-batches from a DataLoader, and read the noisier loss curve that gives
Quick check
Have a guess before you start. Your NumPy network and a PyTorch copy of it start from the same weights and train on the same 24 signs at rate 0.5. After 200 steps, how do their losses compare?
Equal in every one of their digits
The same, to four decimal places
Far apart: PyTorch starts from its own random weights
PyTorch's is far lower: its optimizer is smarter
Show the answer
The same, to four decimal places
The math is the same, so the numbers are too: 0.0087 for each at step 200, as the Python part prints. They part only around the sixth or seventh digit, because PyTorch keeps its numbers in float32 and NumPy in float64.
The math: every line is a formula you know
Purpose. Read each line of a PyTorch training loop as the formula it computes.
Here is lesson 111's network and its whole training loop, written the way engineers write it:
model = nn.Sequential(nn.Linear(3, 4), nn.ReLU(), nn.Linear(4, 3))
opt = torch.optim.SGD(model.parameters(), lr=0.5)
for step inrange(300):
opt.zero_grad()
loss = F.cross_entropy(model(X), y)
loss.backward()
opt.step()
Nothing in it is new math. nn.Sequential runs the layers it's given in order, each one's output feeding the next, so model(X) is three functions composed (lesson 20). Each piece is a formula you've worked by hand:
PyTorch
the formula it computes
from
nn.Linear(3, 4)
Z₁ = XWᵀ + b, its weight W stored 4 × 3: your W₁ turned around
From lesson 56: a layer made with nn.Linear(25, 3) stores its weight as 3 × 25. A batch X has 10 rows. Which line gives the 10 × 3 scores?
X @ layer.weight + layer.bias
layer.weight @ X + layer.bias
X.T @ layer.weight + layer.bias
X @ layer.weight.T + layer.bias
Show the answer
X @ layer.weight.T + layer.bias
The weight is kept one row per output, 3 × 25, so it's turned around to 25 × 3 for the product: (10 × 25)(25 × 3) gives 10 × 3. That's exactly what layer(X) computes.
The picture. Here is that network on lesson 111's 24 training signs: 3 features in, 4 hidden units, 3 kinds of sign out. The boxes are its layers, and each arrow carries a matrix, labeled with its shape: forward along the top, the slopes back along the bottom. The Python tab lights your NumPy lines as you step. Open the PyTorch tab and the same steps light model(X), loss.backward() and opt.step(): one call for each stretch of the picture.
Try it now
Then set hidden units h to 16 and press Use these numbers: the hidden shapes widen to 24 × 16, and in the PyTorch tab the layers become Linear(3, 16) and Linear(16, 3).
Symbols and formula. The two lines that do the work:
Z=XW⊤+bw←w−η∂w∂L
In plain words: a linear layer is your matrix product plus a bias, with its weights kept one row per output and turned around for the product. The optimizer's step moves every weight a little against its own slope.
Counting what training changes.nn.Linear(d, h) holds d × h weights and h biases. The network's two layers hold 3 × 4 + 4 = 16 and 4 × 3 + 3 = 15 numbers, 31 in all, and nn.ReLU holds none: it has nothing to learn. model.parameters() hands those 31 to the optimizer, which is how opt.step() knows what to move.
Quick check
How many numbers does nn.Linear(3, 4) hold for training to change?
4
7
16
12
Show the answer
16
12 weights, 4 rows of 3, one for each pairing of an input with an output, plus 4 biases, one for each output: 16.
Quick check
model(X) gives the raw scores. Which line gives the same loss as your NumPy code?
F.softmax(model(X), dim=1).mean()
-torch.log(model(X)).mean()
F.cross_entropy(F.softmax(model(X), dim=1), y)
F.cross_entropy(model(X), y)
Show the answer
F.cross_entropy(model(X), y)
F.cross_entropy does the softmax itself, then averages −log of each right class's chance. Give it the raw scores and the class numbers.
In your own words
In your own words: why is model[0].weight 4 × 3 when your NumPy W₁ was 3 × 4, and what does that mean for loading weights?
Write a sentence or two, then compare it with a good answer in the lesson.
In Python: two loops, one run
Purpose. Build the network in PyTorch, load your NumPy weights into it, and train the two side by side.
Start with the network alone. print(model) lists its layers, and model.named_parameters() walks through every tensor that training can change, with its name:
import torch
import torch.nn as nn
model = nn.Sequential(nn.Linear(3, 4), nn.ReLU(), nn.Linear(4, 3))
print(model)
total = 0for name, p in model.named_parameters():
print(name, tuple(p.shape))
total = total + p.numel()
print(total)
The names count the layers from 0: 0.weight is the first layer's 4 × 3 weight and 2.bias the last layer's 3 biases. Layer 1, the ReLU, holds nothing, so no name starts with 1. p.numel() counts a tensor's numbers, and the total is the math part's 31. model[0] and model[2] reach the two layers themselves.
Quick check
What does this print?
import torch.nn as nn
layer = nn.Linear(5, 2)
print(layer.weight.shape, layer.bias.shape)
torch.Size([5, 2]) torch.Size([5])
torch.Size([2, 5]) torch.Size([2])
torch.Size([5, 2]) torch.Size([2])
torch.Size([2, 5]) torch.Size([5])
Show the answer
torch.Size([2, 5]) torch.Size([2])
5 inputs and 2 outputs: the weight is kept one row per output, 2 × 5, with one bias per output, 2.
Code flow. Now the loop's four lines, small enough to watch: lesson 111's one sign and its network with no biases, run for three steps. The weights are plain tensors here, laid out as in your NumPy, so the boxes match that lesson's code flow, and optim.SGD takes a list of them just as it takes model.parameters(). Step through and watch W1 and W2 change on each opt.step(), and the loss fall.
Loading your weights. A new nn.Linear starts with random weights and random biases, so a fresh model can't match your NumPy run. Copy your starting numbers in instead, under torch.no_grad() as in lesson 50: W1.T for the first layer's weight, because it's kept one row per output, and the biases too. The 24 signs are the same as before, made in two lines: centers[y] picks each sign's profile by its kind, as E[ids] picked rows in lesson 65, and np.eye(3)[y] gives the one-hot rows. Then one loop runs both: each pass takes a NumPy step, your twelve lines, and a PyTorch step, four lines, from the same start on the same signs.
import numpy as np
import torch
import torch.nn as nn
import torch.nn.functional as F
rng = np.random.default_rng(7)
centers = np.array([[0.9, 0.2, 0.1], [0.2, 0.9, 0.3], [0.1, 0.3, 0.9]])
y = np.arange(24) % 3
X = centers[y] + rng.normal(0, 0.15, (24, 3))
Y = np.eye(3)[y]
r = np.random.default_rng(1)
W1, b1 = r.normal(0, 0.5, (3, 4)), np.zeros(4)
W2, b2 = r.normal(0, 0.5, (4, 3)), np.zeros(3)
model = nn.Sequential(nn.Linear(3, 4), nn.ReLU(), nn.Linear(4, 3))
with torch.no_grad():
model[0].weight.copy_(torch.tensor(W1.T))
model[0].bias.copy_(torch.tensor(b1))
model[2].weight.copy_(torch.tensor(W2.T))
model[2].bias.copy_(torch.tensor(b2))
opt = torch.optim.SGD(model.parameters(), lr=0.5)
Xt, yt = torch.tensor(X, dtype=torch.float32), torch.tensor(y)
for step inrange(201):
Z1 = X @ W1 + b1 # NumPy: the forward pass and the loss
A1 = np.maximum(Z1, 0)
Z2 = A1 @ W2 + b2
E = np.exp(Z2 - Z2.max(axis=1, keepdims=True))
P = E / E.sum(axis=1, keepdims=True)
np_loss = -np.mean(np.sum(Y * np.log(P), axis=1))
dZ2 = (P - Y) / 24# NumPy: the five lines, then the update
dW2, db2 = A1.T @ dZ2, dZ2.sum(axis=0)
dZ1 = (dZ2 @ W2.T) * (Z1 > 0)
dW1, db1 = X.T @ dZ1, dZ1.sum(axis=0)
W1, b1, W2, b2 = W1 - 0.5 * dW1, b1 - 0.5 * db1, W2 - 0.5 * dW2, b2 - 0.5 * db2
opt.zero_grad() # PyTorch: the same step
loss = F.cross_entropy(model(Xt), yt)
loss.backward()
opt.step()
if step % 100 == 0:
print(step, round(float(np_loss), 4), round(loss.item(), 4))
The two columns agree at every step printed: 1.044 at the start, 0.0241 at step 100 and 0.0087 at step 200. Twelve lines of NumPy and four of PyTorch make the same training run. dtype=torch.float32 makes the signs PyTorch's usual kind of number, to match the layers, and torch.tensor(y) keeps the kinds as whole numbers, the class numbers F.cross_entropy wants. The runs aren't equal to the last digit, though. PyTorch keeps 24 binary digits a number (float32) where NumPy keeps 53 (float64), as in lesson 12, so printed to six places they part in the last digit or two.
Worked example
Worked example: the same slopes, number for number
Before any training, compare every slope loss.backward() finds with your five lines, at the starting weights.
PyTorch's first-layer slope is (4, 3), the shape of the weight it keeps, and your dW₁ is (3, 4). Turned around, they agree. The slope of the very first weight is 0.028542 both ways, the gradient check's number from lesson 111, and the largest gap anywhere, for weights and biases alike, is 0.0 to six places. bias.zero_() sets a bias to 0 in place, the start your NumPy biases had. One call to backward() gave all 31 slopes.
A named mistake: forgetting opt.zero_grad().backward() adds each slope into .grad instead of replacing it (lesson 82). Leave out the clearing line and every step uses the sum of all the slopes so far. This cell runs the PyTorch loop twice from the same start, with and without opt.zero_grad(), and prints the loss, the largest stored slope in the last layer and its largest weight:
import numpy as np
import torch
import torch.nn as nn
import torch.nn.functional as F
rng = np.random.default_rng(7)
centers = np.array([[0.9, 0.2, 0.1], [0.2, 0.9, 0.3], [0.1, 0.3, 0.9]])
y = np.arange(24) % 3
Xt = torch.tensor(centers[y] + rng.normal(0, 0.15, (24, 3)), dtype=torch.float32)
yt = torch.tensor(y)
r = np.random.default_rng(1)
W1, W2 = r.normal(0, 0.5, (3, 4)), r.normal(0, 0.5, (4, 3))
defrun(clear):
model = nn.Sequential(nn.Linear(3, 4), nn.ReLU(), nn.Linear(4, 3))
with torch.no_grad():
model[0].weight.copy_(torch.tensor(W1.T))
model[2].weight.copy_(torch.tensor(W2.T))
model[0].bias.zero_()
model[2].bias.zero_()
opt = torch.optim.SGD(model.parameters(), lr=0.5)
for step inrange(201):
if clear:
opt.zero_grad()
loss = F.cross_entropy(model(Xt), yt)
loss.backward()
if step in [1, 2, 200]:
print(clear, step, round(loss.item(), 4), round(model[2].weight.grad.abs().max().item(), 3), round(model[2].weight.abs().max().item(), 2))
opt.step()
run(True)
run(False)
At step 1 the losses match, 1.0148, but the forgetful run already stores 0.118 where the clean one stores 0.068: this step's slope plus the last one's. From step 2 the losses part, 0.958 against your 0.9854. By step 200 the new slopes have almost vanished, yet the stored sum sits at 0.677, so every step pushes the weights another 0.5 × 0.677 ≈ 0.34 the same way, for as long as it runs: the largest weight is 66.01 against 2.21. The loss reads 0.0, which looks perfect and isn't. The network is sure of every answer for no good reason, and on noisier data a runaway like this throws the loss up instead. Comparing with a run you trust is how you catch it.
Quick check
What does this print?
import torch
w = torch.tensor([1.0, 2.0], requires_grad=True)
opt = torch.optim.SGD([w], lr=0.5)
loss = (w * w).sum()
loss.backward()
opt.step()
print(w.tolist())
[0.5, 1.5]
[2.0, 4.0]
[1.0, 2.0]
[0.0, 0.0]
Show the answer
[0.0, 0.0]
backward() puts the slope of w₁² + w₂², which is 2w = (2, 4), into w.grad. step() does w − 0.5 × (2, 4) = (1 − 1, 2 − 2) = (0, 0): straight to the bottom of this bowl.
Drill
Match the NumPy run
This PyTorch run should match your NumPy one: a starting loss of 1.044 and 0.0087 at step 200, to four places. The starter has three slips. Fix them so that start and loss200 match.
import numpy as np
import torch
import torch.nn as nn
import torch.nn.functional as F
rng = np.random.default_rng(7)
centers = np.array([[0.9, 0.2, 0.1], [0.2, 0.9, 0.3], [0.1, 0.3, 0.9]])
y = np.arange(24) % 3
X = centers[y] + rng.normal(0, 0.15, (24, 3))
r = np.random.default_rng(1)
W1, W2 = r.normal(0, 0.5, (3, 4)), r.normal(0, 0.5, (4, 3))
model = nn.Sequential(nn.Linear(3, 4), nn.ReLU(), nn.Linear(4, 3))
with torch.no_grad():
model[0].weight.copy_(torch.tensor(W1))
model[2].weight.copy_(torch.tensor(W2.T))
opt = torch.optim.SGD(model.parameters(), lr=0.5)
Xt, yt = torch.tensor(X, dtype=torch.float32), torch.tensor(y)
losses = []
for step inrange(201):
loss = F.cross_entropy(model(Xt), yt)
losses.append(round(loss.item(), 4))
loss.backward()
opt.step()
start, loss200 = losses[0], losses[200]
print(start, loss200)
In the lesson, your code is checked as you write it.
In your own words
Your PyTorch run matched NumPy's to four places. Why is a match like that worth more than a PyTorch run that simply trains well?
Write a sentence or two, then compare it with a good answer in the lesson.
In AI and ML: mini-batches, the way real models train
Purpose. Train on a few signs at a time, served by a DataLoader, and read the noisier loss curve that comes out.
Mini-batches. Your loops used all 24 signs for every step. A real dataset holds thousands or millions of examples, far too many to read for every small step. So training takes a mini-batch, a few examples, works out the loss and the slopes on those alone, steps, and moves on to the next few. One pass through all the data is an epoch. PyTorch's DataLoader deals the batches: wrap the tensors in a TensorDataset, which pairs each sign with its kind, and a loop over the loader gets one batch of signs and their kinds at a time.
Quick check
The 24 signs go into a DataLoader with batch_size=8. How many times does opt.step() run in one epoch?
24
1
3
8
Show the answer
3
Each batch of 8 gets one step, and 24 signs make 3 batches.
Here is the network trained that way: 100 epochs of 3 steps, from the same starting weights. Each step records two losses: the batch's own, the one the step works on, and the loss on all 24 signs, measured under torch.no_grad() so it stays off the tape that backward() reads (lesson 82). The 12 held-back signs are scored at the end. Real training code passes shuffle=True to deal the signs in a new order every epoch; it's left off here, so the cell prints the same numbers in your browser and on a real install.
import numpy as np
import torch
import torch.nn as nn
import torch.nn.functional as F
from torch.utils.data import DataLoader, TensorDataset
rng = np.random.default_rng(7)
centers = np.array([[0.9, 0.2, 0.1], [0.2, 0.9, 0.3], [0.1, 0.3, 0.9]])
y = np.arange(36) % 3
X = centers[y] + rng.normal(0, 0.15, (36, 3))
r = np.random.default_rng(1)
W1, W2 = r.normal(0, 0.5, (3, 4)), r.normal(0, 0.5, (4, 3))
model = nn.Sequential(nn.Linear(3, 4), nn.ReLU(), nn.Linear(4, 3))
with torch.no_grad():
model[0].weight.copy_(torch.tensor(W1.T))
model[2].weight.copy_(torch.tensor(W2.T))
model[0].bias.zero_()
model[2].bias.zero_()
opt = torch.optim.SGD(model.parameters(), lr=0.5)
Xt, yt = torch.tensor(X[:24], dtype=torch.float32), torch.tensor(y[:24])
loader = DataLoader(TensorDataset(Xt, yt), batch_size=8)
batch_loss, all_loss = [], []
for epoch inrange(100):
for xb, yb in loader:
with torch.no_grad():
all_loss.append(round(F.cross_entropy(model(Xt), yt).item(), 4))
opt.zero_grad()
loss = F.cross_entropy(model(xb), yb)
loss.backward()
opt.step()
batch_loss.append(round(loss.item(), 4))
print(len(loader), len(batch_loss), xb.shape)
print(batch_loss[::20])
print(all_loss[::20])
with torch.no_grad():
pred = model(torch.tensor(X[24:], dtype=torch.float32)).argmax(dim=1)
print(round((pred == torch.tensor(y[24:])).float().mean().item(), 3))
Three batches an epoch, 300 steps, each batch 8 signs of 3 features. The lists show every 20th step. The batch losses wobble: 0.0197 at step 80, 0.0263 at step 100, then 0.0254, then 0.0076. The loss on all 24 signs only falls: 0.034, 0.0237, 0.0179, 0.0141. Each batch is a sample of 8 from the 24, so its loss is a sample's average, and it wobbles around the whole set's, just as sample means wobbled around the true mean in lesson 98. It still falls. At step 200 the loss on all 24 is 0.0085, against 0.0087 in the full-batch run, though each step here reads only a third of the signs, and all 12 held-back signs are read right: 1.0.
Plot both curves. This cell types in the two lists the last cell printed. plt.semilogy is plt.plot with a log scale on the y-axis (lesson 25): each mark down the axis is a tenth of the one above, so the small losses at the end get as much room as the big ones at the start.
On your plot the batch curve rises from step 80 to step 100, from 0.0197 to 0.0263. Has training got worse?
No: a rising loss always falls next step
No: the loss on all 24 signs kept falling
Yes: the learning rate has become too high
Yes: the network has started to overfit
Show the answer
No: the loss on all 24 signs kept falling
Over the same steps the loss on all 24 signs fell, from 0.034 to 0.0237. A batch's loss is a sample of 8, so it wobbles around the whole set's. Judge training by the trend, or by the loss on all the data.
What a real install adds. Every line here runs unchanged on PyTorch on your own machine, which brings three things the browser can't. A graphics card runs the same arithmetic many times faster. Real datasets, from image collections to your own files, come in through the same DataLoader. And shuffle=True uses PyTorch's own random numbers. The kind of number stays the same: float32, unless you ask for another.
Drill
Train in batches of 6
Train the network in batches of 6 for 50 epochs, from the same starting weights. Set loader, then steps to the number of updates, final to the loss on all 24 training signs after training, rounded to 4 places, and acc to the share of the 12 held-back signs read right.
import numpy as np
import torch
import torch.nn as nn
import torch.nn.functional as F
from torch.utils.data import DataLoader, TensorDataset
rng = np.random.default_rng(7)
centers = np.array([[0.9, 0.2, 0.1], [0.2, 0.9, 0.3], [0.1, 0.3, 0.9]])
y = np.arange(36) % 3
X = centers[y] + rng.normal(0, 0.15, (36, 3))
r = np.random.default_rng(1)
W1, W2 = r.normal(0, 0.5, (3, 4)), r.normal(0, 0.5, (4, 3))
model = nn.Sequential(nn.Linear(3, 4), nn.ReLU(), nn.Linear(4, 3))
with torch.no_grad():
model[0].weight.copy_(torch.tensor(W1.T))
model[2].weight.copy_(torch.tensor(W2.T))
model[0].bias.zero_()
model[2].bias.zero_()
opt = torch.optim.SGD(model.parameters(), lr=0.5)
Xt, yt = torch.tensor(X[:24], dtype=torch.float32), torch.tensor(y[:24])
Xte, yte = torch.tensor(X[24:], dtype=torch.float32), torch.tensor(y[24:])
loader = None
steps = 0
final = 0
acc = 0print(steps, final, acc)
In the lesson, your code is checked as you write it.
In your own words
A batch's loss went up between two steps. Why is that no reason to stop training, and what would be?
Write a sentence or two, then compare it with a good answer in the lesson.
Goes deeper in
PyTorch tensors: the tensor toolkit in full, and how a tensor moves to a graphics card.
31Formulas with fractions: clear them firstFreeClearing denominators, cross-multiplying, a letter in the bottom, and reciprocal formulas like 1/R = 1/R₁ + 1/R₂.35 min
37Inverse functions, lines and scalingFreeUndoing a function by making x the subject, domains that allow it, line forms, and unscaling predictions.40 min
38Sequences and sums: find n, a, d and rFreeEvery letter of the arithmetic and geometric formulas as the subject, and an average's window from its decay.40 min
40Review: rearrange anything (algebra)FreeBroken rearrangements to fix, mixed practice from the whole algebra block, a strategy chart, and its formula sheet.35 min
41Coordinates and distanceFreePoints as (x, y), Pythagoras, distance in any number of coordinates, and nearest neighbors.42 min
42Angles, degrees and radiansFreeMeasuring a turn, converting units, arcs, and the angle between two arrows.36 min
43Sine and cosineFreeAcross and up: right triangles, the unit circle, and directions and rotations.42 min
44Waves: height, speed and shiftFreeAmplitude, period and phase, sampling a wave, and position codes in transformers.37 min
68Vector equations: rearranging with arrowsFreeSolving for a vector or a scale, recovering a gradient from two weight snapshots, and an angle from a dot product.40 min
87Counting: choices, orders and groupsFreeThe counting principle, factorials, ordered picks and groups, and why attention's cost grows with the square of the length.43 min
88Probability: from counts to chancesFreeEqually likely outcomes, the complement, distributions, seeded simulation, and a next-word model.43 min
89Combining events: and, or, independentFreeThe or rule, independence, probability trees, drawing with and without replacement, and chains of steps.42 min
100Maximum likelihoodFreeLikelihoods as products, log-likelihoods as sums, grid search, underflow, and the negative log-likelihood loss.45 min
101Linear regression: the best lineFreeResiduals, least squares by formula, three roads to one line, the matrix form, and train versus test error.45 min
109One neuron, start to finishFreeWeighted sum, activation, loss, slopes and update, by hand and in code, and what one straight line can and can't split.45 min
111Training a small networkFreeThe backward pass in matrix form, gradient checks, a full training loop, and reading loss curves.45 min
112The same network in PyTorchFreenn.Sequential with the NumPy weights loaded, the same loss and slopes, then SGD and mini-batches.45 min
113Attention as matrix mathFreeScaled dot-product scores, the causal mask, a row-by-row softmax, and blending value vectors.45 min
114Residual connections and layer normFreeAdding the input back and why its slope keeps learning alive, layer norm by hand, and add and norm.40 min
115A transformer block, start to finishFreeA sentence through embeddings, attention, add and norm, a small network and next-word scores, every shape named.45 min
117Final test: transpose any formulaFreeSlips to fix from every topic, mixed practice of every kind of rearranging, the strategy chart and the full sheet.45 min
Learn the math behind AI from arithmetic up: numbers, algebra, vectors and matrices, calculus, probability and statistics, each shown as numbers moving and checked in Python, ending with a network you build and train from the formulas.
See the course →
Never trained an AI? Teach a small one to write, right in your browser. It takes about 20 seconds and needs no account. Teach it to write