Index Ai Academy
Foundation · Foundation mathPutting it together

The same network in PyTorch

About 45 minLooking ahead: some prerequisites aren't done yet

Why this matters

Lessons 109 to 111 built a neural network from formulas and trained it, every line written by hand. Engineers write the same network in about ten lines of PyTorch. This lesson puts those ten lines beside your NumPy and shows they are the same thing, number for number: from the same starting weights they give the same loss, the same slopes and the same training run. Then it trains the network the way real models are trained, a few examples at a time, and reads the noisier loss curve that comes out.

  • Build the two-layer network as nn.Sequential, and name the formula behind each layer and each call
  • Load the NumPy weights in nn.Linear's (out, in) layout, and get the same loss and the same slopes
  • Train it with optim.SGD beside the NumPy loop, step for step, and catch a missing zero_grad() by its numbers
  • Train it on mini-batches from a DataLoader, and read the noisier loss curve that gives
Quick check
Have a guess before you start. Your NumPy network and a PyTorch copy of it start from the same weights and train on the same 24 signs at rate 0.5. After 200 steps, how do their losses compare?

The math: every line is a formula you know

Purpose. Read each line of a PyTorch training loop as the formula it computes.

Here is lesson 111's network and its whole training loop, written the way engineers write it:

model = nn.Sequential(nn.Linear(3, 4), nn.ReLU(), nn.Linear(4, 3))
opt = torch.optim.SGD(model.parameters(), lr=0.5)
for step in range(300):
    opt.zero_grad()
    loss = F.cross_entropy(model(X), y)
    loss.backward()
    opt.step()

Nothing in it is new math. nn.Sequential runs the layers it's given in order, each one's output feeding the next, so model(X) is three functions composed (lesson 20). Each piece is a formula you've worked by hand:

PyTorchthe formula it computesfrom
nn.Linear(3, 4)Z₁ = XWᵀ + b, its weight W stored 4 × 3: your W₁ turned aroundlesson 56
nn.ReLU()A₁ = max(0, Z₁), entry by entrylesson 110
nn.Linear(4, 3)the scores, Z₂ = A₁Wᵀ + b, W stored 3 × 4lesson 110
F.cross_entropy(scores, y)softmax of each row, then the mean of −log of each right class's chancelesson 104, lesson 105
loss.backward()the five lines, dZ₂ = (P − Y) ÷ n back to dW₁, each slope stored in its weight's .gradlesson 111
opt.step()w ← w − η·g, for every weight and biaslesson 79
opt.zero_grad()every .grad emptied, because backward() adds into itlesson 82