Index Ai Academy
Foundation · Putting it together

The same network in PyTorch

nn.Sequential with the NumPy weights loaded, the same loss and slopes, then SGD and mini-batches.

About 45 minutes · Free lesson

What it covers6

Why it matters

Lessons 109 to 111 built a neural network from formulas and trained it, every line written by hand. Engineers write the same network in about ten lines of PyTorch. This lesson puts those ten lines beside your NumPy and shows they are the same thing, number for number: from the same starting weights they give the same loss, the same slopes and the same training run. Then it trains the network the way real models are trained, a few examples at a time, and reads the noisier loss curve that comes out.

What you'll be able to do

  • Build the two-layer network as nn.Sequential, and name the formula behind each layer and each call
  • Load the NumPy weights in nn.Linear's (out, in) layout, and get the same loss and the same slopes
  • Train it with optim.SGD beside the NumPy loop, step for step, and catch a missing zero_grad() by its numbers
  • Train it on mini-batches from a DataLoader, and read the noisier loss curve that gives
Quick check
Have a guess before you start. Your NumPy network and a PyTorch copy of it start from the same weights and train on the same 24 signs at rate 0.5. After 200 steps, how do their losses compare?
  1. Equal in every one of their digits
  2. The same, to four decimal places
  3. Far apart: PyTorch starts from its own random weights
  4. PyTorch's is far lower: its optimizer is smarter
Show the answer

The same, to four decimal places

The math is the same, so the numbers are too: 0.0087 for each at step 200, as the Python part prints. They part only around the sixth or seventh digit, because PyTorch keeps its numbers in float32 and NumPy in float64.

The math: every line is a formula you know

Purpose. Read each line of a PyTorch training loop as the formula it computes.

Here is lesson 111's network and its whole training loop, written the way engineers write it:

model = nn.Sequential(nn.Linear(3, 4), nn.ReLU(), nn.Linear(4, 3))
opt = torch.optim.SGD(model.parameters(), lr=0.5)
for step in range(300):
    opt.zero_grad()
    loss = F.cross_entropy(model(X), y)
    loss.backward()
    opt.step()

Nothing in it is new math. nn.Sequential runs the layers it's given in order, each one's output feeding the next, so model(X) is three functions composed (lesson 20). Each piece is a formula you've worked by hand:

PyTorchthe formula it computesfrom
nn.Linear(3, 4)Z₁ = XWᵀ + b, its weight W stored 4 × 3: your W₁ turned aroundlesson 56
nn.ReLU()A₁ = max(0, Z₁), entry by entrylesson 110
nn.Linear(4, 3)the scores, Z₂ = A₁Wᵀ + b, W stored 3 × 4lesson 110
F.cross_entropy(scores, y)softmax of each row, then the mean of −log of each right class's chancelesson 104, lesson 105
loss.backward()the five lines, dZ₂ = (P − Y) ÷ n back to dW₁, each slope stored in its weight's .gradlesson 111
opt.step()w ← w − η·g, for every weight and biaslesson 79
opt.zero_grad()every .grad emptied, because backward() adds into itlesson 82
Quick check
From lesson 56: a layer made with nn.Linear(25, 3) stores its weight as 3 × 25. A batch X has 10 rows. Which line gives the 10 × 3 scores?
  1. X @ layer.weight + layer.bias
  2. layer.weight @ X + layer.bias
  3. X.T @ layer.weight + layer.bias
  4. X @ layer.weight.T + layer.bias
Show the answer

X @ layer.weight.T + layer.bias

The weight is kept one row per output, 3 × 25, so it's turned around to 25 × 3 for the product: (10 × 25)(25 × 3) gives 10 × 3. That's exactly what layer(X) computes.

The picture. Here is that network on lesson 111's 24 training signs: 3 features in, 4 hidden units, 3 kinds of sign out. The boxes are its layers, and each arrow carries a matrix, labeled with its shape: forward along the top, the slopes back along the bottom. The Python tab lights your NumPy lines as you step. Open the PyTorch tab and the same steps light model(X), loss.backward() and opt.step(): one call for each stretch of the picture.

Try it now

Then set hidden units h to 16 and press Use these numbers: the hidden shapes widen to 24 × 16, and in the PyTorch tab the layers become Linear(3, 16) and Linear(16, 3).

Symbols and formula. The two lines that do the work:

Z=XW⊤+bw←w−η ∂L∂wZ = XW^{\top} + \mathbf{b} \qquad w \leftarrow w - \eta\, \frac{\partial L}{\partial w}

In plain words: a linear layer is your matrix product plus a bias, with its weights kept one row per output and turned around for the product. The optimizer's step moves every weight a little against its own slope.

Counting what training changes. nn.Linear(d, h) holds d × h weights and h biases. The network's two layers hold 3 × 4 + 4 = 16 and 4 × 3 + 3 = 15 numbers, 31 in all, and nn.ReLU holds none: it has nothing to learn. model.parameters() hands those 31 to the optimizer, which is how opt.step() knows what to move.

Quick check
How many numbers does nn.Linear(3, 4) hold for training to change?
  1. 4
  2. 7
  3. 16
  4. 12
Show the answer

16

12 weights, 4 rows of 3, one for each pairing of an input with an output, plus 4 biases, one for each output: 16.

Quick check
model(X) gives the raw scores. Which line gives the same loss as your NumPy code?
  1. F.softmax(model(X), dim=1).mean()
  2. -torch.log(model(X)).mean()
  3. F.cross_entropy(F.softmax(model(X), dim=1), y)
  4. F.cross_entropy(model(X), y)
Show the answer

F.cross_entropy(model(X), y)

F.cross_entropy does the softmax itself, then averages −log of each right class's chance. Give it the raw scores and the class numbers.

In your own words
In your own words: why is model[0].weight 4 × 3 when your NumPy W₁ was 3 × 4, and what does that mean for loading weights?

Write a sentence or two, then compare it with a good answer in the lesson.

In Python: two loops, one run

Purpose. Build the network in PyTorch, load your NumPy weights into it, and train the two side by side.

Start with the network alone. print(model) lists its layers, and model.named_parameters() walks through every tensor that training can change, with its name:

import torch
import torch.nn as nn

model = nn.Sequential(nn.Linear(3, 4), nn.ReLU(), nn.Linear(4, 3))
print(model)
total = 0
for name, p in model.named_parameters():
    print(name, tuple(p.shape))
    total = total + p.numel()
print(total)

The names count the layers from 0: 0.weight is the first layer's 4 × 3 weight and 2.bias the last layer's 3 biases. Layer 1, the ReLU, holds nothing, so no name starts with 1. p.numel() counts a tensor's numbers, and the total is the math part's 31. model[0] and model[2] reach the two layers themselves.

Quick check
What does this print?
import torch.nn as nn

layer = nn.Linear(5, 2)
print(layer.weight.shape, layer.bias.shape)
  1. torch.Size([5, 2]) torch.Size([5])
  2. torch.Size([2, 5]) torch.Size([2])
  3. torch.Size([5, 2]) torch.Size([2])
  4. torch.Size([2, 5]) torch.Size([5])
Show the answer

torch.Size([2, 5]) torch.Size([2])

5 inputs and 2 outputs: the weight is kept one row per output, 2 × 5, with one bias per output, 2.

Code flow. Now the loop's four lines, small enough to watch: lesson 111's one sign and its network with no biases, run for three steps. The weights are plain tensors here, laid out as in your NumPy, so the boxes match that lesson's code flow, and optim.SGD takes a list of them just as it takes model.parameters(). Step through and watch W1 and W2 change on each opt.step(), and the loss fall.

Loading your weights. A new nn.Linear starts with random weights and random biases, so a fresh model can't match your NumPy run. Copy your starting numbers in instead, under torch.no_grad() as in lesson 50: W1.T for the first layer's weight, because it's kept one row per output, and the biases too. The 24 signs are the same as before, made in two lines: centers[y] picks each sign's profile by its kind, as E[ids] picked rows in lesson 65, and np.eye(3)[y] gives the one-hot rows. Then one loop runs both: each pass takes a NumPy step, your twelve lines, and a PyTorch step, four lines, from the same start on the same signs.

import numpy as np
import torch
import torch.nn as nn
import torch.nn.functional as F

rng = np.random.default_rng(7)
centers = np.array([[0.9, 0.2, 0.1], [0.2, 0.9, 0.3], [0.1, 0.3, 0.9]])
y = np.arange(24) % 3
X = centers[y] + rng.normal(0, 0.15, (24, 3))
Y = np.eye(3)[y]
r = np.random.default_rng(1)
W1, b1 = r.normal(0, 0.5, (3, 4)), np.zeros(4)
W2, b2 = r.normal(0, 0.5, (4, 3)), np.zeros(3)

model = nn.Sequential(nn.Linear(3, 4), nn.ReLU(), nn.Linear(4, 3))
with torch.no_grad():
    model[0].weight.copy_(torch.tensor(W1.T))
    model[0].bias.copy_(torch.tensor(b1))
    model[2].weight.copy_(torch.tensor(W2.T))
    model[2].bias.copy_(torch.tensor(b2))
opt = torch.optim.SGD(model.parameters(), lr=0.5)
Xt, yt = torch.tensor(X, dtype=torch.float32), torch.tensor(y)

for step in range(201):
    Z1 = X @ W1 + b1                        # NumPy: the forward pass and the loss
    A1 = np.maximum(Z1, 0)
    Z2 = A1 @ W2 + b2
    E = np.exp(Z2 - Z2.max(axis=1, keepdims=True))
    P = E / E.sum(axis=1, keepdims=True)
    np_loss = -np.mean(np.sum(Y * np.log(P), axis=1))
    dZ2 = (P - Y) / 24                      # NumPy: the five lines, then the update
    dW2, db2 = A1.T @ dZ2, dZ2.sum(axis=0)
    dZ1 = (dZ2 @ W2.T) * (Z1 > 0)
    dW1, db1 = X.T @ dZ1, dZ1.sum(axis=0)
    W1, b1, W2, b2 = W1 - 0.5 * dW1, b1 - 0.5 * db1, W2 - 0.5 * dW2, b2 - 0.5 * db2
    opt.zero_grad()                         # PyTorch: the same step
    loss = F.cross_entropy(model(Xt), yt)
    loss.backward()
    opt.step()
    if step % 100 == 0:
        print(step, round(float(np_loss), 4), round(loss.item(), 4))

The two columns agree at every step printed: 1.044 at the start, 0.0241 at step 100 and 0.0087 at step 200. Twelve lines of NumPy and four of PyTorch make the same training run. dtype=torch.float32 makes the signs PyTorch's usual kind of number, to match the layers, and torch.tensor(y) keeps the kinds as whole numbers, the class numbers F.cross_entropy wants. The runs aren't equal to the last digit, though. PyTorch keeps 24 binary digits a number (float32) where NumPy keeps 53 (float64), as in lesson 12, so printed to six places they part in the last digit or two.

Worked example

Worked example: the same slopes, number for number

Before any training, compare every slope loss.backward() finds with your five lines, at the starting weights.

import numpy as np
import torch
import torch.nn as nn
import torch.nn.functional as F

rng = np.random.default_rng(7)
centers = np.array([[0.9, 0.2, 0.1], [0.2, 0.9, 0.3], [0.1, 0.3, 0.9]])
y = np.arange(24) % 3
X = centers[y] + rng.normal(0, 0.15, (24, 3))
Y = np.eye(3)[y]
r = np.random.default_rng(1)
W1, W2 = r.normal(0, 0.5, (3, 4)), r.normal(0, 0.5, (4, 3))

Z1 = X @ W1
A1 = np.maximum(Z1, 0)
Z2 = A1 @ W2
E = np.exp(Z2 - Z2.max(axis=1, keepdims=True))
dZ2 = (E / E.sum(axis=1, keepdims=True) - Y) / 24
dW2, db2 = A1.T @ dZ2, dZ2.sum(axis=0)
dZ1 = (dZ2 @ W2.T) * (Z1 > 0)
dW1, db1 = X.T @ dZ1, dZ1.sum(axis=0)

model = nn.Sequential(nn.Linear(3, 4), nn.ReLU(), nn.Linear(4, 3))
with torch.no_grad():
    model[0].weight.copy_(torch.tensor(W1.T))
    model[2].weight.copy_(torch.tensor(W2.T))
    model[0].bias.zero_()
    model[2].bias.zero_()
loss = F.cross_entropy(model(torch.tensor(X, dtype=torch.float32)), torch.tensor(y))
loss.backward()
g1, g2 = model[0].weight.grad, model[2].weight.grad
print(tuple(g1.shape), dW1.shape, round(g1[0, 0].item(), 6), round(dW1[0, 0], 6))
print(round(float(np.abs(g1.numpy() - dW1.T).max()), 6), round(float(np.abs(g2.numpy() - dW2.T).max()), 6))
print(round(float(np.abs(model[0].bias.grad.numpy() - db1).max()), 6), round(float(np.abs(model[2].bias.grad.numpy() - db2).max()), 6))
In the lesson you can change this and run it.

PyTorch's first-layer slope is (4, 3), the shape of the weight it keeps, and your dW₁ is (3, 4). Turned around, they agree. The slope of the very first weight is 0.028542 both ways, the gradient check's number from lesson 111, and the largest gap anywhere, for weights and biases alike, is 0.0 to six places. bias.zero_() sets a bias to 0 in place, the start your NumPy biases had. One call to backward() gave all 31 slopes.

A named mistake: forgetting opt.zero_grad(). backward() adds each slope into .grad instead of replacing it (lesson 82). Leave out the clearing line and every step uses the sum of all the slopes so far. This cell runs the PyTorch loop twice from the same start, with and without opt.zero_grad(), and prints the loss, the largest stored slope in the last layer and its largest weight:

import numpy as np
import torch
import torch.nn as nn
import torch.nn.functional as F

rng = np.random.default_rng(7)
centers = np.array([[0.9, 0.2, 0.1], [0.2, 0.9, 0.3], [0.1, 0.3, 0.9]])
y = np.arange(24) % 3
Xt = torch.tensor(centers[y] + rng.normal(0, 0.15, (24, 3)), dtype=torch.float32)
yt = torch.tensor(y)
r = np.random.default_rng(1)
W1, W2 = r.normal(0, 0.5, (3, 4)), r.normal(0, 0.5, (4, 3))

def run(clear):
    model = nn.Sequential(nn.Linear(3, 4), nn.ReLU(), nn.Linear(4, 3))
    with torch.no_grad():
        model[0].weight.copy_(torch.tensor(W1.T))
        model[2].weight.copy_(torch.tensor(W2.T))
        model[0].bias.zero_()
        model[2].bias.zero_()
    opt = torch.optim.SGD(model.parameters(), lr=0.5)
    for step in range(201):
        if clear:
            opt.zero_grad()
        loss = F.cross_entropy(model(Xt), yt)
        loss.backward()
        if step in [1, 2, 200]:
            print(clear, step, round(loss.item(), 4), round(model[2].weight.grad.abs().max().item(), 3), round(model[2].weight.abs().max().item(), 2))
        opt.step()

run(True)
run(False)

At step 1 the losses match, 1.0148, but the forgetful run already stores 0.118 where the clean one stores 0.068: this step's slope plus the last one's. From step 2 the losses part, 0.958 against your 0.9854. By step 200 the new slopes have almost vanished, yet the stored sum sits at 0.677, so every step pushes the weights another 0.5 × 0.677 ≈ 0.34 the same way, for as long as it runs: the largest weight is 66.01 against 2.21. The loss reads 0.0, which looks perfect and isn't. The network is sure of every answer for no good reason, and on noisier data a runaway like this throws the loss up instead. Comparing with a run you trust is how you catch it.

Quick check
What does this print?
import torch

w = torch.tensor([1.0, 2.0], requires_grad=True)
opt = torch.optim.SGD([w], lr=0.5)
loss = (w * w).sum()
loss.backward()
opt.step()
print(w.tolist())
  1. [0.5, 1.5]
  2. [2.0, 4.0]
  3. [1.0, 2.0]
  4. [0.0, 0.0]
Show the answer

[0.0, 0.0]

backward() puts the slope of w₁² + w₂², which is 2w = (2, 4), into w.grad. step() does w − 0.5 × (2, 4) = (1 − 1, 2 − 2) = (0, 0): straight to the bottom of this bowl.

Drill

Match the NumPy run

This PyTorch run should match your NumPy one: a starting loss of 1.044 and 0.0087 at step 200, to four places. The starter has three slips. Fix them so that start and loss200 match.

import numpy as np
import torch
import torch.nn as nn
import torch.nn.functional as F

rng = np.random.default_rng(7)
centers = np.array([[0.9, 0.2, 0.1], [0.2, 0.9, 0.3], [0.1, 0.3, 0.9]])
y = np.arange(24) % 3
X = centers[y] + rng.normal(0, 0.15, (24, 3))
r = np.random.default_rng(1)
W1, W2 = r.normal(0, 0.5, (3, 4)), r.normal(0, 0.5, (4, 3))

model = nn.Sequential(nn.Linear(3, 4), nn.ReLU(), nn.Linear(4, 3))
with torch.no_grad():
    model[0].weight.copy_(torch.tensor(W1))
    model[2].weight.copy_(torch.tensor(W2.T))
opt = torch.optim.SGD(model.parameters(), lr=0.5)
Xt, yt = torch.tensor(X, dtype=torch.float32), torch.tensor(y)
losses = []
for step in range(201):
    loss = F.cross_entropy(model(Xt), yt)
    losses.append(round(loss.item(), 4))
    loss.backward()
    opt.step()
start, loss200 = losses[0], losses[200]
print(start, loss200)
In the lesson, your code is checked as you write it.
In your own words
Your PyTorch run matched NumPy's to four places. Why is a match like that worth more than a PyTorch run that simply trains well?

Write a sentence or two, then compare it with a good answer in the lesson.

In AI and ML: mini-batches, the way real models train

Purpose. Train on a few signs at a time, served by a DataLoader, and read the noisier loss curve that comes out.

Mini-batches. Your loops used all 24 signs for every step. A real dataset holds thousands or millions of examples, far too many to read for every small step. So training takes a mini-batch, a few examples, works out the loss and the slopes on those alone, steps, and moves on to the next few. One pass through all the data is an epoch. PyTorch's DataLoader deals the batches: wrap the tensors in a TensorDataset, which pairs each sign with its kind, and a loop over the loader gets one batch of signs and their kinds at a time.

Quick check
The 24 signs go into a DataLoader with batch_size=8. How many times does opt.step() run in one epoch?
  1. 24
  2. 1
  3. 3
  4. 8
Show the answer

3

Each batch of 8 gets one step, and 24 signs make 3 batches.

Here is the network trained that way: 100 epochs of 3 steps, from the same starting weights. Each step records two losses: the batch's own, the one the step works on, and the loss on all 24 signs, measured under torch.no_grad() so it stays off the tape that backward() reads (lesson 82). The 12 held-back signs are scored at the end. Real training code passes shuffle=True to deal the signs in a new order every epoch; it's left off here, so the cell prints the same numbers in your browser and on a real install.

import numpy as np
import torch
import torch.nn as nn
import torch.nn.functional as F
from torch.utils.data import DataLoader, TensorDataset

rng = np.random.default_rng(7)
centers = np.array([[0.9, 0.2, 0.1], [0.2, 0.9, 0.3], [0.1, 0.3, 0.9]])
y = np.arange(36) % 3
X = centers[y] + rng.normal(0, 0.15, (36, 3))
r = np.random.default_rng(1)
W1, W2 = r.normal(0, 0.5, (3, 4)), r.normal(0, 0.5, (4, 3))
model = nn.Sequential(nn.Linear(3, 4), nn.ReLU(), nn.Linear(4, 3))
with torch.no_grad():
    model[0].weight.copy_(torch.tensor(W1.T))
    model[2].weight.copy_(torch.tensor(W2.T))
    model[0].bias.zero_()
    model[2].bias.zero_()
opt = torch.optim.SGD(model.parameters(), lr=0.5)
Xt, yt = torch.tensor(X[:24], dtype=torch.float32), torch.tensor(y[:24])
loader = DataLoader(TensorDataset(Xt, yt), batch_size=8)

batch_loss, all_loss = [], []
for epoch in range(100):
    for xb, yb in loader:
        with torch.no_grad():
            all_loss.append(round(F.cross_entropy(model(Xt), yt).item(), 4))
        opt.zero_grad()
        loss = F.cross_entropy(model(xb), yb)
        loss.backward()
        opt.step()
        batch_loss.append(round(loss.item(), 4))
print(len(loader), len(batch_loss), xb.shape)
print(batch_loss[::20])
print(all_loss[::20])
with torch.no_grad():
    pred = model(torch.tensor(X[24:], dtype=torch.float32)).argmax(dim=1)
print(round((pred == torch.tensor(y[24:])).float().mean().item(), 3))

Three batches an epoch, 300 steps, each batch 8 signs of 3 features. The lists show every 20th step. The batch losses wobble: 0.0197 at step 80, 0.0263 at step 100, then 0.0254, then 0.0076. The loss on all 24 signs only falls: 0.034, 0.0237, 0.0179, 0.0141. Each batch is a sample of 8 from the 24, so its loss is a sample's average, and it wobbles around the whole set's, just as sample means wobbled around the true mean in lesson 98. It still falls. At step 200 the loss on all 24 is 0.0085, against 0.0087 in the full-batch run, though each step here reads only a third of the signs, and all 12 held-back signs are read right: 1.0.

Plot both curves. This cell types in the two lists the last cell printed. plt.semilogy is plt.plot with a log scale on the y-axis (lesson 25): each mark down the axis is a tenth of the one above, so the small losses at the end get as much room as the big ones at the start.

import matplotlib.pyplot as plt

batch = [1.0485, 0.4091, 0.1489, 0.0792, 0.0197, 0.0263, 0.0254, 0.0076, 0.0129, 0.0142, 0.0045, 0.0083, 0.0096, 0.0031, 0.006]
whole = [1.044, 0.4351, 0.131, 0.0581, 0.034, 0.0237, 0.0179, 0.0141, 0.0117, 0.0099, 0.0085, 0.0075, 0.0067, 0.006, 0.0054]
plt.semilogy(range(0, 300, 20), batch, label="each batch of 8")
plt.semilogy(range(0, 300, 20), whole, label="all 24 signs")
plt.xlabel("step")
plt.ylabel("training loss")
plt.legend()
plt.show()
Quick check
On your plot the batch curve rises from step 80 to step 100, from 0.0197 to 0.0263. Has training got worse?
  1. No: a rising loss always falls next step
  2. No: the loss on all 24 signs kept falling
  3. Yes: the learning rate has become too high
  4. Yes: the network has started to overfit
Show the answer

No: the loss on all 24 signs kept falling

Over the same steps the loss on all 24 signs fell, from 0.034 to 0.0237. A batch's loss is a sample of 8, so it wobbles around the whole set's. Judge training by the trend, or by the loss on all the data.

What a real install adds. Every line here runs unchanged on PyTorch on your own machine, which brings three things the browser can't. A graphics card runs the same arithmetic many times faster. Real datasets, from image collections to your own files, come in through the same DataLoader. And shuffle=True uses PyTorch's own random numbers. The kind of number stays the same: float32, unless you ask for another.

Drill

Train in batches of 6

Train the network in batches of 6 for 50 epochs, from the same starting weights. Set loader, then steps to the number of updates, final to the loss on all 24 training signs after training, rounded to 4 places, and acc to the share of the 12 held-back signs read right.

import numpy as np
import torch
import torch.nn as nn
import torch.nn.functional as F
from torch.utils.data import DataLoader, TensorDataset

rng = np.random.default_rng(7)
centers = np.array([[0.9, 0.2, 0.1], [0.2, 0.9, 0.3], [0.1, 0.3, 0.9]])
y = np.arange(36) % 3
X = centers[y] + rng.normal(0, 0.15, (36, 3))
r = np.random.default_rng(1)
W1, W2 = r.normal(0, 0.5, (3, 4)), r.normal(0, 0.5, (4, 3))
model = nn.Sequential(nn.Linear(3, 4), nn.ReLU(), nn.Linear(4, 3))
with torch.no_grad():
    model[0].weight.copy_(torch.tensor(W1.T))
    model[2].weight.copy_(torch.tensor(W2.T))
    model[0].bias.zero_()
    model[2].bias.zero_()
opt = torch.optim.SGD(model.parameters(), lr=0.5)
Xt, yt = torch.tensor(X[:24], dtype=torch.float32), torch.tensor(y[:24])
Xte, yte = torch.tensor(X[24:], dtype=torch.float32), torch.tensor(y[24:])

loader = None
steps = 0
final = 0
acc = 0
print(steps, final, acc)
In the lesson, your code is checked as you write it.
In your own words
A batch's loss went up between two steps. Why is that no reason to stop training, and what would be?

Write a sentence or two, then compare it with a good answer in the lesson.

Goes deeper in

Practice what you just read

  • 7 code examples you can edit and run
  • a practice exercise that checks your code
  • a 5-question quiz
  • 9 review cards for spaced repetition

Try a quiz question

Which call in a PyTorch training loop changes the weights?

  1. opt.step()
  2. loss.backward()
  3. opt.zero_grad()
  4. model(X)

Answer it at the end of the lesson to see if you're right, and why.

Before this lesson

Keep learning

Where this lesson leads.

575 lessons in thirteen phases. 176 are free without an account, and a free account opens the rest. See every lesson.

Foundation · about 96 hours of study

Foundation math

From whole numbers to the math inside a neural network: each idea by hand, in Python, and in AI 118 lessons All free
  1. Numbers and place value FreeWhat a digit's place is worth, the number line, and Python as a calculator that never tires.36 min
  2. Multiplying and dividing: groups FreeEqual groups, quotient and remainder, and cutting training data into batches.35 min
  3. Order of operations, and names for numbers FreeBrackets, powers, then multiply and divide, then add and subtract; variables hold numbers.36 min
  4. Numbers below zero FreeThe number line through zero, sign rules, distance from zero, and an error's direction and size.37 min
  5. Fractions: equal parts, equal fractions, adding FreePieces of a whole, equal fractions, and re-cutting to add; accuracy as a fraction.45 min
  6. Multiplying and dividing fractions FreeA share of a share, dividing by flipping, and why chances in a row get small.35 min
  7. Decimals and rounding FreeFractions in place value, rounding, and why 0.1 + 0.2 isn't exactly 0.3.36 min
  8. Percentages FreePer hundred, percent change, percentage points, and the accuracy and error rates of models.35 min
  9. Ratios, rates and proportion FreeSharing in a ratio, rates per unit, scaling, and splitting data 80/10/10.35 min
  10. Powers and roots FreeRepeated multiplying, the laws of exponents, square roots, and squared error.43 min
  11. Very big and very small numbers FreePowers of ten, e-notation like 3e-4, orders of magnitude, and a model's size in memory.36 min
  12. Bits, bytes and floats: how numbers are stored FreeBase 2, what n bits can hold, a float as sign, power and digits, and a tensor's memory.35 min
  13. Review: find the slip (numbers) FreeBroken calculations to fix, mixed practice from the whole module, and its formula sheet.34 min
  14. Letters for numbers FreeVariables, terms and like terms, substitution, and a model's formula as a Python function.41 min
  15. Expanding and factoring FreeBrackets as areas, pulling out common factors, and a squared error expanded into a bowl.41 min
  16. Solving equations: keep the balance FreeThe same move on both sides, undoing steps in reverse, and running a model's formula backward.41 min
  17. Rearranging a formula FreeMaking any letter the subject, round-trip checks, and planning a training run.36 min
  18. Inequalities and true-or-false FreeRanges, flipping the sign, if and elif, and the thresholds, clips and ReLU inside a model.42 min
  19. Functions: a rule from input to output Freef(x) notation, tables and graphs, domain and range, loops over lists, and loss as a function.42 min
  20. Chaining and undoing functions Freeg(f(x)), inverses that undo, scaling and unscaling, and a network as a chain.42 min
  21. Straight lines: slope and intercept FreeRise over run, the line through two points, and a model's weight and bias as slope and intercept.42 min
  22. Two equations, two unknowns FreeSubstitution and elimination, where lines cross, and why models are fitted, not solved.42 min
  23. Quadratics: bowls and their lowest point FreeRoots by factoring and by formula, the discriminant, the vertex, and the best weight of a model.42 min
  24. Exponentials: growth and decay FreeThe same factor each step, the number e, and decaying learning rates.43 min
  25. Logarithms: the power question FreeTo what power?, the laws of logs, and adding log-probabilities instead of multiplying.45 min
  26. Sequences and Σ FreeArithmetic and geometric sequences, Σ as a loop that adds, and mean squared error.40 min
  27. Weighted and moving averages FreeWeights that add to 1, a window average, the exponential moving average, and smoothing a loss curve.35 min
  28. Sets, and reading the symbols FreeSets, union and intersection, ℝⁿ and f : ℝⁿ → ℝᵐ, and reading a paper's first equation.41 min
  29. Review: find the slip (algebra) FreeBroken algebra and code to fix, mixed practice from the whole module, and its formula sheet.34 min
  30. Transposing step by step: undo in reverse order FreeA formula's layers, the opposite of each, peeling from the outside in, conditions, and round-trip checks.35 min
  31. Formulas with fractions: clear them first FreeClearing denominators, cross-multiplying, a letter in the bottom, and reciprocal formulas like 1/R = 1/R₁ + 1/R₂.35 min
  32. When the letter appears twice: collect and factor FreeGathering the letter on one side, factoring it out, dividing by the bracket, and rearranging precision and F1.35 min
  33. Squares, roots and powers FreeSquare roots with ±, squaring to undo a root, cube and nth roots, and the √d inside attention.40 min
  34. When the letter is squared and plain: quadratics FreeRearranging into ax² + bx + c = 0, the formula, completing the square, and choosing the root that fits.35 min
  35. Letters in the power: exponentials and logarithms FreeTaking logs to bring a power down, exponentiating to free a letter, and the logit that undoes the sigmoid.40 min
  36. Rearranging inequalities: when the sign flips FreeThe four moves on an inequality, negatives and reciprocals that flip it, logs below 1, and sample sizes.35 min
  37. Inverse functions, lines and scaling FreeUndoing a function by making x the subject, domains that allow it, line forms, and unscaling predictions.40 min
  38. Sequences and sums: find n, a, d and r FreeEvery letter of the arithmetic and geometric formulas as the subject, and an average's window from its decay.40 min
  39. Combining formulas: substitute and eliminate FreePutting one formula inside another to remove a letter, and sizing a training run from a time or money budget.35 min
  40. Review: rearrange anything (algebra) FreeBroken rearrangements to fix, mixed practice from the whole algebra block, a strategy chart, and its formula sheet.35 min
  41. Coordinates and distance FreePoints as (x, y), Pythagoras, distance in any number of coordinates, and nearest neighbors.42 min
  42. Angles, degrees and radians FreeMeasuring a turn, converting units, arcs, and the angle between two arrows.36 min
  43. Sine and cosine FreeAcross and up: right triangles, the unit circle, and directions and rotations.42 min
  44. Waves: height, speed and shift FreeAmplitude, period and phase, sampling a wave, and position codes in transformers.37 min
  45. Review: find the slip (shapes and waves) FreeBroken geometry and code to fix, mixed practice from the module, and its formula sheet.29 min
  46. Angles, triangles and waves: rearranging with sin, cos and tan FreeSides and angles from SOH CAH TOA, principal values, arcs, polar coordinates, and a wave's time and period.45 min
  47. Vectors: lists of numbers that point FreeA list, a point and an arrow; components, dimension, NumPy arrays, and data as vectors.42 min
  48. Adding and scaling vectors FreeComponent-wise sums, scalar multiples, mixes, and the weight update w − ηg.43 min
  49. Length and distance: norms FreeL2, L1 and max norms, unit vectors, distance, and the norms inside losses and clipping.42 min
  50. The dot product FreeMultiply in pairs and add; a·a as squared length; a @ b; one neuron as a dot product.43 min
  51. Angle, cosine similarity and projection FreeDirection without length, perpendicular vectors, shadows, and search by meaning.45 min
  52. Perpendicular directions: bases and coordinates FreeOrthonormal directions, coordinates as dot products, rebuilding a vector, and taking a direction out.35 min
  53. Matrices: tables of numbers FreeRows, columns and entries; two-dimensional arrays and slices; a data set as a matrix.41 min
  54. Adding, scaling and transposing matrices FreeEntry-by-entry sums and scalings, the transpose, mean images, and tables turned around.36 min
  55. A matrix times a vector FreeEach row dotted with the vector, the size rule, A @ x, and a layer acting on one example.42 min
  56. Matrix multiplication FreeRow meets column, the size rule, order matters, and a batch through a layer: X W + b.45 min
  57. Identity, diagonal, triangular and symmetric FreeFour patterns and their jobs: rescaling, distance tables, and the causal mask.37 min
  58. Matrices as movements of space FreeStretches, flips, shears and rotations; columns as landing spots; chaining; a layer moving data.42 min
  59. Determinant and inverse FreeThe area factor ad − bc, singular matrices, undoing a matrix, and redundant features.42 min
  60. Solving systems by elimination FreeAugmented matrices, row moves, back-substitution, the three outcomes, and fitting weights.45 min
  61. Independence, span and rank FreeRedundant vectors, span, rank, outer products, and low-rank updates like LoRA.42 min
  62. Eigenvectors: directions that only stretch FreeAv = λv, checking candidates, np.linalg.eig, and the main direction of a data cloud.41 min
  63. SVD and PCA: the important directions first FreeA matrix as pieces by size, the best rank-k copy, compression, and PCA maps of data.45 min
  64. Tensors, shapes and broadcasting FreeAxes and shapes, reshape, axis=, broadcasting, and shapes through a small model.43 min
  65. One-hot vectors and lookup tables: embeddings FreeA one-hot row picks a table's row, a sentence looked up as vectors, and a score for every word.35 min
  66. Sliding dot products: convolution FreeA small pattern slid along a list and over a grid, output sizes, stride and padding, and shared weights.40 min
  67. Review: find the slip (vectors and matrices) FreeShape slips to fix, mixed practice from the module, and its formula sheet with shapes.36 min
  68. Vector equations: rearranging with arrows FreeSolving for a vector or a scale, recovering a gradient from two weight snapshots, and an angle from a dot product.40 min
  69. Matrix equations: multiply by the inverse, on the correct side FreeNo dividing by a matrix: inverses from the left or right, order, transposes, and the normal equations.40 min
  70. Rates of change FreeChange over change, secants, rates from readings with np.diff, and reading a loss curve.41 min
  71. Limits: what a value heads toward FreeApproaching from both sides, holes, corners and jumps, epsilon, and judging activations.37 min
  72. The derivative: slope at a point FreeShrinking secants, f′(x) and dy/dx, Δy ≈ f′Δx, and which way a weight should move.45 min
  73. Derivative rules FreeConstants, powers, multiples, sums, eˣ and ln x, checked numerically, and a squared error's slope.41 min
  74. The product rule and the chain rule FreeProducts and chains of functions, forward then backward, and one neuron's dL/dw.45 min
  75. Lowest points and curvature Freef′ = 0, the second-derivative test, tables, local and global minima, and convex losses.43 min
  76. Standing in for a curve near a point FreeTangent lines, a curvature term, growing error, and why learning rates must be small.37 min
  77. Partial derivatives FreeOne input at a time, ∂, nudging in Python, and the two slopes of a line's loss.41 min
  78. The gradient FreeAll the slopes as a vector, steepest climb and descent, a numerical gradient, and the loss gradient.42 min
  79. Gradient descent FreeStepping against the gradient, the learning rate, a training loop, and fitting a line.45 min
  80. Faster descent: momentum and Adam FreeMomentum and Adam by hand, the bias correction, and three optimizers down one narrow valley.40 min
  81. The chain rule with many paths: backpropagation by hand FreeGraphs, multiply along paths and add across them, gradient checks, and a 2-2-1 network by hand.45 min
  82. Automatic differentiation: how PyTorch finds every slope FreeA tape of steps walked backward, and requires_grad, backward(), .grad, zero_grad() and no_grad().40 min
  83. Jacobians and Hessians FreeSlopes of vector functions, a layer's Jacobian, second partials, and bowls, domes and saddles.42 min
  84. Integrals: adding up thin slices FreeStrip and trapezoid sums, summing undoes differencing, probability as area, and AUC.43 min
  85. Review: find the slip (calculus) FreeCalculus slips to fix, mixed practice from the module, and its formula sheet.36 min
  86. Rearranging in calculus: slopes, steps and constants FreeSetting a slope to zero, Newton's step, the chain rule turned around, constants from a point, and learning rates.45 min
  87. Counting: choices, orders and groups FreeThe counting principle, factorials, ordered picks and groups, and why attention's cost grows with the square of the length.43 min
  88. Probability: from counts to chances FreeEqually likely outcomes, the complement, distributions, seeded simulation, and a next-word model.43 min
  89. Combining events: and, or, independent FreeThe or rule, independence, probability trees, drawing with and without replacement, and chains of steps.42 min
  90. Conditional probability and Bayes' rule FreeConditionals from tables, why P(A | B) isn't P(B | A), Bayes' rule, and precision and recall.45 min
  91. Random variables and distributions FreeUncertain numbers, the probability of a range, sampling by slices, and sampling versus argmax.42 min
  92. Expected value, variance and standard deviation FreeProbability-weighted averages, spread in the data's own units, and a training loss as an expected value.42 min
  93. Bernoulli, binomial, uniform and normal FreeYes-or-no trials, counts of yeses, the 68-95-99.7 rule, z-scores, and an accuracy's error bar.45 min
  94. Two variables at once: covariance and correlation FreeJoint tables and marginals, covariance, correlation, the covariance matrix, and features that carry the same information.45 min
  95. Review: find the slip (probability) FreeProbability slips to fix, mixed practice from the module, and its formula sheet.36 min
  96. Probability formulas: solve for the missing chance FreeConditionals, Bayes' rule for the prior, the or rule, expected values, variance, and z-scores turned around.45 min
  97. Describing data: center, spread and scale FreeMean, median and mode, variance and standard deviation, percentiles, outliers, and standardizing features.42 min
  98. Samples, the law of large numbers and the central limit theorem FreeSamples and populations, the standard error σ/√n, the bell of averages, and error bars on test scores.43 min
  99. Is the difference real? Comparing two models FreePaired differences, a bootstrap interval, what a p-value is and isn't, and why many tries make lucky wins.40 min
  100. Maximum likelihood FreeLikelihoods as products, log-likelihoods as sums, grid search, underflow, and the negative log-likelihood loss.45 min
  101. Linear regression: the best line FreeResiduals, least squares by formula, three roads to one line, the matrix form, and train versus test error.45 min
  102. Logistic regression: a line that answers yes or no FreeThe sigmoid, the log loss, its slope p − y, a training loop, and choosing a threshold.45 min
  103. Scoring a classifier: confusion table, precision, recall, F1 FreeThe four cells, accuracy, precision, recall and F1, and what moving a threshold does to each.35 min
  104. Softmax: scores into probabilities FreeExponentiate and share out, shifting and temperature, the safe softmax, and a classifier's last step.43 min
  105. Surprise, entropy and cross-entropy FreeSurprise in bits, entropy, cross-entropy and KL divergence, perplexity, and a language model's loss.45 min
  106. KL divergence: how far one distribution is from another FreeKL by hand, cross-entropy as entropy plus KL, why it isn't symmetric, and a soft-target loss.35 min
  107. Review: find the slip (statistics and information) FreeStatistics slips to fix, mixed practice from the module, and its formula sheet.38 min
  108. Statistics and information: n, z, logits and bits FreeSample sizes from error bars, inputs from probabilities, logit gaps from softmax, and chances from bits and losses.40 min
  109. One neuron, start to finish FreeWeighted sum, activation, loss, slopes and update, by hand and in code, and what one straight line can and can't split.45 min
  110. A layer and a small network as matrices FreeTwo layers on a batch, the shape of every matrix, adding a hidden unit, and XOR solved.45 min
  111. Training a small network FreeThe backward pass in matrix form, gradient checks, a full training loop, and reading loss curves.45 min
  112. The same network in PyTorch Freenn.Sequential with the NumPy weights loaded, the same loss and slopes, then SGD and mini-batches.45 min
  113. Attention as matrix math FreeScaled dot-product scores, the causal mask, a row-by-row softmax, and blending value vectors.45 min
  114. Residual connections and layer norm FreeAdding the input back and why its slope keeps learning alive, layer norm by hand, and add and norm.40 min
  115. A transformer block, start to finish FreeA sentence through embeddings, attention, add and norm, a small network and next-word scores, every shape named.45 min
  116. Planning a model: rearranging the formulas of a working network FreeLayer widths from parameter counts, sizes from memory and compute, the attention scale, and a training plan.40 min
  117. Final test: transpose any formula FreeSlips to fix from every topic, mixed practice of every kind of rearranging, the strategy chart and the full sheet.45 min
  118. Capstone: a sign reader from the formulas FreeBuild, train and test a trail-sign classifier with NumPy alone, and report its accuracy with an error bar.45 min
Part of a free course 118 lessons

The Foundation Math For Ai & ML: from arithmetic to attention

Learn the math behind AI from arithmetic up: numbers, algebra, vectors and matrices, calculus, probability and statistics, each shown as numbers moving and checked in Python, ending with a network you build and train from the formulas.

Never trained an AI? Teach a small one to write, right in your browser. It takes about 20 seconds and needs no account. Teach it to write