Build a neural network from scratch with NumPy
A two-layer classifier for trail signs written straight from the formulas: forward pass, cross-entropy, a backward pass, training, an honest test score with its error bar, and next steps in PyTorch.
Build, train and test a trail-sign classifier with NumPy alone, and report its accuracy with an error bar.
This is the Foundation's final project, and it uses nearly every module. Vectors and matrices hold the signs and the weights (Module 4); the forward pass is matrix products, relu and softmax (Modules 4, 5 and 7); the loss is cross-entropy (Module 7); the backward pass is the chain rule along many paths (Module 5); the update is gradient descent (Module 5); and the honest verdict, an accuracy with an error bar, is Module 7's statistics. Nothing here is new. What's new is doing all of it at once, from an empty file, then explaining the result as an engineer would: what was built, how well it works, how sure that number is, and where it goes wrong. The guided steps below build one version together, then rebuild it in PyTorch; the brief at the end asks you to build your own and write it up.
An error bar: how far 0.9 could wobble
30 signs is a small test. Lesson 98's standard error, √(0.9 × 0.1 ÷ 30) ≈ 0.055, puts the true accuracy roughly between 0.79 and 1.
Purpose. Build, train and test a sign reader, then report it honestly.
A trail-sign camera describes each sign by three numbers between about 0 and 1: how pointed its shape is, how wide and how tall. Arrows are pointed, tent-site signs are wide, and trail markers are tall, but real signs are worn, tilted and half-hidden, so the numbers are noisy and the kinds overlap. The task: tell the three kinds apart.
The model is lesson 110's network: 3 inputs, 8 hidden relu units, 3 classes, a softmax and the cross-entropy loss. Training is lesson 111's loop. The verdict is lesson 98's error bar. Six steps follow: the data, the forward pass and loss, the backward pass with its check, training, the report, and the same reader in PyTorch. Each step names the lesson it comes from.
0.055
√(0.9 × 0.1 ÷ 30) = √0.003 ≈ 0.055. A 95% range reaches 1.96 of those either side.
Purpose. Make 120 signs, keep 30 aside, and never look at those 30 until the end.
The code below invents 40 signs of each kind around three typical profiles, arrow (0.9, 0.2, 0.1), tent (0.2, 0.9, 0.3) and trail (0.1, 0.3, 0.9), with noise of standard deviation 0.3 added to every number (lesson 93's normal distribution). The first 90 signs train the model; the last 30 are the test set. The labels become one-hot rows of Y, one 1 per row, so the loss can pick out each right class's chance (lesson 110).
import numpy as np rng = np.random.default_rng(11) centers = np.array([[0.9, 0.2, 0.1], [0.2, 0.9, 0.3], [0.1, 0.3, 0.9]]) X = np.zeros((120, 3)) y = np.zeros(120, dtype=int) for i in range(120): X[i] = centers[i % 3] + rng.normal(0, 0.3, 3) y[i] = i % 3 Xtr, ytr, Xte, yte = X[:90], y[:90], X[90:], y[90:] print(Xtr.shape, Xte.shape, X[0].round(2).tolist(), y[:6].tolist())
The very first sign, an arrow, came out (0.91, 0.61, 0.47): pointed, but wider and taller than its profile.
import numpy as np y = np.array([0, 2, 1]) Y = np.zeros((3, 3)) for i in range(3): Y[i, y[i]] = 1.0 print(Y.tolist())
[[1.0, 0.0, 0.0], [0.0, 0.0, 1.0], [0.0, 1.0, 0.0]]
Each row gets a single 1, in the column of its label: 0, then 2, then 1.
Purpose. Turn the 90 training signs into chances and one number for how wrong they are.
Start the weights small and random, W₁ (3 × 8) and W₂ (8 × 3) from a seeded generator, with zero biases. The forward pass is lesson 110's: Z₁ = XW₁ + b₁, A₁ = relu(Z₁), Z₂ = A₁W₂ + b₂, P = softmax(Z₂) row by row. The loss is the average of −log of each sign's right-class chance. Before any training it should be close to log 3 ≈ 1.099, the loss of a model that guesses a third for every class; a starting loss far from that usually means a bug.
The picture draws the whole project as boxes, each matrix's shape on the arrow it travels along: forward along the top, the slopes back along the bottom. Its PyTorch tab is the code of Step 6.
Write forward(X, W1, b1, W2, b2) returning Z1, A1 and P, and loss(P, Y). Then set start to the loss on the training signs at the starting weights (seed 2), rounded to 3.
import numpy as np rng = np.random.default_rng(11) centers = np.array([[0.9, 0.2, 0.1], [0.2, 0.9, 0.3], [0.1, 0.3, 0.9]]) X = np.zeros((120, 3)) y = np.zeros(120, dtype=int) for i in range(120): X[i] = centers[i % 3] + rng.normal(0, 0.3, 3) y[i] = i % 3 Xtr, ytr = X[:90], y[:90] Y = np.zeros((90, 3)) for i in range(90): Y[i, ytr[i]] = 1.0 r = np.random.default_rng(2) W1, b1 = r.normal(0, 0.5, (3, 8)), np.zeros(8) W2, b2 = r.normal(0, 0.5, (8, 3)), np.zeros(3) def forward(X, W1, b1, W2, b2): return 0, 0, 0 def loss(P, Y): return 0.0 start = 0 print(start)
Write a sentence or two, then compare it with a good answer in the lesson.
Purpose. Get every weight's slope, and prove the code right before training on it.
Lesson 111's five lines: dZ₂ = (P − Y)/n, dW₂ = A₁ᵀdZ₂, dA₁ = dZ₂W₂ᵀ, dZ₁ = dA₁ where Z₁ > 0, dW₁ = XᵀdZ₁, with each bias's slope the column sums of its dZ. Then a gradient check: nudge one weight of W₁ up and down by 0.000001, recompute the loss each time, and compare (L₊ − L₋)/0.000002 with the backward pass's number. If they agree to six or more digits, the backward pass is right; if not, no amount of training will save it.
A stray factor of 2, such as a wrong n
A slope off by exactly the same factor everywhere points to a constant: a wrong batch size or a missing ½.
Purpose. Run the loop until the loss stops falling, printing it as it goes.
Run 500 steps at learning rate 0.5 on the 90 training signs, printing the loss every 100 steps and, at the end, how many training signs the network reads right.
Train the network on the 90 signs and print the loss every 100 steps.
import numpy as np rng = np.random.default_rng(11) centers = np.array([[0.9, 0.2, 0.1], [0.2, 0.9, 0.3], [0.1, 0.3, 0.9]]) X = np.zeros((120, 3)) y = np.zeros(120, dtype=int) for i in range(120): X[i] = centers[i % 3] + rng.normal(0, 0.3, 3) y[i] = i % 3 Xtr, ytr = X[:90], y[:90] Y = np.zeros((90, 3)) for i in range(90): Y[i, ytr[i]] = 1.0 r = np.random.default_rng(2) W1, b1 = r.normal(0, 0.5, (3, 8)), np.zeros(8) W2, b2 = r.normal(0, 0.5, (8, 3)), np.zeros(3) for step in range(501): Z1 = Xtr @ W1 + b1 A1 = np.maximum(Z1, 0) Z2 = A1 @ W2 + b2 E = np.exp(Z2 - Z2.max(axis=1, keepdims=True)) P = E / E.sum(axis=1, keepdims=True) if step % 100 == 0: print(step, round(float(-np.mean(np.sum(Y * np.log(P), axis=1))), 3)) dZ2 = (P - Y) / 90 dZ1 = (dZ2 @ W2.T) * (Z1 > 0) W2, b2 = W2 - 0.5 * (A1.T @ dZ2), b2 - 0.5 * dZ2.sum(axis=0) W1, b1 = W1 - 0.5 * (Xtr.T @ dZ1), b1 - 0.5 * dZ1.sum(axis=0) print("training accuracy", round(float(np.mean(np.argmax(P, axis=1) == ytr)), 3))
Every line is one you've written in an earlier lesson: the forward pass of lesson 110, the backward lines of lesson 111, the update of lesson 79. The loop runs to step 500 so the last line printed is the loss after 500 updates, and the accuracy is read from the chances P of that same pass. Two choices were made by trying, and a report should say so: 8 hidden units (4 reached about the same training loss, 0.134, so either would do) and a learning rate of 0.5 (0.05 ended the same 500 steps at 0.205, still higher). Neither choice looked at the test signs.
Plot the curve. The loss column is the curve that goes in the report, as a picture. This cell types in the six losses the run printed:
import matplotlib.pyplot as plt steps = [0, 100, 200, 300, 400, 500] losses = [1.162, 0.158, 0.144, 0.141, 0.139, 0.138] plt.plot(steps, losses) plt.xlabel("step") plt.ylabel("training loss") plt.show()
It drops fast, then flattens: lesson 111's "about right". It doesn't reach 0, and it shouldn't: some signs are so worn that their numbers sit inside another kind's cloud, and no smooth boundary gets all of them. On the training signs the network ends 94.4% right, 0.944.
Some signs sit in another kind's cloud
Noisy, overlapping data has a floor. Driving the loss to 0 would mean memorizing the noise, which tends to hurt on new signs.
Train as in the example, then score the 30 held-back signs once. Set acc to the test accuracy, low and high to the 95% range acc ± 1.96·√(acc(1 − acc)/30), rounded to 3 (cap high at 1), and wrong to the positions of the misread test signs as a list.
import numpy as np rng = np.random.default_rng(11) centers = np.array([[0.9, 0.2, 0.1], [0.2, 0.9, 0.3], [0.1, 0.3, 0.9]]) X = np.zeros((120, 3)) y = np.zeros(120, dtype=int) for i in range(120): X[i] = centers[i % 3] + rng.normal(0, 0.3, 3) y[i] = i % 3 Xtr, ytr, Xte, yte = X[:90], y[:90], X[90:], y[90:] Y = np.zeros((90, 3)) for i in range(90): Y[i, ytr[i]] = 1.0 r = np.random.default_rng(2) W1, b1 = r.normal(0, 0.5, (3, 8)), np.zeros(8) W2, b2 = r.normal(0, 0.5, (8, 3)), np.zeros(3) for step in range(500): Z1 = Xtr @ W1 + b1 A1 = np.maximum(Z1, 0) Z2 = A1 @ W2 + b2 E = np.exp(Z2 - Z2.max(axis=1, keepdims=True)) P = E / E.sum(axis=1, keepdims=True) dZ2 = (P - Y) / 90 dZ1 = (dZ2 @ W2.T) * (Z1 > 0) W2, b2 = W2 - 0.5 * (A1.T @ dZ2), b2 - 0.5 * dZ2.sum(axis=0) W1, b1 = W1 - 0.5 * (Xtr.T @ dZ1), b1 - 0.5 * dZ1.sum(axis=0) acc = 0 low = 0 high = 0 wrong = [] print(acc, low, high, wrong)
Write a sentence or two, then compare it with a good answer in the lesson.
Purpose. Say what was built, how well it works, how sure that is, and where it fails.
The verdict. 27 of the 30 held-back signs are read correctly: an accuracy of 0.9, with a 95% range from about 0.79 to 1. With only 30 test signs, "between 0.8 and 1" is all the data can honestly support; a model scoring 0.85 on the same 30 signs couldn't be called worse.
acc = 0.9 se = (acc * (1 - acc) / 30) ** 0.5 print(round(se, 4), round(acc - 1.96 * se, 3))
0.0548 0.793
√(0.09 ÷ 30) ≈ 0.0548, and 0.9 − 1.96 × 0.0548 ≈ 0.793: the low end of the 95% range.
The three mistakes. Test sign 11, a trail marker, has features (1.19, −0.08, 1.0): tall, as it should be, but the noise made it as pointed as any arrow, and the network said arrow with 0.89. Sign 25, a tent-site sign, came out (0.01, 0.06, 0.61): its width was wiped out entirely, and the network said trail with 0.99. Sign 29, another trail marker, is (0.65, 0.73, 0.55), middling on all three, and was called tent with 0.93. Each mistake is a sign whose numbers really do look like another kind, and each was made with high confidence, which says the network has never been shown how unreliable a single feature can be. More worn examples in training would help more than a bigger network.
Where the mistakes fall. A confusion table (lesson 103) counts each test sign into a row for its true kind and a column for the kind the reader said. The test signs run arrow, tent, trail, ten of each, and the reader matched them everywhere except at signs 11, 25 and 29, so the cell can rebuild its answers from the drill's:
import numpy as np yte = np.arange(90, 120) % 3 pred = yte.copy() pred[11], pred[25], pred[29] = 0, 2, 1 table = np.zeros((3, 3), dtype=int) for i in range(30): table[yte[i], pred[i]] = table[yte[i], pred[i]] + 1 print(table.tolist())
The picture shades each count; press Next to move from the signs read right to the three that weren't.
Against a simpler reader. A score means little alone, so the report also compares the network with a baseline: the same signs read by one layer with no hidden units, trained the same 500 steps. Lesson 99 showed how: score both readers on every held-back sign, take the difference per sign (+1 where only the network is right, −1 where only the baseline is, 0 where they agree) and put a 95% range around its mean. A range that includes 0 says the data can't separate the two.
A finished report holds five things: the accuracy with its range, the confusion table, the mistakes with their confidence, this interval for the difference from the baseline, and the paragraph comparing NumPy with PyTorch below.
Train both readers from the same signs, then compare them sign by sign on the 30 held-back ones.
import numpy as np rng = np.random.default_rng(11) centers = np.array([[0.9, 0.2, 0.1], [0.2, 0.9, 0.3], [0.1, 0.3, 0.9]]) X = np.zeros((120, 3)) y = np.zeros(120, dtype=int) for i in range(120): X[i] = centers[i % 3] + rng.normal(0, 0.3, 3) y[i] = i % 3 Xtr, ytr, Xte, yte = X[:90], y[:90], X[90:], y[90:] Y = np.zeros((90, 3)) for i in range(90): Y[i, ytr[i]] = 1.0 def chances(Z): E = np.exp(Z - Z.max(axis=1, keepdims=True)) return E / E.sum(axis=1, keepdims=True) r = np.random.default_rng(2) W1, b1 = r.normal(0, 0.5, (3, 8)), np.zeros(8) W2, b2 = r.normal(0, 0.5, (8, 3)), np.zeros(3) V, c = np.random.default_rng(3).normal(0, 0.5, (3, 3)), np.zeros(3) for step in range(500): Z1 = Xtr @ W1 + b1 A1 = np.maximum(Z1, 0) P = chances(A1 @ W2 + b2) dZ2 = (P - Y) / 90 dZ1 = (dZ2 @ W2.T) * (Z1 > 0) W1, b1 = W1 - 0.5 * Xtr.T @ dZ1, b1 - 0.5 * dZ1.sum(axis=0) W2, b2 = W2 - 0.5 * A1.T @ dZ2, b2 - 0.5 * dZ2.sum(axis=0) dZ = (chances(Xtr @ V + c) - Y) / 90 V, c = V - 0.5 * Xtr.T @ dZ, c - 0.5 * dZ.sum(axis=0) reader = np.argmax(np.maximum(Xte @ W1 + b1, 0) @ W2 + b2, axis=1) baseline = np.argmax(Xte @ V + c, axis=1) d = (reader == yte).astype(float) - (baseline == yte).astype(float) se = d.std() / len(d) ** 0.5 print("right:", int(np.sum(reader == yte)), "and", int(np.sum(baseline == yte))) print("signs they read differently:", int(np.sum(reader != baseline))) low, high = d.mean() - 1.96 * se, d.mean() + 1.96 * se print("mean difference", round(float(d.mean()), 3), "range", round(float(low), 3), round(float(high), 3))
Both readers get 27 of 30 and read every sign the same way, so the difference is 0 on all 30 signs and its range is 0 to 0. That does not prove the hidden layer is useless: it says these 30 signs show no gap. The three signs both readers miss, 11, 25 and 29, sit in the overlap of the kinds, which more hidden units can't unmix. To claim a gain you would need signs the baseline gets wrong and the network gets right.
Purpose. Rebuild the reader in a few lines of PyTorch, check it reaches the same verdict, then train it in mini-batches.
Lesson 112 showed that nn.Sequential, loaded with your NumPy weights, is the same network, number for number. Do it for the reader: the seed-2 starting weights, turned around into nn.Linear's one-row-per-output layout, zero biases, and optim.SGD at rate 0.5 in place of your update lines. If the two agree, every number in your report can come from either.
Train the PyTorch reader from the same start, then test it once on the 30 held-back signs.
import numpy as np import torch import torch.nn as nn import torch.nn.functional as F rng = np.random.default_rng(11) centers = np.array([[0.9, 0.2, 0.1], [0.2, 0.9, 0.3], [0.1, 0.3, 0.9]]) X = np.zeros((120, 3)) y = np.zeros(120, dtype=int) for i in range(120): X[i] = centers[i % 3] + rng.normal(0, 0.3, 3) y[i] = i % 3 Xtr, ytr = torch.tensor(X[:90], dtype=torch.float32), torch.tensor(y[:90]) Xte, yte = torch.tensor(X[90:], dtype=torch.float32), torch.tensor(y[90:]) r = np.random.default_rng(2) W1, W2 = r.normal(0, 0.5, (3, 8)), r.normal(0, 0.5, (8, 3)) model = nn.Sequential(nn.Linear(3, 8), nn.ReLU(), nn.Linear(8, 3)) with torch.no_grad(): model[0].weight.copy_(torch.tensor(W1.T)) model[2].weight.copy_(torch.tensor(W2.T)) model[0].bias.zero_() model[2].bias.zero_() opt = torch.optim.SGD(model.parameters(), lr=0.5) for step in range(500): opt.zero_grad() loss = F.cross_entropy(model(Xtr), ytr) if step % 100 == 0: print(step, round(loss.item(), 3)) loss.backward() opt.step() with torch.no_grad(): pred = model(Xte).argmax(dim=1) wrong = [] for i in range(30): if pred[i] != yte[i]: wrong.append(i) print("test accuracy", round((pred == yte).float().mean().item(), 3), "misread:", wrong)
The loss column is Step 4's, to three places: 1.162, 0.158, 0.144, 0.141, 0.139. The verdict is Step 5's: 0.9, with signs 11, 25 and 29 misread. Four lines of loop did the work of your forward pass, softmax, loss, five backward lines and update. The match is to three places, not every digit, because PyTorch keeps its numbers in float32.
Mini-batches. Real models never see all their data in one step. A DataLoader deals the 90 training signs in batches, and each step works on one batch alone. In batches of 9 an epoch is 10 steps, so 50 epochs make 500 steps, as many as the full-batch run, each on a tenth of the signs.
Train the PyTorch reader in batches of 9 for 50 epochs, from the same start. Set loader, steps (the number of updates), final (the loss on all 90 training signs afterward, rounded to 3), acc (the test accuracy, rounded to 3) and wrong (the positions of the misread test signs).
import numpy as np import torch import torch.nn as nn import torch.nn.functional as F from torch.utils.data import DataLoader, TensorDataset rng = np.random.default_rng(11) centers = np.array([[0.9, 0.2, 0.1], [0.2, 0.9, 0.3], [0.1, 0.3, 0.9]]) X = np.zeros((120, 3)) y = np.zeros(120, dtype=int) for i in range(120): X[i] = centers[i % 3] + rng.normal(0, 0.3, 3) y[i] = i % 3 Xtr, ytr = torch.tensor(X[:90], dtype=torch.float32), torch.tensor(y[:90]) Xte, yte = torch.tensor(X[90:], dtype=torch.float32), torch.tensor(y[90:]) r = np.random.default_rng(2) W1, W2 = r.normal(0, 0.5, (3, 8)), r.normal(0, 0.5, (8, 3)) model = nn.Sequential(nn.Linear(3, 8), nn.ReLU(), nn.Linear(8, 3)) with torch.no_grad(): model[0].weight.copy_(torch.tensor(W1.T)) model[2].weight.copy_(torch.tensor(W2.T)) model[0].bias.zero_() model[2].bias.zero_() opt = torch.optim.SGD(model.parameters(), lr=0.5) loader = None steps = 0 final = 0 acc = 0 wrong = [] print(steps, final, acc, wrong)
For the report: NumPy against PyTorch. One paragraph is enough, and it can read like this. The PyTorch reader is the NumPy reader in a few lines: from the same starting weights it printed the same losses to three places and reached the same verdict, 27 of 30, with signs 11, 25 and 29 misread. Training in batches of 9 changed the path, not the verdict: the training loss ended higher, 0.214 against 0.138, because each step followed the slope of 9 signs instead of all 90, but the same three test signs were misread. Say which version made each number in your report, and that the two were checked against each other.
Write a sentence or two, then compare it with a good answer in the lesson.
Purpose. Every formula of Module 8 in one place, with what it says and its Python line.
| lesson | formula | in plain words | in Python |
|---|---|---|---|
| 109 | z = w·x + b; a = relu(z); ∂L/∂wⱼ = ∂L/∂z · xⱼ | one neuron: sum, squeeze, blame each weight by its input | dz * x |
| 110 | Z₁ = XW₁ + b₁; A₁ = relu(Z₁); Z₂ = A₁W₂ + b₂; P = softmax(Z₂) | a whole batch through two layers | np.maximum(X @ W1 + b1, 0) @ W2 + b2 |
| 111 | dZ₂ = (P − Y)/n; dW₂ = A₁ᵀdZ₂; dZ₁ = dZ₂W₂ᵀ where Z₁ > 0; dW₁ = XᵀdZ₁ | the backward pass, shape by shape | A1.T @ dZ2 |
| 113 | softmax(QKᵀ/√d + M)V | score, scale, mask, share out, blend | A @ V |
| 114 | y = x + f(x); LN(x) = γ·(x − μ)/√(σ² + ε) + β | add the input back, then recenter and rescale each vector | x + f(x), g * (x - mu) / (var + eps) ** 0.5 + b |
| 115 | h = x + Attn(LN(x)); out = h + MLP(LN(h)) | attention, then a small network, each added back | h = x + attn(ln1(x)); h + mlp(ln2(h)) |
| 112 | the same two-layer network, as modules | the NumPy formulas, named once each | nn.Sequential(nn.Linear(3, 4), nn.ReLU(), nn.Linear(4, 3)) |
| 118 | acc ± 1.96·√(acc(1 − acc)/n) | an honest test score | (acc * (1 - acc) / n) ** 0.5 |
Write a sentence or two, then compare it with a good answer in the lesson.
You started with place value and the four operations. You've now written, from formulas you can derive, a network that reads signs: vectors and matrices to hold the data and weights, derivatives and the chain rule to train it, probability to turn scores into chances, and statistics to say how far to trust the result. Every model later in the academy, up to a transformer, is these same pieces at a larger scale: more layers, more data, attention between the inputs, and the same loop of forward pass, loss, backward pass and update. When a later lesson writes loss.backward(), you know what the five lines inside it do.
Build a three-class trail-sign reader with NumPy alone, from an empty file, and write it up in one page. The starter makes a fresh set of 150 signs with its own noise and seed, so your numbers will be your own. Treat the last 30 signs as the test set from the start and look at them once, at the very end.
Your report should read as if for a colleague who knows the math but hasn't seen your code: what you built, the choices you made and why, how well it works, how sure that number is, and where it fails.
import numpy as np def make_signs(seed, n_per_kind, noise): """Signs described by (pointed, wide, tall). Kinds: 0 arrow, 1 tent site, 2 trail marker.""" rng = np.random.default_rng(seed) centers = np.array([[0.9, 0.2, 0.1], [0.2, 0.9, 0.3], [0.1, 0.3, 0.9]]) n = 3 * n_per_kind X = np.zeros((n, 3)) y = np.zeros(n, dtype=int) for i in range(n): X[i] = centers[i % 3] + rng.normal(0, noise, 3) y[i] = i % 3 return X, y X, y = make_signs(seed=86, n_per_kind=50, noise=0.32) Xtr, ytr, Xte, yte = X[:120], y[:120], X[120:], y[120:] print(Xtr.shape, Xte.shape, "kinds in the test set:", np.bincount(yte).tolist())
Write a sentence or two, then compare it with a good answer in the lesson.
Why must the test signs be set aside before any training or tuning?
575 lessons in thirteen phases. 176 are free without an account, and a free account opens the rest. See every lesson.
A two-layer classifier for trail signs written straight from the formulas: forward pass, cross-entropy, a backward pass, training, an honest test score with its error bar, and next steps in PyTorch.
Learn the math behind AI from arithmetic up: numbers, algebra, vectors and matrices, calculus, probability and statistics, each shown as numbers moving and checked in Python, ending with a network you build and train from the formulas.
Never trained an AI? Teach a small one to write, right in your browser. It takes about 20 seconds and needs no account. Teach it to write