Index Ai Academy
Project tutorial · 7 steps

Build a neural network from scratch with NumPy

A two-layer classifier for trail signs written straight from the formulas: forward pass, cross-entropy, a backward pass, training, an honest test score with its error bar, and next steps in PyTorch.

Free to read, with no account. The code comes from the lesson Capstone: a sign reader from the formulas, and has been run.

The steps9

What you'll build

Libraries hide a neural network behind one call. This tutorial writes every line of one instead, from formulas you can check by hand. The network reads trail signs from three numbers, how pointed, wide and tall each one is, and sorts them into arrows, tent-site signs and trail markers. The data is noisy on purpose, so the network can't be perfect, and the last step reports how good it really is, with a range rather than one flattering number.

What you'll learn

  • Build a trail-sign classifier with NumPy alone: data, a two-layer network, softmax and cross-entropy
  • Train it with a backward pass you have checked numerically, and plot its loss curve
  • Test it once on held-back signs and report the accuracy with a 95% error bar
  • Study its wrong predictions in a confusion table, and write a one-page report naming every formula and shape you used
  • Rebuild the same reader in PyTorch, check it reaches the same verdict, and train it on mini-batches

What you need

Python and NumPy: arrays, matrix products with @, and broadcasting. In the lesson everything runs in your browser. On your own machine you need pip install numpy. There are no model calls, so it costs nothing and gives the same result every time.

Step 1: Turn labels into one-hot rows

The loss needs, for every sign, a row with a 1 in the column of its true kind and 0s elsewhere. With that row, multiplying by the log-chances and summing picks out the log-chance of the right answer, with no special cases.

import numpy as np

def one_hot(y, k):
    Y = np.zeros((len(y), k))
    for i in range(len(y)):
        Y[i, y[i]] = 1.0
    return Y

Step 2: Write the forward pass

Two layers: a weighted sum and a relu, then another weighted sum. A softmax turns each row of scores into chances. Subtracting each row's largest score before np.exp changes nothing in the result, but stops large scores from overflowing to infinity.

def forward(X, W1, b1, W2, b2):
    Z1 = X @ W1 + b1
    A1 = np.maximum(Z1, 0)
    Z2 = A1 @ W2 + b2
    E = np.exp(Z2 - Z2.max(axis=1, keepdims=True))
    return Z1, A1, E / E.sum(axis=1, keepdims=True)

Step 3: Write the backward pass

Five lines carry the loss's slope back to every weight. They start from P − Y, the slope of softmax followed by cross-entropy, and each slope has exactly the shape of its weight, which is what decides where each transpose goes. The (Z1 > 0) mask lets the slope through only where relu was on.

def backward(X, Y, Z1, A1, P, W2):
    dZ2 = (P - Y) / len(X)
    dZ1 = (dZ2 @ W2.T) * (Z1 > 0)
    return X.T @ dZ1, dZ1.sum(axis=0), A1.T @ dZ2, dZ2.sum(axis=0)

Rather write it yourself?

In the lesson you write this from a starter, in your browser with nothing to install, and checks tell you what's wrong as you go. It needs no account.

Step 4: Score honestly, and list the mistakes

An accuracy measured on a few dozen signs is a sample, so it gets a 95% range: 1.96 standard errors either side, capped at 1. Listing the misread signs is what turns a score into something you can learn from.

def accuracy_range(acc, n):
    se = (acc * (1 - acc) / n) ** 0.5
    return (round(acc - 1.96 * se, 3), round(min(1.0, acc + 1.96 * se), 3))

def mistakes(pred, truth):
    out = []
    for i in range(len(pred)):
        if pred[i] != truth[i]:
            out.append(i)
    return out

Step 5: Make the signs and split them once

120 signs are invented around three typical profiles with plenty of noise. The first 90 train the network; the last 30 are set aside before any training and touched only at the very end.

rng = np.random.default_rng(11)
centers = np.array([[0.9, 0.2, 0.1], [0.2, 0.9, 0.3], [0.1, 0.3, 0.9]])
X = np.zeros((120, 3))
y = np.zeros(120, dtype=int)
for i in range(120):
    X[i] = centers[i % 3] + rng.normal(0, 0.3, 3)
    y[i] = i % 3
Xtr, ytr, Xte, yte = X[:90], y[:90], X[90:], y[90:]
Y = one_hot(ytr, 3)

Step 6: Train, watching the loss

Small random starting weights, then 500 rounds of forward pass, backward pass and update with a learning rate of 0.5. The loss is printed every 100 steps: it should start near log 3 ≈ 1.1, the loss of guessing a third for each kind, then fall fast and flatten.

r = np.random.default_rng(2)
W1, b1 = r.normal(0, 0.5, (3, 8)), np.zeros(8)
W2, b2 = r.normal(0, 0.5, (8, 3)), np.zeros(3)
for step in range(500):
    Z1, A1, P = forward(Xtr, W1, b1, W2, b2)
    if step % 100 == 0:
        print(step, round(float(-np.mean(np.sum(Y * np.log(P), axis=1))), 3))
    dW1, db1, dW2, db2 = backward(Xtr, Y, Z1, A1, P, W2)
    W1, b1, W2, b2 = W1 - 0.5 * dW1, b1 - 0.5 * db1, W2 - 0.5 * dW2, b2 - 0.5 * db2

Step 7: Test once, and report

27 of the 30 held-back signs are read correctly, an accuracy of 0.9, and with only 30 test signs the honest claim is "somewhere between about 0.8 and 1". The three misread signs each have numbers that really do look like another kind: noise wiped out one's width and made another as pointed as an arrow.

pred = np.argmax(forward(Xte, W1, b1, W2, b2)[2], axis=1)
acc = float(np.mean(pred == yte))
print("test accuracy", acc, "95% range", accuracy_range(acc, 30))
print("misread:", mistakes(pred, yte))
What it prints
0 1.162
100 0.158
200 0.144
300 0.141
400 0.139
test accuracy 0.9 95% range (0.793, 1.0)
misread: [11, 25, 29]

Where to take it next

  • Check the backward pass with a gradient check: nudge one weight both ways and compare the change in the loss with the computed slope.
  • Carve a validation set from the training signs and use it to choose the hidden size and learning rate.
  • Print a 3 × 3 confusion matrix to see which kinds get mixed up.
  • Add a second hidden layer, and see whether the test score moves outside its error bar.
  • Rebuild the same reader in PyTorch with nn.Sequential, load your NumPy weights, and check it reaches the same 27 of 30; then train it in mini-batches with a DataLoader.
  • Compare the network with a one-layer baseline: score both on every held-back sign and put a 95% range around the mean difference per sign.

Mistakes to watch for

  • Looking at the test set while choosing the hidden size or learning rate.
  • Reporting an accuracy without an error bar from a small test set.
  • Training until the loss reaches 0 on noisy data.
  • Trusting a backward pass without a gradient check.
  • Counting mistakes without looking at what they have in common.
The complete program
import numpy as np

def one_hot(y, k):
    Y = np.zeros((len(y), k))
    for i in range(len(y)):
        Y[i, y[i]] = 1.0
    return Y

def forward(X, W1, b1, W2, b2):
    Z1 = X @ W1 + b1
    A1 = np.maximum(Z1, 0)
    Z2 = A1 @ W2 + b2
    E = np.exp(Z2 - Z2.max(axis=1, keepdims=True))
    return Z1, A1, E / E.sum(axis=1, keepdims=True)

def backward(X, Y, Z1, A1, P, W2):
    dZ2 = (P - Y) / len(X)
    dZ1 = (dZ2 @ W2.T) * (Z1 > 0)
    return X.T @ dZ1, dZ1.sum(axis=0), A1.T @ dZ2, dZ2.sum(axis=0)

def accuracy_range(acc, n):
    se = (acc * (1 - acc) / n) ** 0.5
    return (round(acc - 1.96 * se, 3), round(min(1.0, acc + 1.96 * se), 3))

def mistakes(pred, truth):
    out = []
    for i in range(len(pred)):
        if pred[i] != truth[i]:
            out.append(i)
    return out

rng = np.random.default_rng(11)
centers = np.array([[0.9, 0.2, 0.1], [0.2, 0.9, 0.3], [0.1, 0.3, 0.9]])
X = np.zeros((120, 3))
y = np.zeros(120, dtype=int)
for i in range(120):
    X[i] = centers[i % 3] + rng.normal(0, 0.3, 3)
    y[i] = i % 3
Xtr, ytr, Xte, yte = X[:90], y[:90], X[90:], y[90:]
Y = one_hot(ytr, 3)

r = np.random.default_rng(2)
W1, b1 = r.normal(0, 0.5, (3, 8)), np.zeros(8)
W2, b2 = r.normal(0, 0.5, (8, 3)), np.zeros(3)
for step in range(500):
    Z1, A1, P = forward(Xtr, W1, b1, W2, b2)
    if step % 100 == 0:
        print(step, round(float(-np.mean(np.sum(Y * np.log(P), axis=1))), 3))
    dW1, db1, dW2, db2 = backward(Xtr, Y, Z1, A1, P, W2)
    W1, b1, W2, b2 = W1 - 0.5 * dW1, b1 - 0.5 * db1, W2 - 0.5 * dW2, b2 - 0.5 * db2

pred = np.argmax(forward(Xte, W1, b1, W2, b2)[2], axis=1)
acc = float(np.mean(pred == yte))
print("test accuracy", acc, "95% range", accuracy_range(acc, 30))
print("misread:", mistakes(pred, yte))

Write it yourself

In the lesson you write this from a starter, in your browser with nothing to install, and checks tell you what's wrong as you go. It needs no account.

Keep building

More tutorials.

This is one stage of the AI engineer roadmap: the bigger picture, in order.

5 steps

Build an AI ad-copy reviewer in Python

A small Python tool that checks ad headlines against your rules, saves a JSON report, and asks a language model to rewrite the worst one.

3 steps

Train a neural network classifier in PyTorch

Standardize from the training rows only, train a small network with a validation score after every epoch, and compare it fairly with logistic regression.

6 steps

Build a GPT from scratch in Python

Write a tiny GPT in PyTorch, train it on a product catalog, and watch it write new descriptions. The whole program is about 70 lines.

9 steps

Build a REST API in Python from scratch

Authentication, validation, pagination, idempotency and clean errors, in a working API built with the standard library so you can see how each one works.

6 steps

Build an MCP server in Python from scratch

A working Model Context Protocol server with tools, resources and prompts, written without an SDK so you can see exactly what goes over the wire.

7 steps

Build a coding agent in Python

An agent that reads and edits a project, runs its tests and stops when they pass, with the safety checks that keep it inside the project.

8 steps

Ship a production AI service in Python

API keys, rate limits, validation, caching, model fallbacks, structured logs and an eval, around a service that classifies support tickets.

4 steps

Ship a small machine learning library in Python

The core of a typed, tested training package: a config that rejects unknown settings, mini-batches as a generator, a seeded training loop and an evaluation function.

Never trained an AI? Teach a small one to write, right in your browser. It takes about 20 seconds and needs no account. Teach it to write