Lesson 109's single neuron could only draw one straight line, so it failed at XOR. The fix is to put several neurons side by side, a layer, and feed their outputs into another layer. Written one neuron at a time, that's a tangle of sums; written as matrices, it's two matrix products (lesson 56) with a relu and a softmax between them, and a whole batch of examples goes through at once. That is exactly how every network library computes, from this lesson's toy to a language model: a stack of "multiply by a matrix, add a bias, apply an activation", which PyTorch builds with nn.Sequential. The skill that makes it manageable is keeping track of shapes. Every bug in this lesson's code is a shape that doesn't line up, and every shape follows from two rules you already know.
What you'll be able to do
Push a whole batch through two layers: Z₁ = XW₁ + b₁, A₁ = relu(Z₁), Z₂ = A₁W₂ + b₂, P = softmax(Z₂)
Name the shape of every matrix along the way, and compute single entries by hand
Write the forward pass in six lines of NumPy, and add a hidden unit by fixing every shape
See hidden units fit a problem one neuron can't, score a small network on a batch of signs, and build the same network from PyTorch's nn.Sequential
Quick check
Have a guess before you start. A batch of 8 signs with 3 features each goes through a layer with 4 hidden units. What shape is X @ W₁?
3 × 4
8 × 3
4 × 8
8 × 4
Show the answer
8 × 4
X is 8 × 3 and W₁ is 3 × 4; the inner 3s match and the result keeps the outer sizes: one row per sign, one column per hidden unit.
The math: two layers, one batch
Purpose. Compute a small network's predictions for a whole batch, and know every shape on the way.
Stack the batch's examples as the rows of a matrix X, n rows of d features. A layer of h neurons is a weight matrix W₁ of shape d × h, one column per neuron, plus a bias b₁ with h entries. The whole forward pass:
step
formula
shape
hidden scores
Z₁ = X W₁ + b₁
(n × d)(d × h) → n × h
hidden outputs
A₁ = relu(Z₁)
n × h
class scores
Z₂ = A₁ W₂ + b₂
(n × h)(h × k) → n × k
chances
P = softmax(Z₂), row by row
n × k
loss
L = mean of −log P[right class]
one number
Each row of every matrix belongs to one example, all the way through. The bias b₁ is added to every row: lesson 64's broadcasting.
Quick check
From lesson 56: what is the shape of (8 × 3)(3 × 4)?
It can't be multiplied
8 × 3
8 × 4
3 × 3
Show the answer
8 × 4
The inner sizes, 3 and 3, must match; the result has the outer sizes, 8 × 4.
The picture. Here is the table drawn as boxes, for the batch from the opening question: 8 signs, 3 features, 4 hidden units and 3 classes. Each box is one step, and each arrow carries a matrix, labeled with its shape. Press Next, and say each new shape before it appears.
Try it now
XOR, by hand. The four inputs (0, 0), (0, 1), (1, 0) and (1, 1) are the rows of X. A hidden layer of two relu neurons, both with weights (1, 1), and biases 0 and −1: the first counts how many inputs are on, the second fires only when both are. Step through the first layer. The drawing is the whole network, with the examples passing through one at a time: an example's inputs on the left, the two hidden units in the middle, each starting at its bias, and on the right the two class scores, waiting for the second layer.
The second layer reads the hidden outputs. Its weights W₂ = [[0, 4], [0, −8]] and bias b₂ = (2, 0) give two class scores per example, "no" and "yes": the "yes" score is 4 × (inputs on) − 8 × (both on). Step through it. In the drawing, each example now goes all the way: in on the left, through the hidden units, and out as two scores.
In plain words: each layer multiplies by its weights, adds its bias and applies its activation; the last one's scores become chances, and the loss averages −log of the right class's chance over the batch.
For XOR every right answer gets 0.881, so the loss is −log 0.881 = 0.127, against lesson 109's stuck 0.693. One entry by hand checks the shapes: Z₁ for example 3, unit 1, is row 3 of X dotted with column 1 of W₁ plus b₁'s first entry, 1 × 1 + 0 × 1 + 0 = 1. Count the parameters: W₁ has d × h, b₁ has h, W₂ has h × k and b₂ has k. With 3 features, 4 hidden units and 3 classes that's 12 + 4 + 12 + 3 = 31 numbers to learn.
Quick check
A network has 4 hidden units and 3 classes. What shape is W₂?
4 × 3
n × 3
4 × 4
3 × 4
Show the answer
4 × 3
W₂ takes each example's 4 hidden outputs to 3 class scores: (n × 4)(4 × 3) → n × 3.
In your own words
In your own words: how did the hidden layer make XOR solvable when one neuron couldn't do it?
Write a sentence or two, then compare it with a good answer in the lesson.
In Python: the forward pass in six lines
Purpose. Run a batch through the network with matrix products, and keep every shape straight.
It runs right here in your browser, with nothing to install.
Y holds the right answers one-hot, one row per example, so np.sum(Y * np.log(P), axis=1) picks out the log of each right class's chance. Six lines of arithmetic, the whole forward pass, for any batch size.
Code flow. Step through the same lines one at a time. Each array shows as a grid of boxes, like the boxes in the math part's flows, and X, Z1, A1, Z2 and P keep one row per example: Z1 and A1 are the first flow's numbers before and after relu, and Z2 is the second flow's answer. Watch every line keep the batch's four rows until the last one, the loss.
A named mistake: a bias of the wrong length. b₁ must have one entry per hidden unit, h of them, so that broadcasting adds it to every row. A bias with one entry per example, n of them, is wrong, but when n happens to equal h NumPy adds it without complaint, across the wrong axis. Print the shapes after every line the first time: Z₁ should be n × h, and b₁ should have shape (h,).
Worked example
Worked example: shapes at every step
Send a batch of 8 signs with 3 features through 4 hidden units to 3 classes, and print each shape.
The numbers are placeholders; only the shapes matter here. X (8, 3) times W₁ (3, 4) gives Z₁ (8, 4); relu keeps the shape; A₁ (8, 4) times W₂ (4, 3) gives Z₂ (8, 3), one score per sign per class. The last line counts the parameters, 31, all of which training will set. A shape printout like this is the first thing to write when a network's code fails.
Quick check
What does this print?
import numpy as np
X = np.ones((8, 3))
W1 = np.ones((3, 4))
W2 = np.ones((4, 3))
Z1 = X @ W1
Z2 = Z1 @ W2
print(Z1.shape, Z2.shape)
(3, 4) (4, 3)
(8, 4) (4, 3)
(8, 3) (8, 4)
(8, 4) (8, 3)
Show the answer
(8, 4) (8, 3)
(8 × 3)(3 × 4) → (8, 4), and (8 × 4)(4 × 3) → (8, 3): one row per sign throughout.
Drill
Add a hidden unit
Grow the sign network from 4 hidden units to 5 without changing its predictions: give W1b a fifth column of zeros, b1b a fifth bias of 0, and W2b a fifth row of zeros. Then set same to whether the new network's chances match the old ones.
In the lesson, your code is checked as you write it.
In your own words
Adding a hidden unit changed three arrays. Why did W₁ gain a column but W₂ gain a row?
Write a sentence or two, then compare it with a good answer in the lesson.
In AI and ML: what hidden units are for
Purpose. See a small network classify a batch, and read its mistakes.
Each hidden unit draws its own line through the input space and reports which side an example is on. The next layer combines those reports, so the network can carve out regions no single line could: XOR above, or three kinds of trail sign below. More hidden units mean more lines and more complicated regions; more layers let features be built from features.
Quick check
What does a hidden layer give a network that a single neuron lacks?
Several lines, combined into one decision
Fewer parameters to learn overall
A different loss
Faster training, since layers share work
Show the answer
Several lines, combined into one decision
Each hidden unit draws a line; the next layer combines them, which is what XOR needed.
Quick check
What does this print? (XOR's first layer, then its yes-score weights.)
Both inputs on: the first unit counts 2, the second fires with 1, and 4 × 2 − 8 × 1 = 0: no reason to say yes.
In PyTorch. Real code builds this network from ready-made layers. nn.Sequential, the chain you met in lesson 20, feeds each layer's output into the next: nn.Linear(2, 2) is a layer with 2 inputs and 2 outputs that computes XW + b; nn.ReLU() is relu as a layer; and a second nn.Linear(2, 2) makes the two class scores. net[0] and net[2] are the two layers. New layers start with random weights, so the cell loads XOR's hand weights under torch.no_grad(), as in lesson 55. A layer stores its weights as (out, in), the transpose of this lesson's W (lesson 56), so each W goes in as .T.
import torch
import torch.nn as nn
import torch.nn.functional as F
X = torch.tensor([[0.0, 0.0], [0.0, 1.0], [1.0, 0.0], [1.0, 1.0]])
y = torch.tensor([0, 1, 1, 0])
net = nn.Sequential(nn.Linear(2, 2), nn.ReLU(), nn.Linear(2, 2))
with torch.no_grad():
net[0].weight.copy_(torch.tensor([[1.0, 1.0], [1.0, 1.0]]).T)
net[0].bias.copy_(torch.tensor([0.0, -1.0]))
net[2].weight.copy_(torch.tensor([[0.0, 4.0], [0.0, -8.0]]).T)
net[2].bias.copy_(torch.tensor([2.0, 0.0]))
Z2 = net(X)
print(Z2.tolist())
print(Z2.argmax(dim=1).tolist(), round(F.cross_entropy(Z2, y).item(), 3))
The first line is the second flow's Z₂, number for number. F.cross_entropy takes the scores and the right classes, 0 for no and 1 for yes, and does the softmax and the average −log itself: 0.127, the loss the NumPy cell printed. The predictions, 0, 1, 1, 0, are XOR. Mind the .T. W₂ is square, so it also loads without .T, with no error, and the scores come out wrong. Check one row against your hand numbers whenever you load weights.
Quick check
The sign network has 3 features, 4 hidden units and 3 classes. What does this print?
import torch
import torch.nn as nn
net = nn.Sequential(nn.Linear(3, 4), nn.ReLU(), nn.Linear(4, 3))
print(net[0].weight.shape, net[2].weight.shape)
torch.Size([3, 4]) torch.Size([3, 4])
torch.Size([4, 3]) torch.Size([3, 4])
torch.Size([3, 4]) torch.Size([4, 3])
torch.Size([4, 3]) torch.Size([4, 3])
Show the answer
torch.Size([4, 3]) torch.Size([3, 4])
nn.Linear(3, 4) stores (out, in), 4 × 3: one row of 3 weights per hidden unit. nn.Linear(4, 3) stores 3 × 4. Each is the transpose of this lesson's W, which is why the cell above loads W.T.
Eight signs. A sign is described by three features, how pointed it is, how wide and how tall, and a network with 4 hidden units and 3 classes (arrow, tent, trail) has already been trained; its weights, rounded to one decimal, are in the cell below. Run on eight new signs, it gets seven right, an accuracy of 0.875, with an average loss of 0.258. The miss is instructive: sign 7, features (0.5, 0.5, 0.2), is really a tent, and the network says arrow. Here is that one sign going through, with the hidden outputs, the scores and the chances printed:
Two hidden units fire. The second, at 2.29, rises with pointed and falls with tall (its weights are 3.1 and −2.3); the fourth, at 1.19, rises with wide (3.6) and also falls with tall. The arrow score leans on the second unit (weight 3.2), the tent score on the fourth (weight 3.5), and the second unit's bigger value wins: arrow 3.24, tent 1.48, which softmax turns into 0.85 for arrow. Half pointed and half wide, the sign sets off both detectors, and the network is confidently wrong there, a hint that it saw few examples like it. That's what a held-back batch is for.
Drill
Score the network on eight signs
Run the trained network on the eight signs. Set pred to the predicted class of each (0 arrow, 1 tent, 2 trail) as a list, acc to the accuracy, and loss to the average cross-entropy (natural log) with the one-hot Y, rounded to 3.
31Formulas with fractions: clear them firstFreeClearing denominators, cross-multiplying, a letter in the bottom, and reciprocal formulas like 1/R = 1/R₁ + 1/R₂.35 min
37Inverse functions, lines and scalingFreeUndoing a function by making x the subject, domains that allow it, line forms, and unscaling predictions.40 min
38Sequences and sums: find n, a, d and rFreeEvery letter of the arithmetic and geometric formulas as the subject, and an average's window from its decay.40 min
40Review: rearrange anything (algebra)FreeBroken rearrangements to fix, mixed practice from the whole algebra block, a strategy chart, and its formula sheet.35 min
41Coordinates and distanceFreePoints as (x, y), Pythagoras, distance in any number of coordinates, and nearest neighbors.42 min
42Angles, degrees and radiansFreeMeasuring a turn, converting units, arcs, and the angle between two arrows.36 min
43Sine and cosineFreeAcross and up: right triangles, the unit circle, and directions and rotations.42 min
44Waves: height, speed and shiftFreeAmplitude, period and phase, sampling a wave, and position codes in transformers.37 min
68Vector equations: rearranging with arrowsFreeSolving for a vector or a scale, recovering a gradient from two weight snapshots, and an angle from a dot product.40 min
87Counting: choices, orders and groupsFreeThe counting principle, factorials, ordered picks and groups, and why attention's cost grows with the square of the length.43 min
88Probability: from counts to chancesFreeEqually likely outcomes, the complement, distributions, seeded simulation, and a next-word model.43 min
89Combining events: and, or, independentFreeThe or rule, independence, probability trees, drawing with and without replacement, and chains of steps.42 min
100Maximum likelihoodFreeLikelihoods as products, log-likelihoods as sums, grid search, underflow, and the negative log-likelihood loss.45 min
101Linear regression: the best lineFreeResiduals, least squares by formula, three roads to one line, the matrix form, and train versus test error.45 min
109One neuron, start to finishFreeWeighted sum, activation, loss, slopes and update, by hand and in code, and what one straight line can and can't split.45 min
111Training a small networkFreeThe backward pass in matrix form, gradient checks, a full training loop, and reading loss curves.45 min
112The same network in PyTorchFreenn.Sequential with the NumPy weights loaded, the same loss and slopes, then SGD and mini-batches.45 min
113Attention as matrix mathFreeScaled dot-product scores, the causal mask, a row-by-row softmax, and blending value vectors.45 min
114Residual connections and layer normFreeAdding the input back and why its slope keeps learning alive, layer norm by hand, and add and norm.40 min
115A transformer block, start to finishFreeA sentence through embeddings, attention, add and norm, a small network and next-word scores, every shape named.45 min
117Final test: transpose any formulaFreeSlips to fix from every topic, mixed practice of every kind of rearranging, the strategy chart and the full sheet.45 min
Learn the math behind AI from arithmetic up: numbers, algebra, vectors and matrices, calculus, probability and statistics, each shown as numbers moving and checked in Python, ending with a network you build and train from the formulas.
See the course →
Never trained an AI? Teach a small one to write, right in your browser. It takes about 20 seconds and needs no account. Teach it to write