A layer and a small network as matrices
Why this matters
Lesson 109's single neuron could only draw one straight line, so it failed at XOR. The fix is to put several neurons side by side, a layer, and feed their outputs into another layer. Written one neuron at a time, that's a tangle of sums; written as matrices, it's two matrix products (lesson 56) with a relu and a softmax between them, and a whole batch of examples goes through at once. That is exactly how every network library computes, from this lesson's toy to a language model: a stack of "multiply by a matrix, add a bias, apply an activation", which PyTorch builds with nn.Sequential. The skill that makes it manageable is keeping track of shapes. Every bug in this lesson's code is a shape that doesn't line up, and every shape follows from two rules you already know.
- Push a whole batch through two layers: Z₁ = XW₁ + b₁, A₁ = relu(Z₁), Z₂ = A₁W₂ + b₂, P = softmax(Z₂)
- Name the shape of every matrix along the way, and compute single entries by hand
- Write the forward pass in six lines of NumPy, and add a hidden unit by fixing every shape
- See hidden units fit a problem one neuron can't, score a small network on a batch of signs, and build the same network from PyTorch's nn.Sequential
The math: two layers, one batch
Purpose. Compute a small network's predictions for a whole batch, and know every shape on the way.
Stack the batch's examples as the rows of a matrix X, n rows of d features. A layer of h neurons is a weight matrix W₁ of shape d × h, one column per neuron, plus a bias b₁ with h entries. The whole forward pass:
| step | formula | shape |
|---|---|---|
| hidden scores | Z₁ = X W₁ + b₁ | (n × d)(d × h) → n × h |
| hidden outputs | A₁ = relu(Z₁) | n × h |
| class scores | Z₂ = A₁ W₂ + b₂ | (n × h)(h × k) → n × k |
| chances | P = softmax(Z₂), row by row | n × k |
| loss | L = mean of −log P[right class] | one number |
Each row of every matrix belongs to one example, all the way through. The bias b₁ is added to every row: lesson 64's broadcasting.