Index Ai Academy
Foundation · Foundation mathPutting it together

A layer and a small network as matrices

About 45 minLooking ahead: some prerequisites aren't done yet

Why this matters

Lesson 109's single neuron could only draw one straight line, so it failed at XOR. The fix is to put several neurons side by side, a layer, and feed their outputs into another layer. Written one neuron at a time, that's a tangle of sums; written as matrices, it's two matrix products (lesson 56) with a relu and a softmax between them, and a whole batch of examples goes through at once. That is exactly how every network library computes, from this lesson's toy to a language model: a stack of "multiply by a matrix, add a bias, apply an activation", which PyTorch builds with nn.Sequential. The skill that makes it manageable is keeping track of shapes. Every bug in this lesson's code is a shape that doesn't line up, and every shape follows from two rules you already know.

  • Push a whole batch through two layers: Z₁ = XW₁ + b₁, A₁ = relu(Z₁), Z₂ = A₁W₂ + b₂, P = softmax(Z₂)
  • Name the shape of every matrix along the way, and compute single entries by hand
  • Write the forward pass in six lines of NumPy, and add a hidden unit by fixing every shape
  • See hidden units fit a problem one neuron can't, score a small network on a batch of signs, and build the same network from PyTorch's nn.Sequential
Quick check
Have a guess before you start. A batch of 8 signs with 3 features each goes through a layer with 4 hidden units. What shape is X @ W₁?

The math: two layers, one batch

Purpose. Compute a small network's predictions for a whole batch, and know every shape on the way.

Stack the batch's examples as the rows of a matrix X, n rows of d features. A layer of h neurons is a weight matrix W₁ of shape d × h, one column per neuron, plus a bias b₁ with h entries. The whole forward pass:

stepformulashape
hidden scoresZ₁ = X W₁ + b₁(n × d)(d × h) → n × h
hidden outputsA₁ = relu(Z₁)n × h
class scoresZ₂ = A₁ W₂ + b₂(n × h)(h × k) → n × k
chancesP = softmax(Z₂), row by rown × k
lossL = mean of −log P[right class]one number

Each row of every matrix belongs to one example, all the way through. The bias b₁ is added to every row: lesson 64's broadcasting.