Index Ai Academy
Foundation · Foundation mathPutting it together

Residual connections and layer norm

About 40 minLooking ahead: some prerequisites aren't done yet

Why this matters

Attention (lesson 113) and a small network (lesson 110) are two pieces of a transformer block. Two more pieces glue them together, and both are short. The first adds each piece's input back onto its output: x + f(x). It looks like nothing, but its slope has a 1 in it, and that 1 is why networks dozens of layers deep can learn at all. The second rescales each word's vector to mean 0 and deviation 1, the standardizing of lesson 97 done one vector at a time, so the numbers stay a steady size from layer to layer. Together they're called "add and norm".

  • Compute x + f(x) and its slope, 1 + f′(x), and say why the added 1 keeps slopes alive through many layers
  • Standardize one vector to mean 0 and deviation 1, then apply a learned scale γ and shift β
  • Tell layer norm from batch norm by which direction is averaged, and catch the slip of averaging over the batch
  • Follow the shapes through add and norm, and get the same numbers from PyTorch's nn.LayerNorm
Quick check
Have a guess before you start. Start with 1 and halve it, twenty times in a row. About how much is left?

The math: add the input back, then rescale

Purpose. Two short steps that let a network stack many layers: add each layer's input back, and keep every vector the same size.

Adding the input back. Any layer, a small network or attention, takes a vector x and gives back f(x). A residual connection sends x around the layer and adds it back on:

out=x+f(x)ddx(x+f(x))=1+f′(x)\text{out} = x + f(x) \qquad \frac{d}{dx}\big(x + f(x)\big) = 1 + f'(x)

In plain words: keep what came in, and add the layer's change to it. The slope is the slope of x, which is 1, plus the slope of f: the sum rule of lesson 73.

The layer now only has to learn a change, and the 1 in the slope turns out to matter most.