Residual connections and layer norm
Why this matters
Attention (lesson 113) and a small network (lesson 110) are two pieces of a transformer block. Two more pieces glue them together, and both are short. The first adds each piece's input back onto its output: x + f(x). It looks like nothing, but its slope has a 1 in it, and that 1 is why networks dozens of layers deep can learn at all. The second rescales each word's vector to mean 0 and deviation 1, the standardizing of lesson 97 done one vector at a time, so the numbers stay a steady size from layer to layer. Together they're called "add and norm".
- Compute x + f(x) and its slope, 1 + f′(x), and say why the added 1 keeps slopes alive through many layers
- Standardize one vector to mean 0 and deviation 1, then apply a learned scale γ and shift β
- Tell layer norm from batch norm by which direction is averaged, and catch the slip of averaging over the batch
- Follow the shapes through add and norm, and get the same numbers from PyTorch's nn.LayerNorm
The math: add the input back, then rescale
Purpose. Two short steps that let a network stack many layers: add each layer's input back, and keep every vector the same size.
Adding the input back. Any layer, a small network or attention, takes a vector x and gives back f(x). A residual connection sends x around the layer and adds it back on:
In plain words: keep what came in, and add the layer's change to it. The slope is the slope of x, which is 1, plus the slope of f: the sum rule of lesson 73.
The layer now only has to learn a change, and the 1 in the slope turns out to matter most.