Glossary · Phase 4 · Deep learning
RMSNorm
A simpler LayerNorm that only rescales by the typical size of the numbers, without subtracting the average. Used in many recent LLMs.
Try it now
They all rescale numbers to a steady size. They differ only in which numbers go into the average.
Learn it properly
Normalization layers · Batch and layer norm keep training stable. About 30 minutes.
Other terms from the same lesson
- batch normalization: Rescaling each feature across a batch to an average of 0 and a spread of 1, then letting the network adjust it. Common in image models.
- normalization: Rescaling the numbers inside a network so they stay in a steady range, which makes training smoother.
- pre-norm: Putting the normalization before each sub-layer instead of after it. It makes deep transformers much easier to train.
Never trained an AI?
Teach a small one to write, right in your browser. It takes about 20 seconds and needs no account.