Glossary · Phase 5 · Transformers
Irreducible loss
The floor the loss approaches however big the model and data get, because some text is simply unpredictable.
Try it now
The Chinchilla advice is about 20 tokens per parameter. Slide the ratio: the same budget buys a bigger model trained on less, or a smaller one trained on more.
Learn it properly
Scaling laws · How loss falls predictably with size, data and compute. About 35 minutes.
Other terms from the same lesson
- compute-optimal: The split of a fixed compute budget between model size and training tokens that gives the lowest loss: about 20 tokens per parameter.
- log-log plot: A chart where both axes use a log scale, so each step along an axis multiplies by 10. Power laws turn into straight lines.
- overtraining: Training a model on far more tokens than compute-optimal. Training costs more, but the smaller model is cheaper to run for every user afterwards.
- parameter count: How many adjustable numbers (weights) a model has. Large language models have billions.
- power law: A curve where each doubling of the input cuts the output by the same fraction. It shows up as a straight line on a log-log plot.
- scaling laws: The finding that models improve predictably as you add data, size and compute.
Never trained an AI?
Teach a small one to write, right in your browser. It takes about 20 seconds and needs no account.