Glossary · Phase 5 · Transformers
Power law
A curve where each doubling of the input cuts the output by the same fraction. It shows up as a straight line on a log-log plot.
Try it now
The Chinchilla advice is about 20 tokens per parameter. Slide the ratio: the same budget buys a bigger model trained on less, or a smaller one trained on more.
Learn it properly
Scaling laws · How loss falls predictably with size, data and compute. About 35 minutes.
Other terms from the same lesson
- compute-optimal: The split of a fixed compute budget between model size and training tokens that gives the lowest loss: about 20 tokens per parameter.
- irreducible loss: The floor the loss approaches however big the model and data get, because some text is simply unpredictable.
- log-log plot: A chart where both axes use a log scale, so each step along an axis multiplies by 10. Power laws turn into straight lines.
- overtraining: Training a model on far more tokens than compute-optimal. Training costs more, but the smaller model is cheaper to run for every user afterwards.
- parameter count: How many adjustable numbers (weights) a model has. Large language models have billions.
- scaling laws: The finding that models improve predictably as you add data, size and compute.
Never trained an AI?
Teach a small one to write, right in your browser. It takes about 20 seconds and needs no account.