Overtraining
Training a model on far more tokens than compute-optimal. Training costs more, but the smaller model is cheaper to run for every user afterwards.
"Compute-optimal" gives the lowest loss for a given training budget. But a model is trained once and then used billions of times. A smaller model is cheaper for every one of those requests. So labs now overtrain small models, far past 20 tokens per parameter. Meta trained Llama 3's 8-billion-parameter model on about 15 trillion tokens. That's nearly 2,000 tokens per parameter. Its loss kept improving long past the Chinchilla point, only more slowly.
Try it now
The Chinchilla advice is about 20 tokens per parameter. Slide the ratio: the same budget buys a bigger model trained on less, or a smaller one trained on more.
Learn it properly
Scaling laws · How loss falls predictably with size, data and compute. About 35 minutes.
Other terms from the same lesson
- compute-optimal: The split of a fixed compute budget between model size and training tokens that gives the lowest loss: about 20 tokens per parameter.
- irreducible loss: The floor the loss approaches however big the model and data get, because some text is simply unpredictable.
- log-log plot: A chart where both axes use a log scale, so each step along an axis multiplies by 10. Power laws turn into straight lines.
- parameter count: How many adjustable numbers (weights) a model has. Large language models have billions.
- power law: A curve where each doubling of the input cuts the output by the same fraction. It shows up as a straight line on a log-log plot.
- scaling laws: The finding that models improve predictably as you add data, size and compute.
Never trained an AI?
Teach a small one to write, right in your browser. It takes about 20 seconds and needs no account.