Glossary · Phase 6 · Modern LLM engineering
Training cutoff
The date a model's training data stops. It knows nothing about events after it unless you give it the information.
Newer benchmarks fight contamination in three ways. They keep test sets private. They write new questions all the time. Or they use problems created after the model's training cutoff, the date its training data ends.
Try it now
Two models a couple of points apart on 200 questions aren't really different. Quadrupling the questions only halves the margin.
Learn it properly
Benchmarks and their limits · What leaderboards measure, and what they miss. About 30 minutes.
Other terms from the same lesson
- benchmark: A standard test set used to compare models. Useful, but models can be tuned to the test.
- contamination: When a benchmark's test questions (or their answers) leaked into a model's training data, so its score overstates what it can really do.
- Goodhart's law: "When a measure becomes a target, it ceases to be a good measure": optimize for a score and the score stops meaning what it did.
Never trained an AI?
Teach a small one to write, right in your browser. It takes about 20 seconds and needs no account.