Glossary · Phase 6 · Modern LLM engineering
Benchmark
A standard test set used to compare models. Useful, but models can be tuned to the test.
A benchmark is a standard exam for AI models. Every model gets the same questions, so the scores can be compared. Think of a school entrance exam: one test, many students, one number each. Here are five well-known benchmarks.
Try it now
Two models a couple of points apart on 200 questions aren't really different. Quadrupling the questions only halves the margin.
Learn it properly
Benchmarks and their limits · What leaderboards measure, and what they miss. About 30 minutes.
Other terms from the same lesson
- contamination: When a benchmark's test questions (or their answers) leaked into a model's training data, so its score overstates what it can really do.
- Goodhart's law: "When a measure becomes a target, it ceases to be a good measure": optimize for a score and the score stops meaning what it did.
- training cutoff: The date a model's training data stops. It knows nothing about events after it unless you give it the information.
Never trained an AI?
Teach a small one to write, right in your browser. It takes about 20 seconds and needs no account.