Glossary · Phase 6 · Modern LLM engineering
Goodhart's law
"When a measure becomes a target, it ceases to be a good measure": optimize for a score and the score stops meaning what it did.
Try it now
Two models a couple of points apart on 200 questions aren't really different. Quadrupling the questions only halves the margin.
Learn it properly
Benchmarks and their limits · What leaderboards measure, and what they miss. About 30 minutes.
Other terms from the same lesson
- benchmark: A standard test set used to compare models. Useful, but models can be tuned to the test.
- contamination: When a benchmark's test questions (or their answers) leaked into a model's training data, so its score overstates what it can really do.
- training cutoff: The date a model's training data stops. It knows nothing about events after it unless you give it the information.
Never trained an AI?
Teach a small one to write, right in your browser. It takes about 20 seconds and needs no account.