Benchmarks and their limits

import math

p, n = 0.80, 100                    # 80 right out of 100 questions
se = math.sqrt(p * (1 - p) / n)     # standard error: the typical wobble
margin = 1.96 * se                  # 95% margin of error
print(round(se, 3), round(margin, 3))
print(f"{p - margin:.1%} to {p + margin:.1%}")

Simulating two close models

The code below makes the wobble visible. Two imaginary models have true accuracies of 82% and 80%. Each one sits a benchmark 10,000 times, with fresh luck every time. The code tries four benchmark sizes. For each size, it prints the 95% margin on one score. It also prints how often the worse model came out ahead.

• rng.binomial(n, better, 10_000) counts how many of n questions the better model gets right, in each of 10,000 runs.
• Dividing by n turns those counts into accuracies.

import numpy as np

rng = np.random.default_rng(0)
better, worse = 0.82, 0.80                     # true accuracies of two models
print(f"{'questions':>9}  {'± (95%) on one score':>21}  {'worse model scores higher':>26}")
for n in [100, 500, 2_000, 10_000]:
    a = rng.binomial(n, better, 10_000) / n      # 10,000 imaginary benchmark runs
    b = rng.binomial(n, worse, 10_000) / n
    margin = 1.96 * np.sqrt(better * (1 - better) / n)
    print(f"{n:>9,}  {margin:>20.1%}  {(b > a).mean():>25.0%}")

Read the table row by row. On a 100-question test, each score is only good to about ±7.5 points. The worse model comes out ahead about a third of the time (32%). At 2,000 questions, the margin shrinks to ±1.7 points, and the worse model wins only 5% of the time. At 10,000 questions, that share rounds to 0%.

Many headline benchmarks have a few hundred questions. So do the slices of them that make the news. On tests that size, a gap of one or two points is often noise. When you see a score table, ask three things. How many questions were there? Are there confidence intervals? Were the results averaged over several runs? Your own evaluation sets from Phase 1 follow exactly the same arithmetic.

When scores stop meaning what they say

A score can be precise and still misleading. Here are four common reasons.

1. Contamination

Imagine a student who found last year's exam, with the answers, a week before the test. A top grade then says little about what they know. Models face the same problem. Benchmarks are published on the web, and the web is training data. A model that saw the test questions, or their answers, during training can score well without the skill the test is meant to measure. This is called contamination.

One way researchers check: compare scores on the original questions with scores on reworded or freshly written versions of the same problems. Real skill carries over to new wording. Memory doesn't.

The code below fakes this. The model's true skill is 60%. But 30% of the questions leaked into its training, and it gets every leaked question right.

import numpy as np

rng = np.random.default_rng(1)
n, skill, leaked = 1_000, 0.60, 0.30              # true skill 60%; 30% of questions leaked into training
is_leaked = rng.random(n) < leaked
original = np.where(is_leaked, True, rng.random(n) < skill).mean()      # memorized questions: always right
reworded = (rng.random(n) < skill).mean()                               # same problems, new wording
print(f"original questions: {original:.0%}   reworded versions: {reworded:.0%}")

The original questions score 73% and the reworded ones 63%. The reworded score sits near the true 60%; the rest is ordinary sampling noise. The gap of about ten points comes from memory. A gap like that is a red flag.

Newer benchmarks fight contamination in three ways. They keep test sets private. They write new questions all the time. Or they use problems created after the model's training cutoff, the date its training data ends.

2. Saturation

When nearly every student scores 98% on a test, the test can't pick out the best student. The same happens to benchmarks. Once top models score near the ceiling, the benchmark is saturated. It no longer separates them. Many of the remaining errors are mistakes in the answer key, not in the models. Benchmarks used to stay useful for years. Now it's often months.

3. Optimizing for the leaderboard

Goodhart's law says that when a number becomes a target, it stops being a good measure. Labs compete on leaderboard numbers, so they tune their models for them. Here are some ways to lift a score without making the model more useful:

• answering in the exact format a grader rewards;
• tuning the prompt separately for each benchmark;
• running the model many times and reporting the best run.

Preference leaderboards have their own version. Longer answers with more headings and bullet points win more votes. That's why LMArena added a "style control" view. It adjusts ratings for length and formatting.

4. Distance from your task

A model that's best at graduate physics may not be best at your work. Your work might be writing product descriptions in your brand's voice. It might be following your support policy, or handling your data formats. A public benchmark rarely tests things like these.

Reading a benchmark table

Ask these five questions every time.

1. What exactly was tested? How many items were there, and is the test set public?
2. Is the gap larger than the noise? Are there confidence intervals, or several runs?
3. Were all models run the same way: same prompt, same number of attempts, same tools?
4. Who ran it: the model's maker, or an independent group?
5. How close is the task to yours, and what does your own evaluation say?

For marketers: use leaderboards to build a shortlist. Then decide with a test set of 100 to 300 examples from your own work, scored the same way for every model. On a set that size, trust only clear gaps. Expect the ranking to change.