Comparing two models on the same test
Paired results, disagreements, a quick z and a bootstrap for the gap between two models.
Create a free account Try part of it here
Why it matters
Two models answer the same 200 test questions, and model B gets 4 more right. Is B better, or was that luck? Because both models faced exactly the same questions, you can do much better than comparing two overall scores: look at the questions where they disagree. This lesson builds the paired comparison that careful eval reports use, including a bootstrap you write yourself, so a 2-point win on a leaderboard stops being a mystery and becomes something you can check.
What you'll be able to do
- Split two models' results on the same questions into agreements and disagreements
- Explain why only the disagreements say which model is better
- Judge a difference with a quick z-score and with a bootstrap interval for the gap
- Read an eval report's claim that one model beats another, and say what's missing
What it covers
- Same questions, paired results
- Sort the questions into four groups
- Why pairing beats two separate scores
- Where the z comes from
- A bootstrap for the gap
- Bootstrap the gap
- Reading an eval report
- Paired comparisons in AI
Show the answer
7
They disagreed on 10 questions. On each one, exactly one model was right, and B's 4-point lead means B won 4 more of them than A: 7 against 3. Only those 10 questions say anything about which model is better.
Same questions, paired results
When two models answer the same questions, each question gives a pair of results. With 1 for right and 0 for wrong, every question lands in one of four groups:
| blank | B right | B wrong |
|---|---|---|
| A right | both right | only A |
| A wrong | only B | both wrong |
Questions both models got right, or both got wrong, say nothing about which is better: they were easy, or hard, for both. The comparison lives entirely in the two disagreement cells.
For this to work, the two lists must line up: position i has to be the same question for both models. In practice, keep each result next to its question's id, and match them up by id before comparing, rather than trusting that two files happen to be in the same order. A shifted list quietly compares each answer with the wrong question.
a = [1, 1, 0, 1, 0, 1, 1, 0] # model A, question by question b = [1, 0, 1, 1, 1, 1, 1, 0] # model B, the same questions only_a = sum(1 for i in range(len(a)) if a[i] == 1 and b[i] == 0) only_b = sum(1 for i in range(len(a)) if a[i] == 0 and b[i] == 1) print("only A:", only_a, "| only B:", only_b)
a = [1, 0, 0, 1, 1, 0] b = [1, 1, 1, 0, 1, 0] print(sum(1 for i in range(6) if a[i] == 0 and b[i] == 1))
Show the answer
2
B is right and A wrong on questions 2 and 3 (positions 1 and 2): 2 questions. On position 3 it's the other way around, and the rest agree.
Sort the questions into four groups
Two models answered the same 10 questions. Count both_right, only_a, only_b and both_wrong.
a = [1, 1, 1, 0, 1, 0, 1, 1, 0, 1] b = [1, 0, 1, 1, 1, 0, 1, 1, 1, 1] both_right = 0 only_a = 0 only_b = 0 both_wrong = 0 print(both_right, only_a, only_b, both_wrong)
Why pairing beats two separate scores
Test questions vary hugely in difficulty, and that variation makes overall scores noisy. If you compared A on one set of questions with B on a different set, B might win just because it drew easier ones. Giving both models the same questions removes that noise: a question's difficulty affects both models at once, so it cancels out when you look at disagreements.
That's why a paired comparison can spot a real difference that two overall scores, each with its own margin of error, would leave buried. Some rough numbers show the size of the effect. On 200 questions, an accuracy around 70% wobbles by about ±6 points from one random test set to another. Two separate tests, each wobbling that much, can hide a 4-point difference completely. On the same questions, most of that wobble is shared, and the disagreements carry what's left.
Pairing also tells you how much evidence you really have. A 4-point lead built on 10 disagreements is far weaker than a 4-point lead built on 4 disagreements all going B's way, even though both reports would say "70% against 74%".
If the two models were equally good, each disagreement would be a coin flip. z measures how lopsided the flips came out, in units of the usual wobble.
Show the answer
Eval 2: every disagreement went B's way
In eval 2, z = 4 / √4 = 2.0; in eval 1, z = 4 / √10 ≈ 1.3. A clean sweep of the disagreements is harder to explain by luck than a 7-to-3 split.
Where the z comes from
If the two models were equally good, each disagreement would be a fair coin flip: on any one of them, A and B would be equally likely to be the one that's right. With d disagreements you'd expect about d / 2 each, and B's lead, b − a, would wobble around 0 by about √d. Dividing the lead by √d measures it in units of that wobble. That's McNemar's test in its quickest form. A z beyond about 2, either way, would happen less than one time in twenty by luck.
A bootstrap for the gap
The z-score is a quick rule of thumb. A bootstrap, which you met for medians in the samples lesson, gives an interval for the gap itself. The trick for paired data: resample whole questions, keeping each question's pair of results together.
- Pick n questions at random, with replacement, from the n you have.
- Work out B's accuracy minus A's on that resample.
- Repeat a thousand times, sort the gaps, and keep the middle 95%.
import random random.seed(1) # 100 questions: 63 both right, 3 only A, 7 only B, 27 both wrong a = [1] * 63 + [1] * 3 + [0] * 7 + [0] * 27 b = [1] * 63 + [0] * 3 + [1] * 7 + [0] * 27 n = len(a) gaps = [] for _ in range(1000): picks = random.choices(range(n), k=n) gaps.append(sum(b[i] - a[i] for i in picks) / n) gaps.sort() print(round(gaps[25], 3), round(gaps[974], 3))
If the interval includes 0, the data can't yet tell the two models apart. Its width matters too: a wide interval around 0 says "test on more questions", while a narrow one around 0 says the models really are about equally good. Resampling whole questions, rather than each model's results separately, is what keeps the pairing, and with it the canceling of question difficulty.
Show the answer
Not clear yet: a gap of 0 is inside the interval
The interval includes 0, so "no real difference" is consistent with the data. B may well be better, but this test can't show it; more questions would narrow the interval.
Bootstrap the gap
Write the paired bootstrap. Repeat 1,000 times: pick n positions with random.choices(range(n), k=n), and append B's accuracy minus A's on those positions to gaps. Then sort gaps and set low and high to the values at positions 25 and 974.
import random random.seed(7) a = [1, 1, 0, 1, 1, 0, 1, 0, 1, 1, 1, 0, 1, 1, 0, 1, 1, 1, 0, 1] b = [1, 1, 1, 1, 1, 0, 1, 1, 1, 1, 1, 1, 1, 1, 0, 1, 1, 1, 1, 1] n = len(a) gaps = [] # repeat 1,000 times: resample positions, append B's accuracy minus A's low = 0 high = 0 print(len(gaps), low, high)
Reading an eval report
When a report says one model beats another, look for four things. Did both models answer the same questions? How many questions, and how many disagreements? Is there a paired interval, or at least the disagreement counts, rather than two bare percentages? And how many models or prompts were tried before this winner was picked? Trying twenty variants and reporting the best one finds a "winner" by luck alone, as the hypothesis testing lesson warned.
A report that answers all four is worth trusting, whichever model it says has won. One that gives only two percentages isn't wrong, but it hasn't yet shown that the difference is real.
Show the answer
Each resample is a new, slightly different test set of the same size
Drawing with replacement means some questions appear twice and others not at all, which mimics drawing a fresh test set of the same size. The spread of the gaps across resamples shows how much the result could wobble.
Paired comparisons in AI
You'll use this whenever you compare two prompts, two models or two settings on the same eval set. The same idea powers side-by-side evaluation, where people (or a judge model) see two answers to one prompt and pick the better one: each prompt is a pair, and ties say nothing. Public leaderboards built from side-by-side votes use a version of it too. Whenever you can test two options on the same items, do it, and report the disagreements.
One wrinkle is particular to language models: the same model can answer the same question differently on two runs. For generation tasks, run each question a few times and use each model's average score per question, or at least fix the sampling settings for both, so the comparison measures the models rather than the dice.
a_wins, b_wins = 100, 120 print(round((b_wins - a_wins) / (a_wins + b_wins) ** 0.5, 2))
Show the answer
1.35
Ties are left out, like questions both models got right: (120 − 100) / √220 ≈ 1.35. That's below 2, so this isn't yet clear evidence that B is better.
Practice what you just read
- 2 code examples you can edit and run
- a practice exercise that checks your code
- a 6-question quiz
- 5 review cards for spaced repetition
Try a quiz question
Two models answer the same 100 questions. Which questions tell you which model is better?