Is the difference real? Comparing two models
Why this matters
"Model B scored 2 points higher than model A" is the most common claim in AI work, and often it's noise: on another set of test items the order could flip. Lesson 98 put an error bar on one score. This lesson puts one on the difference between two. The trick is to compare the models item by item, on the same test: most items are ones both models get right, or both get wrong, and those say nothing about which is better. You'll build the range for a difference two ways, by formula and by resampling, learn what a p-value says and what it doesn't, and see why trying many variants almost guarantees a lucky win.
- Compare two models on the same test items by their paired differences: the mean, its standard error and a 95% range
- Build the same range by resampling the items 2,000 times, and read it off a histogram with 0 marked
- Say what a p-value is in one plain sentence, and what it isn't
- Explain why trying many variants makes a lucky win likely, with 1 − 0.95ᵏ, and check a new model's lead in NumPy and PyTorch
The math: differences, item by item
Purpose. Measure the gap between two models on the same test, and say how sure you can be of it.
Mark each model on each test item: 1 if it's right, 0 if it's wrong. Two models scored on the same items give paired results: one mark from each model for every item. Subtract item by item, B's mark minus A's, and you get each item's paired difference:
- +1 where only B is right,
- −1 where only A is right,
- 0 where they agree, both right or both wrong.
The mean of the differences is exactly B's accuracy minus A's. But now each item speaks for itself, and the items the models agree on add nothing.