Ranking from head-to-head results
Elo updates, Bradley-Terry strengths, and how model leaderboards turn votes into ratings.
Create a free account Try part of it here
Why it matters
Chess ratings, sports leagues and the leaderboards that rank AI models all face the same problem: you only see head-to-head results, who beat whom, and you want one number per player. A raw win rate misleads, because it ignores who each player faced. The Elo system and the Bradley-Terry model turn wins and losses into ratings that account for the opposition. The same Bradley-Terry formula sits inside every reward model you'll meet in Phase 6, so this lesson builds both from scratch in Python.
What you'll be able to do
- Turn a rating gap into a win probability with the Elo formula
- Update two ratings after a game, and explain why upsets move them more
- Fit Bradley-Terry strengths to a whole set of results at once
- Read an AI leaderboard built from pairwise votes, including its uncertainty
What it covers
- Why win rates mislead
- The Elo expected score
- Updating after a game
- Update two ratings after an upset
- Bradley-Terry: fitting all the results at once
- Fit Bradley-Terry strengths
- Leaderboards for AI models
- A leaderboard you can defend: rating chat models from votes
Show the answer
About 76%
A 200-point gap gives the stronger player about a 76% expected score. Ratings don't promise wins; they turn a gap into a probability, which this lesson builds step by step.
Why win rates mislead
Suppose model A wins 80% of its comparisons and model B wins 60%. Is A better? Not necessarily: A might have been matched mostly against weak opponents and B against strong ones. A raw win rate mixes a player's strength with the strength of whoever it happened to face, called the strength of schedule. If A went 8 for 10 against beginners and B went 6 for 10 against champions, B's record is the more impressive one, and a win rate can't show it.
Rating systems fix this by modeling each game. They give every player a number, and a formula turns the gap between two numbers into the probability that one beats the other. A win against a strong opponent then counts for more than a win against a weak one, which is exactly how you'd judge it yourself. The two most common versions, Elo and Bradley-Terry, share the same curve; they differ in how they fit it.
Show the answer
Credits B's wins more, since they came against stronger opponents
Rating systems model each game, so beating a strong opponent is a bigger surprise, and earns more, than beating a weak one. B may well end up rated above A.
The Elo expected score
In Elo, the probability that A beats B, called A's expected score, depends only on the rating gap:
E_A = 1 / (1 + 10^((R_B − R_A) / 400))
Equal ratings give 0.5. Each 400 points of gap multiplies the odds by 10: a 400-point favorite wins about 10 times as often as it loses, about 91% of the time. The expected scores of the two players always add up to 1.
The 400 and the base 10 are a convention from chess, chosen so that a 200-point gap means roughly 3-to-1 odds. Any base and scale would work, as long as everyone compares ratings on the same one; what matters is that equal gaps mean equal odds, wherever on the scale they sit.
def expected(r_a, r_b): return 1 / (1 + 10 ** ((r_b - r_a) / 400)) print(round(expected(1600, 1400), 3)) print(round(expected(1400, 1600), 3)) print(round(expected(1800, 1400), 3))
The curve is an S shape: small gaps matter a lot near even, and huge gaps approach certainty but never reach it.
print(round(1 / (1 + 10 ** ((1500 - 1500) / 400)), 2))
Show the answer
0.5
Equal ratings make the exponent 0, and 10⁰ = 1, so the expected score is 1 / (1 + 1) = 0.5: an even game.
Updating after a game
After each game, Elo nudges both ratings by the surprise: what happened minus what was expected. A win scores 1 and a loss 0:
R_A(new) = R_A + K × (S_A − E_A)
K sets how big the nudges are; 32 is a common choice. A favorite who wins as expected gains a little. An underdog who wins gains a lot, and the favorite loses the same amount, so the total of all ratings stays the same.
def expected(r_a, r_b): return 1 / (1 + 10 ** ((r_b - r_a) / 400)) a, b, k = 1600, 1400, 32 e_a = expected(a, b) print("If A wins:", round(a + k * (1 - e_a), 1)) # a small gain print("If A loses:", round(a + k * (0 - e_a), 1)) # a big drop
Choosing K is a trade-off. A big K reacts quickly to real changes in strength but jitters after every lucky result; a small K is steady but slow to notice improvement. Many systems give new players a bigger K, so their ratings find the right level fast, then shrink it once they've played enough games.
Elo is quick and needs no history: each game updates two numbers. Its weakness is that results depend on the order games were played in, and early ratings are shaky until enough games have come in. For a fixed set of results, like a season's games or a batch of votes, fitting everything at once does better.
Update two ratings after an upset
Player A (1600) loses to player B (1400), with K = 32. Set e_a to A's expected score before the game, then new_a and new_b to the updated ratings, rounded to 1 decimal place. (B's expected score is 1 − e_a.)
r_a, r_b, k = 1600, 1400, 32 e_a = 0 new_a = 0 new_b = 0 print(round(e_a, 3), new_a, new_b)
Show the answer
Each update is proportional to the surprise
The surprise is the result minus the expected score. An expected win has a small one (1 − 0.76 = 0.24), so ratings barely move. An upset has a big one (1 − 0.24 = 0.76), so they move about three times as far.
Bradley-Terry: fitting all the results at once
Bradley-Terry gives each player a strength s, and says player i beats player j with probability s_i / (s_i + s_j). Taking logs of the strengths gives ratings on the same S-shaped curve as Elo, just with a different base. The difference is in fitting: Bradley-Terry looks at every result together and finds the strengths that make all of them most probable. That's maximum likelihood, from earlier in this phase, and it doesn't depend on the order the games were played in.
A simple way to fit it repeats one update until the strengths settle: each player's new strength is its number of wins divided by Σ over its opponents of (games against j) / (s_i + s_j).
wins = [[0, 7, 9], [3, 0, 6], [1, 4, 0]] # wins[i][j]: times i beat j s = [1.0, 1.0, 1.0] for _ in range(100): new = [] for i in range(3): games = sum((wins[i][j] + wins[j][i]) / (s[i] + s[j]) for j in range(3) if j != i) new.append(sum(wins[i]) / games) total = sum(new) s = [x / total for x in new] print([round(x, 3) for x in s]) print(round(s[0] / (s[0] + s[1]), 2)) # P(player 0 beats player 1)
Each round of this update is guaranteed to make the observed results at least as probable as before, so it climbs steadily to the best fit. Ties are usually counted as half a win for each side, and dividing by the total each round just fixes the scale, since only ratios of strengths matter.
s_a, s_b = 2.0, 1.0 print(round(s_a / (s_a + s_b), 3))
Show the answer
0.667
P(A beats B) = s_A / (s_A + s_B) = 2 / 3 ≈ 0.667. Twice the strength means winning two games in three, not every game.
Fit Bradley-Terry strengths
Three models were compared in pairs. Fit Bradley-Terry strengths with the update from the lesson: 200 rounds, each one computing every model's new strength from the old ones, then dividing by their total. Set s to the final list and p_01 to the probability that model 0 beats model 1, rounded to 2 places.
wins = [[0, 12, 15], [8, 0, 11], [5, 9, 0]] # wins[i][j]: times i beat j s = [1.0, 1.0, 1.0] # 200 rounds of the update p_01 = 0 print([round(x, 3) for x in s], p_01)
Leaderboards for AI models
Public leaderboards for chat models collect pairwise votes: a person sees two anonymous answers to the same prompt and picks the better one, or calls a tie. Fitting Bradley-Terry to many thousands of votes gives every model a rating, usually reported on an Elo-like scale.
Read those ratings the way you'd read any estimate. Each comes with an interval, often from a bootstrap over the votes, and models whose intervals overlap heavily can't be told apart yet. The ratings also depend on who voted and which prompts they chose, so a model can rank differently for coding, writing or another language. Votes also carry biases: people tend to prefer longer, more confident answers, and whichever answer is shown first. Careful leaderboards shuffle the order, and some adjust for answer length, so the ratings measure quality more than style. And in Phase 6, the same formula trains reward models: scores whose differences predict which of two answers people preferred.
Show the answer
They can't be told apart yet: the intervals overlap heavily
Most of each interval lies inside the other, so the 5-point gap is well within the uncertainty. More votes would narrow the intervals.
Show the answer
Maximum likelihood estimation
Choosing parameters that make the observed data most probable is maximum likelihood, the same idea used to fit models across machine learning.
A leaderboard you can defend: rating chat models from votes
Your team is choosing a chat model, and for a month colleagues have voted on pairs of anonymous answers to the same prompt: which is better, or a tie. Seven models are in the running. Your job is to turn the vote log into ratings that are honest about their uncertainty, and a short README that tells a busy reader what the ratings do and don't show.
The starter builds votes: one dictionary per vote with the model shown on the left, the one on the right, and the winner ("left", "right" or "tie"). Treat _simulate as a black box and find any quirks from the data itself, as you would with a real log; reading the generator first defeats the point.
The leaderboard is the deliverable. Someone picking a model will give it thirty seconds, so it should say exactly what the votes support and no more.
It must
- Count each model's games and its win rate, with ties as half a win, and say why raw win rates alone can mislead
- Check whether the side an answer is shown on affects the vote: the left side's share of non-tie votes, with a 95% interval
- Run Elo (K = 32, everyone starting at 1000) in vote order, then on at least 20 shuffled orders, and report how far each model's rank moves
- Fit Bradley-Terry strengths with ties as half wins, and put them on the Elo scale: 400 × log₁₀(strength), shifted so the average is 1000
- Bootstrap the votes (at least 200 resamples) to get a 95% interval for every model's Bradley-Terry rating
- Publish a leaderboard table with rating, interval and games, grouping models whose intervals overlap heavily instead of forcing a strict order
- Write a README of at most 200 words: the method, what the intervals mean, which models are clearly separated, and one weakness of this vote data
Go further
- Add a left-side advantage to the Bradley-Terry model as one extra strength, refit, and say whether any ranking changes
- Fit on the first 80% of votes and compare Elo and Bradley-Terry on the last 20%, by the average log loss of their predicted win probabilities
- Work out roughly how many more votes the newest model needs before its interval is as narrow as the others'
import math import random def _simulate(seed=11): """Build the vote log. Treat this as a black box: find any quirks from the data.""" rng = random.Random(seed) strength = {"atlas": 1.2, "birch": 0.9, "cobalt": 0.85, "delta": 0.3, "ember": 0.0, "fjord": -0.4, "glint": 1.0} votes = [] for i in range(3000): pool = [m for m in strength if m != "glint" or i >= 2700] left, right = rng.sample(pool, 2) p_left = 1 / (1 + math.exp(-(strength[left] - strength[right] + 0.25))) if rng.random() < 0.08: winner = "tie" else: winner = "left" if rng.random() < p_left else "right" votes.append({"vote": i, "left": left, "right": right, "winner": winner}) return votes # Each vote: two anonymous answers to the same prompt, shown side by side; the voter picked # the better one or called a tie. Models: atlas, birch, cobalt, delta, ember, fjord, glint. votes = _simulate() print(votes[:3]) print(len(votes), "votes:", {w: sum(v["winner"] == w for v in votes) for w in ["left", "right", "tie"]}) # Your analysis starts here: games and win rates, then the checks, then the ratings.
Practice what you just read
- 3 code examples you can edit and run
- a practice exercise that checks your code
- a 6-question quiz
- 5 review cards for spaced repetition
Try a quiz question
Why can a raw win rate mislead when ranking models?