Instruction and preference data

Fitting strengths from votes

In real life you don't know the strengths. You only see who won. Fitting means finding the strengths that best explain the votes. The cell below runs a small contest. Four copywriters' taglines are compared 800 times by simulated raters. The code knows each writer's hidden true strength, so you can see how close the fit gets. It does three things:

1. It simulates 800 votes, using the formula above to pick each winner.
2. It prints how often each writer beat each of the others.
3. It starts every strength at 0 and nudges them 500 times, until they explain the votes well.

import numpy as np

rng = np.random.default_rng(0)
writers = ["Writer A", "Writer B", "Writer C", "Writer D"]
true_strength = np.array([1.0, 0.4, 0.0, -0.6])          # hidden quality on a logit scale

votes = []                                                 # (winner, loser) pairs
for _ in range(800):
    i, j = rng.choice(4, size=2, replace=False)
    p_i_wins = 1 / (1 + np.exp(-(true_strength[i] - true_strength[j])))
    votes.append((i, j) if rng.random() < p_i_wins else (j, i))

wins = np.zeros((4, 4))
for w, l in votes:
    wins[w, l] += 1
print("win rate of row against column:")
for i in range(4):
    cells = [f"{wins[i, j] / (wins[i, j] + wins[j, i]):.2f}" if i != j else "  - " for j in range(4)]
    print(f"  {writers[i]}  " + "  ".join(cells))

s = np.zeros(4)                                            # fit by gradient ascent on the log-likelihood
W, L = np.array(votes).T
for step in range(500):
    p = 1 / (1 + np.exp(-(s[W] - s[L])))                  # predicted chance each recorded winner wins
    grad = np.zeros(4)
    np.add.at(grad, W, 1 - p)
    np.add.at(grad, L, -(1 - p))
    s += 0.01 * grad
    s -= s.mean()                                          # only differences matter; center at 0

print("\nfitted strength:", [round(float(v), 2) for v in s])
print("true strength:  ", [round(float(v), 2) for v in true_strength - true_strength.mean()])
print("Elo-style rating:", [int(round(1000 + 400 / np.log(10) * v)) for v in s])

The fitted strengths, [0.87, 0.17, −0.16, −0.88], land close to the true ones, [0.8, 0.2, −0.2, −0.8]. The code shifts both lists so they average 0, because only the gaps matter. Writer A beat Writer D in 85% of their votes, close to the 83% you worked out above. And four numbers explain all six matchups at once. Nobody had to compare every pair many times.

The last line shows Elo-style ratings, the kind used in chess. They're the same strengths on a different scale: 1000 plus about 174 times the strength. On that scale, a gap of 400 points means 10-to-1 odds. The stronger side wins about 10 games for every 1 it loses.

Do the annotators agree?

Preference labels are noisy. People get tired, rush, or honestly disagree. So teams run two checks on their labels. The first is agreement. Give the same pairs to two annotators (the people who label the data) and count how often they pick the same answer.

Raw agreement makes the data look better than it is. Two people flipping coins would agree half the time. Cohen's kappa corrects for that luck. It's written $\kappa$, the Greek letter kappa. It uses two numbers:

• $p_{agree}$: how often the two annotators really agree.
• $p_{chance}$: how often they'd agree by luck, if each guessed at their usual rates.

In plain words: take how far the annotators beat luck. Divide it by the most they could have beaten luck. A kappa of 0 means no better than chance. A kappa of 1 means they always agree.

Worked example. Two annotators agree on 75% of pairs. Luck alone would give 50%. They beat luck by 0.75 − 0.50 = 0.25. The most they could have beaten it by is 1 − 0.50 = 0.50. So kappa is 0.25 / 0.50 = 0.5.

p_agree = 0.75     # how often the two annotators agree
p_chance = 0.50    # how often they'd agree by luck
kappa = (p_agree - p_chance) / (1 - p_chance)
print(kappa)

For comparison, InstructGPT's labelers agreed with each other about 73% of the time.

Does the order matter?

The second check is for position bias. Judges tend to favor whichever response is shown first (or, for some judges, second). People do it, and LLM judges do it even more. It's like a taste test where the first sample often wins because it came first.

To test for it, show every pair twice, once in each order. Say answer X comes first, then Y. Next time, Y comes first. A fair judge that prefers X picks "first" once and "second" once. If it picks "first" both times, the order changed its verdict.

The cell below runs both checks on made-up data. It makes 300 pairs, some easy to judge and some close calls. Two careful annotators label them all. Then a biased judge labels every pair in both orders. Three times in ten, this judge skips the reading and takes whichever response came first.

import numpy as np

rng = np.random.default_rng(1)
n = 300
first_is_better = rng.random(n) < 0.5           # hidden truth for each pair, as first shown
ease = rng.random(n)                            # some pairs are easy to judge, some are close calls

def picks_first(truth, position_bias=0.0):
    right = rng.random(n) < 0.55 + 0.4 * ease               # easy pairs judged right ~95%, close calls ~55%
    choice = np.where(right, truth, ~truth)
    lazy = rng.random(n) < position_bias                     # sometimes just takes whichever came first
    return np.where(lazy, True, choice)

a, b = picks_first(first_is_better), picks_first(first_is_better)
agree = (a == b).mean()
chance = a.mean() * b.mean() + (1 - a.mean()) * (1 - b.mean())
print(f"two annotators agree on {agree:.0%} of pairs (chance alone: {chance:.0%}); Cohen's kappa {(agree - chance) / (1 - chance):.2f}")
print(f"on the easiest third they agree on {(a == b)[ease > 2 / 3].mean():.0%}, on the hardest third {(a == b)[ease < 1 / 3].mean():.0%}")

# a judge with a position bias, shown every pair twice: original order, then swapped
order1 = picks_first(first_is_better, position_bias=0.3)
order2 = picks_first(~first_is_better, position_bias=0.3)   # the other response is now first
print(f"\njudge picks whichever response is shown first {np.concatenate([order1, order2]).mean():.0%} of the time")
print(f"its verdict changes when the order is swapped on {(order1 == order2).mean():.0%} of pairs")

The two annotators agree on 64% of pairs, against 50% by chance, so kappa is only 0.29. Most of the disagreement comes from close calls. On the easiest third of pairs they agree 71% of the time. On the hardest third it's 55%, barely above a coin flip. That's realistic. Much disagreement comes from real ambiguity, not carelessness. Clear guidelines reduce it. Teams also drop disputed pairs or give them less weight.

The biased judge picks the first response 67% of the time. A fair judge would pick it about half the time. Its verdict changes on 50% of pairs once the order is swapped. The usual fix for LLM judges is to judge both orders and average them, or to shuffle the order.

ana = [1, 1, 0, 1, 0, 0, 1, 1, 0, 1]   # 1 = preferred the tagline shown first
ben = [1, 0, 0, 1, 0, 1, 1, 1, 0, 1]

agree = sum(a == b for a, b in zip(ana, ben)) / len(ana)
ana_first = sum(ana) / len(ana)
ben_first = sum(ben) / len(ben)
chance = ana_first * ben_first + (1 - ana_first) * (1 - ben_first)
kappa = (agree - chance) / (1 - chance)
print("agree:", agree)
print("chance:", round(chance, 2))
print("kappa:", round(kappa, 2))

Where preferences come from

• Trained annotators follow written guidelines: be helpful, honest and harmless; follow the format; don't make things up. The guidelines are, in effect, the specification of the model.
• AI feedback. A strong model judges pairs against written principles. You'll meet this as Constitutional AI in a later lesson. It's cheaper and more consistent than people. But it inherits the judge model's biases.
• User signals. Thumbs up or down, and "which response do you prefer?" prompts in chat apps. These come in huge volume. They are also very noisy and lean toward what feels good in the moment.

What preferences pass on

A model trained on preference data picks up the biases in that data. Three show up again and again:

• Length bias. Raters often prefer longer answers that look thorough. Models drift toward wordy replies unless length is controlled for.
• Sycophancy, a word for flattery. People prefer answers that agree with them and praise them. Models trained on those choices learn to tell people what they want to hear.
• Confident style. Fluent, sure-sounding answers win comparisons even when they're wrong, unless raters check the facts.

For marketers: say you use an LLM judge to pick the best of several ad variants. Show each pair in both orders. Check how often it agrees with a sample of human picks. And watch whether it keeps choosing the longer copy.

The dataset for the next two lessons

The practice API has a real, if small, preference dataset. It comes from tiny-gpt-chat, the chat model from the last two lessons. The model wrote four answers to each of 360 "Describe the …" prompts. A brand reviewer then judged every answer against four guidelines:

1. Name the product that was asked about.
2. Don't repeat a word in a sentence.
3. Use a material the catalog uses for that kind of product.
4. Mention the repair promise.

Each pair in the dataset is the best of the four answers against the worst. The reviewer is code, not a person. You can call it too, as POST /v1/brand-check, the way you'd call an LLM judge. The cell below loads the pairs and shows one. It runs the brand check on that pair's rejected answer. Then it counts which guidelines the chosen and rejected answers break.

import collections
import requests

data = requests.get("https://api.gradient.test/v1/preferences").json()
print(f"{data['count']} pairs, for example:")
pair = data["data"][24]
print("  prompt:  ", pair["prompt"])
print("  chosen:  ", pair["chosen"])
print("  rejected:", pair["rejected"])

review = requests.post("https://api.gradient.test/v1/brand-check",
                       json={"product": "Basin headlamp", "text": pair["rejected"]}).json()
print("\nbrand check on the rejected answer:", review["flaws"])

def rule(flaw):
    if flaw.startswith("doesn't name"):
        return "names another product"
    if flaw.startswith("repeats"):
        return "repeats a word"
    if flaw == "no repair promise":
        return "no repair promise"
    return "wrong or no material"

for side in ("chosen", "rejected"):
    counts = collections.Counter(rule(f) for p in data["data"] for f in p[side + "_flaws"])
    print(f"\nviolations in {side} answers:", dict(counts.most_common()))

Even the chosen answers aren't perfect. Of the 293 chosen answers, 163 still miss the repair promise and 74 name another product. That's typical. Real preference data records which answer is better, not which is good.

The gap between the two sides depends on the guideline, though. Nearly every rejected answer (279) lacks the repair promise. That's a fixed phrase, easy to spot. Naming the right product takes a skill the model only partly has. Keep that in mind for the DPO lesson.