Index Ai Academy
Phase 11 · Data engineering

Labels and annotator agreement

Labels as measurements: agreement and Cohen's kappa, the pair of classes annotators confuse, votes and gold questions, and what wrong labels do to a model and a test score.

About 45 minutes · A free account opens the whole lesson.
Already have an account? Sign in to open this lesson

What it covers6

Why it matters

A supervised model learns whatever its labels say. If a tenth of them are wrong, it learns from a teacher who is wrong a tenth of the time, and its test score is marked against an answer key with the same fault. Labels are made by people reading a guideline, by programs, or by other models, and each of those makes mistakes of its own kind. This lesson measures those mistakes: how far two annotators agree, where they disagree, what a vote can and can't repair, and how to spend a limited review budget on the labels most likely to be wrong.

What you'll be able to do

  • Treat a label as a measurement with an error, and name where labels come from
  • Measure how well two annotators agree, and correct that agreement for chance with Cohen's kappa
  • Find the pair of classes annotators confuse, and decide between more votes, a clearer guideline and checking an annotator
  • Predict what scattered and one-sided label mistakes do to a trained model and to a test score
  • Use a model to pick out the labels most worth a second look
Quick check
A guess before you start. Two people each label the same 1,000 support tickets as urgent or routine, and their labels match on 90% of the tickets. About one ticket in twenty is urgent. What does the 90% tell you?
  1. That about 10% of all the tickets must be urgent
  2. That roughly 90% of the two sets of labels are correct
  3. Little, since most tickets are routine anyway
  4. That the labels are ready to train a model on
Show the answer

Little, since most tickets are routine anyway

When one answer is far more common than the other, people agree most of the time whether or not they read carefully. Two annotators who never marked a ticket urgent would agree on every one. Agreement has to be compared with the agreement that comes for free, which is what this lesson builds.

The idea: labels are measurements

Kappa rests on the chance of two events both happening, from Probability basics. The first check recalls the lesson on preference data, where two raters disagreed.

Labels are measurements

A label is the answer attached to a training example: this ticket is urgent, this review is positive. Supervised learning treats labels as the truth. They are closer to measurements, each made by someone or something applying a rule to an item, and like any measurement they carry an error.

Labels come from three places.

  • People. An annotator reads each item and applies a written guideline. Slow and costly, and the most accurate source when the guideline is clear.
  • Behavior. A click, a refund, a reply marked helpful. Free and plentiful, but it records what users did, which is not quite the question you asked.
  • Programs and models. A rule such as "mentions a refund, so billing", or a large model asked to label. Cheap and consistent, and wrong in consistent ways.

You rarely have the truth to compare labels with. If you did, you wouldn't need the labels. What you can do is give the same items to two annotators and count how often they agree. The cell below simulates that for 400 tickets, of which one in ten is truly urgent. Ana and Ben read every ticket and each gets 90% of them right. A third annotator marks every ticket routine without reading it.

import random

rng = random.Random(0)
truth = ["urgent" if rng.random() < 0.10 else "routine" for i in range(400)]

def annotate(truth, accuracy, seed):
    rng = random.Random(seed)
    other = {"urgent": "routine", "routine": "urgent"}
    return [t if rng.random() < accuracy else other[t] for t in truth]

def agreement(a, b):
    return sum(x == y for x, y in zip(a, b)) / len(a)

ana, ben = annotate(truth, 0.90, 1), annotate(truth, 0.90, 2)
lazy = ["routine"] * 400                            # marks every ticket routine, unread

for name, labels in (("truth", truth), ("Ana", ana), ("Ben", ben), ("lazy", lazy)):
    print(name.ljust(5), "marks", str(labels.count("urgent")).rjust(2), "of 400 urgent")
print(
    "of Ana's urgent marks,", sum(1 for a, t in zip(ana, truth) if a == t == "urgent"),
    "are truly urgent",
)
print("Ana and Ben agree on          ", round(agreement(ana, ben), 3))
print("Ana and the lazy one agree on ", round(agreement(ana, lazy), 3))
It runs right here in your browser, with nothing to install.

Two things to notice. First, Ana marks 68 tickets urgent where 40 are, and only 35 of her 68 are right. Being wrong on a tenth of 360 routine tickets produces about as many false alarms as there are urgent tickets in total. On a rare class, careful annotators still produce noisy labels.

Second, Ana and Ben agree on 0.818 of the tickets, and Ana agrees more, 0.83, with someone who never read one. Raw agreement can't tell care from laziness when one answer is common.

Quick check
Three annotators label the same eight tickets. What does this print: the agreement between each pair?
ana = [
    "urgent", "routine", "routine", "routine", "urgent", "routine", "routine",
    "routine",
]
ben = [
    "routine", "routine", "routine", "routine", "urgent", "routine", "urgent",
    "routine",
]
lazy = ["routine"] * 8

def agreement(a, b):
    return sum(x == y for x, y in zip(a, b)) / len(a)

print(agreement(ana, ben), agreement(ana, lazy), agreement(ben, lazy))
  1. 0.75 0.25 0.25
  2. 0.75 0.75 0.75
  3. 0.875 0.75 0.75
  4. 0.25 0.75 0.75
Show the answer

0.75 0.75 0.75

Ana and Ben differ on the first and seventh tickets: 6 of 8 match. Each of them marked two tickets urgent, so each matches the all-routine list on the other six. Three identical scores, and only one of the pairs involves two people who both read the tickets.

The fix is one you met in the lesson on preference data. Cohen's kappa asks how far the agreement rises above what the two annotators' own habits would produce by luck.

κ=pagree−pchance1−pchance\kappa = \frac{p_{agree} - p_{chance}}{1 - p_{chance}}

In plain words: take the agreement you observed, subtract the agreement chance alone would give, and divide by the most that agreement could have beaten chance by. 0 means no better than luck. 1 means they always agree.

The chance term comes from each annotator's label rates. For each label, multiply the share of items the first annotator gave it by the share the second gave it, then add those products up.

import random

rng = random.Random(0)
truth = ["urgent" if rng.random() < 0.10 else "routine" for i in range(400)]

def annotate(truth, accuracy, seed):
    rng = random.Random(seed)
    other = {"urgent": "routine", "routine": "urgent"}
    return [t if rng.random() < accuracy else other[t] for t in truth]

def agreement(a, b):
    return sum(x == y for x, y in zip(a, b)) / len(a)

def kappa(a, b):
    observed = agreement(a, b)
    chance = sum((a.count(k) / len(a)) * (b.count(k) / len(b)) for k in set(a) | set(b))
    return (observed - chance) / (1 - chance)

ana, ben, lazy = annotate(truth, 0.90, 1), annotate(truth, 0.90, 2), ["routine"] * 400
print(
    "Ana and Ben:      agreement", round(agreement(ana, ben), 3), "| kappa",
    round(kappa(ana, ben), 3),
)
print(
    "Ana and the lazy: agreement", round(agreement(ana, lazy), 3), "| kappa",
    round(kappa(ana, lazy), 3),
)
print()
for accuracy in (0.99, 0.95, 0.90, 0.80):
    a, b = annotate(truth, accuracy, 1), annotate(truth, accuracy, 2)
    print(
        "each right", accuracy, "of the time | agreement", round(agreement(a, b), 3),
        "| kappa", round(kappa(a, b), 3),
    )

rng = random.Random(0)
balanced = ["urgent" if rng.random() < 0.50 else "routine" for i in range(400)]
a, b = annotate(balanced, 0.90, 1), annotate(balanced, 0.90, 2)
print(
    "each right 0.9, half the tickets urgent | agreement", round(agreement(a, b), 3),
    "| kappa", round(kappa(a, b), 3),
)

Kappa separates the two pairs that raw agreement couldn't: 0.386 for two people who read the tickets, and 0.0 for the pairing with someone who didn't.

The table underneath carries a warning about kappa itself. Two annotators who are each right 95% of the time reach a kappa of only 0.592 here, because urgent tickets are rare. The last line gives the same two annotators, making the same slips, a task where half the tickets are urgent: the agreement is still 0.818 and kappa rises from 0.386 to 0.635. So a kappa has to be read with the class balance beside it. A common rule of thumb asks for 0.6 or more before labels are trusted, and above 0.8 for a test set.

Quick check
From the lesson on preference data: two annotators have a Cohen's kappa of 0. What does that mean?
  1. They disagree on every single item they both labeled
  2. One of the two annotators labeled every item incorrectly
  3. They agree no more often than their label rates would give by luck
  4. They agree on exactly half of the items they both labeled
Show the answer

They agree no more often than their label rates would give by luck

Kappa compares observed agreement with chance agreement. At 0 the two are equal: the annotators match as often as two people with the same label rates would match by accident. Their raw agreement can still be high, as with the annotator who marks everything routine.

Quick check
Two annotators agree on 90% of items. Each marks 10% of the items urgent. What does this print: chance agreement, then kappa?
agree, ana_urgent, ben_urgent = 0.90, 0.10, 0.10
chance = ana_urgent * ben_urgent + (1 - ana_urgent) * (1 - ben_urgent)
print(round(chance, 2), round((agree - chance) / (1 - chance), 2))
  1. 0.82 0.9
  2. 0.82 0.44
  3. 0.01 0.9
  4. 0.5 0.8
Show the answer

0.82 0.44

They both say urgent by chance 0.1 × 0.1 = 0.01 of the time and both say routine 0.9 × 0.9 = 0.81 of the time, so 0.82 of their agreement is free. They beat it by 0.08 out of a possible 0.18, which is 0.44. A 90% match on a lopsided task is a middling kappa.

Where the disagreement is

A single kappa tells you there is a problem. It doesn't tell you where. With more than two classes, disagreement is rarely spread evenly: it piles up on one boundary that the guideline left unclear.

To find it, count the items by the pair of labels the two annotators gave. That table is a confusion matrix between annotators. Its diagonal is agreement, and each cell off the diagonal is one kind of disagreement.

Below, 600 support tickets are labeled billing, bug or account by two annotators. The guideline doesn't say where a ticket about a failed login belongs.

import random

LABELS = ["billing", "bug", "account"]

def annotate(truth, slips, seed):
    # slips[(true, said)]: how often a really-`true` ticket is labeled `said`
    rng = random.Random(seed)
    out = []
    for t in truth:
        weights = [slips.get((t, said), 0.0) for said in LABELS]
        # the rest of the time the label is right
        weights[LABELS.index(t)] = 1 - sum(weights)
        out.append(rng.choices(LABELS, weights=weights)[0])
    return out

def kappa(a, b):
    observed = sum(x == y for x, y in zip(a, b)) / len(a)
    chance = sum((a.count(k) / len(a)) * (b.count(k) / len(b)) for k in set(a) | set(b))
    return (observed - chance) / (1 - chance)

rng = random.Random(4)
truth = [rng.choice(LABELS) for i in range(600)]
vague = {
    ("bug", "account"): 0.30, ("account", "bug"): 0.30, ("billing", "bug"): 0.03,
    ("billing", "account"): 0.03,
}
ana, ben = annotate(truth, vague, 1), annotate(truth, vague, 2)

print("rows: Ana's label. columns: Ben's label.")
print(" " * 10, *(k.rjust(8) for k in LABELS))
for x in LABELS:
    row = [sum(1 for p, q in zip(ana, ben) if p == x and q == y) for y in LABELS]
    print(x.ljust(10), *(str(n).rjust(8) for n in row))
print("kappa:", round(kappa(ana, ben), 3))

pairs = {}
for p, q in zip(ana, ben):
    if p != q:
        key = tuple(sorted((p, q)))
        pairs[key] = pairs.get(key, 0) + 1
print("disagreements by pair:", sorted(pairs.items(), key=lambda kv: -kv[1]))

clear = {
    ("bug", "account"): 0.05, ("account", "bug"): 0.05, ("billing", "bug"): 0.03,
    ("billing", "account"): 0.03,
}
ana, ben = annotate(truth, clear, 1), annotate(truth, clear, 2)
print(
    "after the guideline says where a failed login goes | kappa:",
    round(kappa(ana, ben), 3),
)

The billing row and column are nearly clean: 183 tickets where both said billing. The trouble is the block where bug meets account, 80 one way and 88 the other. Counted by pair, 168 of the disagreements are between account and bug, against 15 and 9 for the other two pairs.

That points at the repair. More annotators wouldn't help, since each new one would face the same unclear rule. The guideline needs a sentence and two or three examples for that boundary. With the rule written down, kappa rises from 0.52 to 0.837 without changing a single annotator.

This is the working loop of a labeling project: label a small batch twice, find the pair that is confused, fix the guideline, and repeat until the agreement holds. Only then label in bulk.

Try it now

Rows are Ana's label, columns are Ben's. The lit cells are the confused pair.

Quick check
Two annotators label product reviews as positive, negative or mixed. Kappa is 0.5, and nearly all of the disagreements are between negative and mixed. What should the team do next?
  1. Drop every review on which the two annotators disagreed
  2. Add a rule and examples for that boundary, then label a batch again
  3. Hire a third annotator and take the most common label per item
  4. Start bulk labeling, since a kappa of 0.5 is good enough
Show the answer

Add a rule and examples for that boundary, then label a batch again

Disagreement concentrated on one pair of classes is a guideline problem. A third annotator would be reading the same unclear rule, so the vote would be decided by habit. Writing the rule down, with examples, and measuring again is what moves the kappa.

In your own words
A vendor reports "our annotators agree 96% of the time". What two things do you ask for before you trust the labels?

Write a sentence or two, then compare it with a good answer in the lesson.

In code: votes, gold questions and noisy labels

The cells below build the repairs a labeling project uses: a vote, a score on gold questions, and a way to see what wrong labels do to a model.

Drill

Find the pair of labels two annotators confuse

Write confused_pairs(a, b). a and b are two annotators' labels for the same items. For every item where they differ, count the pair of labels involved, treating bug/account and account/bug as the same pair and writing each pair as a tuple in alphabetical order. Return a list of (pair, count), most common first. Pairs with the same count go in alphabetical order.

def confused_pairs(a, b):
    return []


ana = ["bug", "bug", "account", "billing", "account", "bug", "billing", "account"]
ben = ["account", "bug", "bug", "billing", "bug", "account", "bug", "account"]
print(confused_pairs(ana, ben))
In the lesson, your code is checked as you write it.

Combining several labels

Once the guideline is settled, annotators still slip: a misread line, a wrong key. The standard repair is to have several people label each item and take a majority vote, the label most of them chose.

A vote works when mistakes are independent. If each annotator is right 80% of the time, two of three being wrong on the same item is much rarer than one being wrong. The first column below shows it. The second column changes one thing: 440 of the 2,000 photos are too blurred to tell, so on those every annotator is guessing, and on the clear ones each is right 87.5% of the time.

import random

def vote(labels):
    return max(sorted(set(labels)), key=labels.count)

rng = random.Random(3)
truth = [rng.choice(["cat", "dog"]) for i in range(2000)]
other = {"cat": "dog", "dog": "cat"}
# photos too blurred to tell
unclear = [random.Random(12 + i).random() < 0.2 for i in range(2000)]
print("unclear photos:", sum(unclear), "of 2000")

print("annotators | all photos clear | some photos unclear")
for n in (1, 3, 5, 9):
    r1, r2, clear_right, unclear_right = random.Random(10), random.Random(10), 0, 0
    for t, u in zip(truth, unclear):
        labels = [t if r1.random() < 0.8 else other[t] for person in range(n)]
        clear_right += vote(labels) == t
        p = 0.5 if u else 0.875
        labels = [t if r2.random() < p else other[t] for person in range(n)]
        unclear_right += vote(labels) == t
    print(
        str(n).rjust(10), "|", str(round(clear_right / 2000, 3)).rjust(16), "|",
        str(round(unclear_right / 2000, 3)).rjust(8),
    )

With independent slips, three votes lift the labels from 0.8 right to 0.903, and nine reach 0.983. Each extra pair of annotators buys less than the last, and the cost grows in a straight line.

The second column starts in the same place, 0.802, and stalls: 0.865 with three, 0.892 with nine. On a photo nobody can read, nine guesses are no better than one. A vote repairs carelessness. It does not repair ambiguity. Those items need a clearer guideline, a class of their own such as "can't tell", or removal from the set.

The vote also tells you which items those are. An item on which the annotators split 5 to 4 is worth more attention than one they all agree on, so keep the vote counts, not only the winner. Items with a split vote go to adjudication: an expert decides, and the decision goes back into the guideline as an example.

Quick check
Three annotators label three tickets. What does this print: the share of annotators behind the most common label on each ticket?
votes = {
    "t1": ["bug", "bug", "account"],
    "t2": ["bug", "account", "billing"],
    "t3": ["account", "account", "account"],
}
top = [max(v.count(k) for k in set(v)) for v in votes.values()]
print([round(n / 3, 2) for n in top])
  1. [2, 1, 3]
  2. [0.67, 0.0, 1.0]
  3. [0.33, 0.33, 1.0]
  4. [0.67, 0.33, 1.0]
Show the answer

[0.67, 0.33, 1.0]

On t1 two of three chose bug: 0.67. On t3 all three agree: 1.0. On t2 every label has one vote, so the most common label has a share of 0.33, and calling any of them the winner would be arbitrary. That ticket should go to a reviewer.

A vote treats every annotator alike. To find out whether one of them is far worse than the rest, use gold questions: items whose right answer is already known, settled by an expert, and mixed into the queue unmarked. An annotator's score on the gold questions estimates their accuracy on everything else.

import random

rng = random.Random(7)
# tickets whose answer is known
gold = {f"g{i}": rng.choice(["urgent", "routine"]) for i in range(25)}
# hidden from the team
skill = {"ana": 0.95, "ben": 0.90, "cy": 0.70, "dee": 0.50}

for person, p in skill.items():
    r = random.Random(sum(map(ord, person)))
    other = {"urgent": "routine", "routine": "urgent"}
    answers = {q: (a if r.random() < p else other[a]) for q, a in gold.items()}
    right = sum(answers[q] == gold[q] for q in gold)
    print(
        person.ljust(4), "right on", right, "of 25 gold questions |",
        round(right / 25, 2), "| true skill", p,
    )

Twenty-five questions are enough to see that dee, at 11 of 25, is guessing, and that cy is well behind. They are not enough to rank ana above ben: 24 against 23 is one question. Use gold questions to catch the annotators who are far off, not to split hairs between good ones.

Drill

Score annotators on the gold questions

Write gold_scores(answers, gold, minimum). answers maps each annotator to a dictionary of item -> label for the items they labeled, gold and ordinary mixed together. gold maps each gold item to its right answer. Return a tuple of two things. First, a dictionary from annotator to their accuracy on the gold items they answered, rounded to 2 places, with annotators in alphabetical order, and None for an annotator who answered no gold items. Second, an alphabetical list of the annotators whose accuracy, before rounding, is below minimum. An annotator with no score isn't flagged.

def gold_scores(answers, gold, minimum):
    return {}, []


gold = {"g1": "urgent", "g2": "routine", "g3": "routine", "g4": "urgent"}
answers = {
    "ben": {
        "g1": "urgent", "g2": "urgent", "t1": "routine", "g3": "routine",
        "g4": "urgent",
    },
    "ana": {
        "g1": "urgent", "t1": "routine", "g2": "routine", "g3": "routine",
        "g4": "urgent", "t2": "urgent",
    },
}
print(gold_scores(answers, gold, 0.8))
In the lesson, your code is checked as you write it.

What wrong labels do to a model

Label noise is the share of labels that are wrong. How much it hurts depends less on how much there is than on its shape.

  • Scattered mistakes have no pattern: any label is as likely to be wrong as any other.
  • One-sided mistakes lean one way: urgent tickets marked routine by tired annotators, never the reverse.

The cell below trains on 2,000 points in two classes and tests on 2,000 fresh ones with clean labels. It spoils a growing share of the training labels in each of the two ways. The scattered mistakes are given to two models: a linear one, and a decision tree grown until it fits every training row.

import numpy as np
from sklearn.linear_model import LogisticRegression
from sklearn.tree import DecisionTreeClassifier

def make(n, seed):
    g = np.random.default_rng(seed)
    X = g.normal(size=(n, 2))
    return X, (X[:, 0] + 0.5 * X[:, 1] > 0).astype(int)

X_train, y_train = make(2000, 0)
X_test, y_test = make(2000, 1)                    # the test labels stay clean
g = np.random.default_rng(5)

print("wrong labels | scattered: linear, tree | one-sided: linear")
for share in (0.0, 0.1, 0.2, 0.3, 0.4):
    scattered = np.where(g.random(2000) < share, 1 - y_train, y_train)
    one_sided = y_train.copy()
    # the same number wrong, all 1 -> 0
    one_sided[np.where(y_train == 1)[0][: int(share * 2000)]] = 0
    linear = LogisticRegression().fit(X_train, scattered).score(X_test, y_test)
    tree = DecisionTreeClassifier(
        random_state=0
    ).fit(X_train, scattered).score(X_test, y_test)
    biased = LogisticRegression().fit(X_train, one_sided).score(X_test, y_test)
    print(
        str(int(share * 100)).rjust(10), "% |", str(round(float(linear), 3)).ljust(6),
        str(round(float(tree), 3)).ljust(6), "           |", round(float(biased), 3),
    )

Read the three columns in turn.

The linear model barely notices scattered mistakes: 0.999 with clean labels, 0.997 with a fifth of them wrong. Wrong labels on one side of the boundary are balanced by wrong labels on the other, so the best line stays where it was. Only at 40%, with the signal nearly drowned, does it fall to 0.881.

The tree has the capacity to memorize, and it memorizes the mistakes: 0.981 clean, 0.852 with a tenth of the labels wrong. A flexible model treats each wrong label as a real exception and carves a region around it. Large neural networks can do the same when trained for long enough.

The last column has the same number of wrong labels as the first, all leaning one way, and the same linear model drops to 0.868 at 10% and 0.724 at 20%. One-sided mistakes don't cancel. They move the boundary, and the model learns the annotators' bias as if it were the task.

Here are the three curves, drawn from the table above:

import numpy as np
import matplotlib.pyplot as plt

shares = [0, 10, 20, 30, 40]
linear = [0.999, 0.998, 0.997, 0.99, 0.881]
tree = [0.981, 0.852, 0.778, 0.707, 0.587]
one_sided = [0.999, 0.868, 0.724, 0.576, 0.514]
for name, ys in (
    ("linear, scattered", linear), ("tree, scattered", tree),
    ("linear, one-sided", one_sided)
):
    plt.plot(shares, ys, marker="o", label=name)
plt.xlabel("training labels wrong (%)")
plt.ylabel("accuracy on a clean test set")
plt.legend()
plt.show()
Quick check
In the plot, which line falls fastest as the share of wrong training labels grows past 10%?
  1. They fall together
  2. Tree with scattered mistakes
  3. Linear with one-sided mistakes
  4. Linear with scattered mistakes
Show the answer

Linear with one-sided mistakes

Between 10% and 30% the one-sided line falls from 0.868 to 0.576, while linear-scattered stays near 0.99.

Quick check
A team has 10% wrong labels in its training set. In which case is a linear model hurt most?
  1. The wrong labels are spread evenly across both classes at random
  2. The wrong labels are all on tickets that have very short text
  3. The wrong labels are all urgent tickets that were marked routine
  4. The wrong labels are all on tickets from one particular weekday
Show the answer

The wrong labels are all urgent tickets that were marked routine

Mistakes that all point from one class to the other shift the boundary the model learns: it sees fewer urgent examples than exist and becomes reluctant to predict urgent. Scattered mistakes largely cancel for a simple model. The other two options describe where the mistakes sit, which matters less than whether they lean one way.

Wrong labels in the test set do a different kind of damage. They don't change the model. They change what you believe about it. Here one model, trained on clean labels, is scored against test sets with a growing share of wrong labels.

import numpy as np
from sklearn.linear_model import LogisticRegression

def make(n, seed):
    g = np.random.default_rng(seed)
    X = g.normal(size=(n, 2))
    return X, (X[:, 0] + 0.5 * X[:, 1] > 0).astype(int)

X_train, y_train = make(2000, 0)
X_test, y_test = make(2000, 1)
model = LogisticRegression().fit(X_train, y_train)

g = np.random.default_rng(6)
for share in (0.0, 0.05, 0.1, 0.2):
    noisy_test = np.where(g.random(2000) < share, 1 - y_test, y_test)
    print(
        "test labels wrong:", str(int(share * 100)).rjust(2), "% | measured accuracy:",
        round(float(model.score(X_test, noisy_test)), 3),
    )

The model didn't change between the lines. Its measured accuracy fell from 0.999 to 0.895 when a tenth of the answer key was wrong, and to 0.794 at a fifth. A nearly perfect model is marked wrong every time the key is wrong, so no model can score much above the accuracy of the test labels themselves.

That sets the order of work. Clean the test set first, with several labels per item, a vote and adjudication, because every later decision is read off it. Training labels can be noisier, provided the mistakes are scattered and not one-sided.

Quick check
A two-class model is right 90% of the time. It is scored on a test set in which 10% of the labels are wrong, at random. What does this print: the accuracy that will be measured?
true_accuracy, wrong_labels = 0.90, 0.10
measured = true_accuracy * (1 - wrong_labels) + (1 - true_accuracy) * wrong_labels
print(round(measured, 2))
  1. 0.82
  2. 0.81
  3. 0.9
  4. 0.8
Show the answer

0.82

The model is marked correct in two cases. It is right and the label is right: 0.9 × 0.9 = 0.81. Or it is wrong and the label is wrong as well, which with two classes makes them match: 0.1 × 0.1 = 0.01. Together 0.82. Eight points of real accuracy have vanished from the report.

Worked example

Worked example: spend a review budget on the labels most likely to be wrong

A set of 2,000 labeled points has arrived and nobody can afford to check every label. Train a model, ask it which labels it finds hardest to believe, and review those first. Each row is scored by a model that never saw that row's label, so a wrong label can't vouch for itself. The run is done twice: once with scattered mistakes and once with a one-sided mistake.

import numpy as np
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import cross_val_predict

g = np.random.default_rng(0)
X = g.normal(size=(2000, 2))
# the classes overlap a little
score = X[:, 0] + 0.5 * X[:, 1] + g.normal(scale=0.4, size=2000)
truth = (score > 0).astype(int)

def review(wrong, given, name):
    # 5-fold: each row is scored by a model that never saw its label.
    proba = cross_val_predict(
        LogisticRegression(), X, given, cv=5, method="predict_proba",
    )
    # how likely the model finds the label that was given
    belief = proba[np.arange(2000), given]
    order = np.argsort(belief)                      # least believable first
    print(name, "| wrong labels:", int(wrong.sum()), "of 2000")
    for top in (100, 200, 400):
        found = int(wrong[order[:top]].sum())
        print(
            "   review the", top, "least believable | truly wrong:",
            str(found).rjust(3), "| share of all the wrong labels:",
            round(found / wrong.sum(), 2),
        )
    at_random = np.random.default_rng(1).permutation(2000)[:200]
    print(
        "   review 200 picked at random    | truly wrong:",
        str(int(wrong[at_random].sum())).rjust(3),
    )

scattered = g.random(2000) < 0.10
review(scattered, np.where(scattered, 1 - truth, truth), "scattered")

# every borderline positive was marked negative
one_sided = (score > 0) & (score < 0.45)
review(one_sided, np.where(one_sided, 0, truth), "one-sided")
In the lesson you can change this and run it.

With scattered mistakes the model is a good guide. Of the 100 labels it finds hardest to believe, 96 are truly wrong, and reviewing 200 rows, a tenth of the data, finds 144 of the 198 wrong labels. Reviewing 200 rows picked at random finds 13. The wrong labels it misses sit close to the boundary, where either label is believable. With the one-sided mistake the same method does much worse: 200 reviews find 85 of the 279. The model was trained on the shifted labels, learned the shifted boundary, and no longer finds those labels surprising. It still beats random review, which finds 27, but it can't be relied on to expose a mistake the annotators made consistently. That kind is caught by people: gold questions, and a second annotator's confusion table.

In your own words
What kind of labeling mistake does a majority vote fix, and what kind can't it fix?

Write a sentence or two, then compare it with a good answer in the lesson.

On the job: guidelines, a tool and a review budget

In a real project the labels arrive from a vendor or a team of annotators. Your job is to make the numbers in this lesson cheap to compute every time a batch comes in.

Where it comes up in a week

A vendor delivers 5,000 labeled rows with a note: "agreement 96%". You ask for the class balance and for kappa, then label 200 of the rows twice yourself and look at the table of pairs. The same week the guideline gets one more example for the confused boundary, and the new version number goes in the dataset's datasheet. Keep the vote counts, not only the winning label: the split items are the ones to adjudicate first.

Goes deeper in

In your own words
You can afford to have a tenth of a noisy data set labeled again. Where do you spend it, and why?

Write a sentence or two, then compare it with a good answer in the lesson.

Practice what you just read

  • 9 code examples you can edit and run
  • a practice exercise that checks your code
  • a 5-question quiz
  • 12 review cards for spaced repetition

Try a quiz question

Two radiographers label scans as normal or abnormal. About 2% of scans are abnormal. They agree on 97% of scans, and their Cohen's kappa is 0.2. How should that be read?

  1. Most of the agreement is what chance gives when nearly all scans are normal
  2. The labels are 97% accurate, and the kappa can be set aside
  3. The two radiographers disagree on about 80% of the scans
  4. A kappa of 0.2 means that 20% of the labels are wrong

Answer it at the end of the lesson to see if you're right, and why.

Before this lesson

Keep learning

Where this lesson leads.

575 lessons in thirteen phases. 176 are free without an account, and a free account opens the rest. See every lesson.

Phase 11 · about 34 hours of study

Professional practice

Design, ship and run real systems, work well in a team, and get hired 43 lessons 1 free
  1. System design fundamentals FreeRequirements, rough numbers, the standard parts and where systems break.40 min
  2. Caches, queues and background jobsSpeed reads with caches, and move slow work to queues that retry safely.40 min
  3. Designing AI systemsLatency, cost, routing, fallbacks and guardrails for features built on models.40 min
  4. Containers and deploymentPackage an app as an image, configure it with the environment, and roll out safely.45 min
  5. CI/CD pipelinesTest every change automatically, and ship with gates, flags and rollbacks.40 min
  6. Docker ComposeSeveral containers together: service names, volumes, secrets and start order.40 min
  7. Kubernetes basicsDesired state, Deployments and Services, labels, probes and rolling updates.40 min
  8. Cloud for AI engineersAccess policies and least privilege, storage, compute choices, managed model endpoints, secrets and budgets.30 min
  9. Metrics, dashboards and alert rulesCounters, gauges and histograms, a rate from a counter, a percentile from buckets, a dashboard panel and an alert rule that fires on a sustained condition.40 min
  10. Reliability and incidentsSLOs, error budgets, alerts, on-call and blameless postmortems.40 min
  11. Incident lab: from the page to the postmortemOne incident run from the page to the all-clear: decide the mitigation, work out the impact in failed requests and error budget, and write a blameless postmortem, a status update and a runbook entry.40 min
  12. Measuring an AI feature from its logsTyped views over event logs, metric specs, cost from a price table, and rollups you can rerun.40 min
  13. A/B testing a model changeHash assignment, per-user results from raw logs, sample ratio checks, guardrails and a decision rule.40 min
  14. Columnar files and a local warehouseRows against columns, the encodings that shrink a column, Parquet's row groups and the ranges that let a query skip them, and SQL over files with DuckDB.40 min
  15. Data quality checks and contractsSix properties of a delivery you can test, each as a count of rows that break it, a contract with the producer, and when to block, quarantine or warn.40 min
  16. Transformations: staging, models and testsRaw, staging, facts and marts as SQL models run in dependency order in DuckDB, tests as queries that count the rows that break a rule, an incremental load that is safe to rerun, and the same models as dbt files.45 min
  17. Streams and windowsEvents that never stop arriving: event time against arrival time, tumbling, sliding and session windows, a watermark for late events, and counting each event once.40 min
  18. Labels and annotator agreementLabels as measurements: agreement and Cohen's kappa, the pair of classes annotators confuse, votes and gold questions, and what wrong labels do to a model and a test score.45 min
  19. Experiment tracking and reproducible runsA run record of settings, seed, data and code behind every score, a run log in a dozen lines, and comparing settings across seeds.40 min
  20. Hyperparameter searchWhat a grid costs, random search on a log scale, stopping weak settings early, and why the winner's validation score flatters it.40 min
  21. Versioning data and modelsFiles stored under the hash of their contents, a manifest that names a whole dataset, and lineage from data and code to each model.35 min
  22. A model registry and promotionVersions under one name, an alias for the live one, gates a candidate must pass, group by group, and a rollback that moves a pointer.40 min
  23. Feature pipelines and training-serving skewWhy a model that passes offline fails live: statistics refit at serving, two code paths, and features from the future, with the fix for each.40 min
  24. Batch and online inferencePredict ahead of time or on request: when the inputs exist, latency near a server's limit, batching what waits, cost, and a hybrid with a freshness limit.40 min
  25. Monitoring a model: drift and decayWhy a model worsens with no code change: data drift and concept drift, a measure of how far inputs have moved, thresholds set from noise, and labels that arrive late.45 min
  26. Retraining and pipelinesWhen to retrain and on which weeks, training as steps in dependency order, a cache that reruns only what changed, and guards before and after.45 min
  27. Designing ML systemsOne method for an ML design, goal and metric to retraining, with the sums that size it: requests a second at peak, a year of feature storage, a label delay and a latency budget.45 min
  28. Datasheets: documenting a datasetThe document that travels with a dataset: a profile by column and by group, claims a program can check, splits that don't leak, a scan for personal data, and the license and limits only people can write.45 min
  29. Curation, active learning and synthetic dataThree levers on a training set: removing copies and broken rows, letting a model choose which rows to label, and manufacturing rows, which helps only when their source knows more than you do.45 min
  30. Measuring fairnessBreak a model's results down by group: selection rate, true and false positive rates and precision, why they can't all be equal, what each fix costs, and when a group is too small to judge.40 min
  31. Privacy engineeringWhy removing names isn't anonymity, how to hold less (minimization, retention, keyed pseudonyms), the idea behind differential privacy and its budget, what a model remembers, and a deletion that reaches every store.40 min
  32. A red-team labAttack a simulated assistant methodically: eight kinds of attack run as a suite, success rates beside ordinary requests blocked, fresh attacks to test the fixes, where a defense belongs, and findings kept as release tests.45 min
  33. What regulation asks of an engineerRisk tiers that follow a system's use, the requirements for high-risk systems as artifacts an engineer builds, evidence tied to a model version, a model card a program can check, and legal dates kept as data.35 min
  34. Git for teamsBranches, pull requests, rebasing and how a three-way merge finds conflicts.40 min
  35. Writing for engineersREADMEs, design docs, commit messages and bug reports people can act on.35 min
  36. Scoping and estimating workA vague request cut into tickets with acceptance criteria and a first shippable slice, estimates as ranges, an honest update for a slip, and a ticket linked to its pull request.30 min
  37. Roles, résumé and the searchWhat AI, ML and data engineering postings ask for, résumé bullets that carry a number, a 100-word project summary, a take-home as a portfolio piece and how to read an offer.35 min
  38. Interviews and your portfolioShow your work, and solve interview problems the way professionals do.40 min
  39. Code reviewWhat to look for, how to give and take feedback, and automating the routine checks.35 min
  40. Reading unfamiliar codeFind your way around a big codebase fast, with tools that map it for you.35 min
  41. Working with AI coding toolsGive agents good context, keep tasks small, and check every change they make.35 min
  42. Capstone: ship a production AI serviceAn AI-backed service with validation, caching, rate limits, fallbacks, logging and evals.45 min
  43. Capstone: take a model to productionA churn model through a data contract, a tracked run, a registry gate by slice, hybrid serving, a PSI monitor and one retrain, shipped with a datasheet, a model card, a fairness audit, a design doc and a runbook.45 min

Never trained an AI? Teach a small one to write, right in your browser. It takes about 20 seconds and needs no account. Teach it to write