Measuring fairness
Break a model's results down by group: selection rate, true and false positive rates and precision, why they can't all be equal, what each fix costs, and when a group is too small to judge.
What it covers7
Why it matters
A model that decides who gets a loan, an interview or a medical referral can work well on average and badly for some of the people it judges. Nobody will tell you: the dashboard shows one accuracy number and it looks fine. Measuring fairness is the engineering part of a larger question. It can't say what is fair, which is for people and the law to decide. It can say, with numbers, who gets which errors and what each fix would cost, and without those numbers that decision gets made blind.
What you'll be able to do
- Break a model's results down by group, and see what an overall number hides
- Compute selection rate, true positive rate, false positive rate and precision for each group
- Name the fairness definition each of those measures belongs to
- Show why the definitions can't all hold at once, and what each choice costs
- Tell a real gap from noise in a small group, and report the difference honestly
Show the answer
No: an average can hide a group that it serves badly
An overall number is an average weighted by how many people are in each group. If three quarters of applicants are in one group, the model can be much worse for the rest and the average barely moves. The only way to know is to compute the numbers again, group by group.
The idea: one number hides the groups
Every fairness measure is a rate computed per group, the kind of count-and-divide you did in Describing data. The recall check below uses the true positive rate.
One number hides the groups
The data for this lesson is made up, so that the truth is known. Eight thousand people applied for a loan: 6,000 in group A and 2,000 in group B. For each, repay says what really happened, and score is a model's guess from 0 to 1. The lender approves a score of 0.5 or more.
Two things differ between the groups, both by construction. People in A repay more often. And the model's scores are noisier for B, as a model's often are for a group it saw less of in training.
Setup (runs before every cell) · 19 lines
The lesson runs this once, before its first cell, and every cell, drill and example starts with what it defines. A cell that changes a setup object changes it for the cells after it, until the page reloads.
import numpy as np def make(n_a, n_b, seed, noise_b=0.22): """Two groups of applicants: repay is the truth, score is the model's guess.""" rng = np.random.default_rng(seed) group = np.array(["A"] * n_a + ["B"] * n_b) repay = rng.random(n_a + n_b) < np.where(group == "A", 0.70, 0.50) spread = np.where(group == "A", 0.14, noise_b) # less sure about group B noise = rng.normal(0, 1, n_a + n_b) * spread score = np.clip(0.35 + 0.30 * repay + noise, 0, 1) return group, repay, score def rates(group, repay, approve, g): mine = group == g y, p = repay[mine], approve[mine] return {"people": int(mine.sum()), "repay": y.mean(), "approved": p.mean(), "tpr": p[y].mean(), "fpr": p[~y].mean(), "precision": y[p].mean()}
group, repay, score = make(6000, 2000, 0) approve = score >= 0.5 print("accuracy, everyone:", round(float((approve == repay).mean()), 3)) print() print("group | people | really repay | approved | accuracy") for g in "AB": r = rates(group, repay, approve, g) print( g.ljust(5), "|", str(r["people"]).rjust(6), "|", str(round(float(r["repay"]), 3)).ljust(12), "|", str(round(float(r["approved"]), 3)).ljust(8), "|", round(float((approve == repay)[group == g].mean()), 3), )
The model is right for 84.2% of applicants. Split by group, that is 86.9% for A and 76.0% for B. The overall number sits close to A's, since A is three quarters of the people.
Two other columns differ as well. A is approved 64.8% of the time and B 50.4%. And A really does repay more often: 70.1% against 50.1%. So some of the difference in approvals reflects a difference in the world, and some reflects a model that is worse for B. Telling those apart is most of this lesson.
To measure anything by group you need each person's group in the evaluation data. That is separate from whether the model uses it. A model that isn't given the group can still learn it from a ZIP code or a first name, so leaving it out guarantees nothing, and makes the checking impossible.
Show the answer
Of the people who really repay, the share it flags
Recall asks about the real positives: how many did the model find? In fairness work the same number is called the true positive rate. Here it means: of the people who would repay, what share were approved. The second option is precision, the third accuracy, and the fourth the false positive rate.
Three meanings of fair
"Fair" has more than one precise meaning, and each compares a different number between groups.
| Compare between groups | The number | The idea |
|---|---|---|
| Demographic parity | Selection rate: the share approved | Each group is approved equally often |
| Equal opportunity | True positive rate: approved, of those who repay | People who deserve a yes get one equally often |
| Equalized odds | True positive rate and false positive rate | Both kinds of error are shared equally |
| Predictive parity | Precision: repay, of those approved | A yes means the same thing in each group |
Uses the setup above, 19 lines.
group, repay, score = make(6000, 2000, 0) approve = score >= 0.5 a, b = rates(group, repay, approve, "A"), rates(group, repay, approve, "B") names = { "approved": "approved (selection rate)", "tpr": "approved, of those who repay", "fpr": "approved, of those who don't", "precision": "repay, of those approved", } print("measure".ljust(30), "| A | B | gap A - B") for key, name in names.items(): print( name.ljust(30), "|", f"{a[key]:.3f}", "|", f"{b[key]:.3f}", "|", f"{a[key] - b[key]:+.3f}", ) print() print( "approval rate of B as a share of A's:", round(float(b["approved"] / a["approved"]), 2), )
By every one of these measures the two groups are treated differently.
The second row is the one a person would feel first. Of the people in B who would have repaid, 76.4% were approved. In A it was 86.9%. One creditworthy applicant in four from group B was turned away.
The third row is the opposite error. Among people who went on not to repay, the model approved 24.3% in B and 13.0% in A. B gets more wrong approvals as well as more wrong refusals, which is what a noisier score does.
The last line gives a ratio that has a rule of thumb attached. In United States employment guidance, a group selected at less than four fifths of the highest group's rate is treated as a sign of adverse impact. Here B is approved at 0.78 of A's rate, just under that line. It is a screening rule, and not a verdict or a universal law.
The audit table from the cell above. Lit: where B is treated worse on errors.
group = ["A", "A", "A", "B", "B", "B"] repay = [1, 1, 0, 1, 1, 0] approve = [1, 1, 1, 1, 0, 0] for g in "AB": rows = [(r, a) for gg, r, a in zip(group, repay, approve) if gg == g] print( g, round(sum(a for r, a in rows) / len(rows), 2), round(sum(a for r, a in rows if r) / sum(r for r, a in rows), 2), )
Show the answer
A 1.0 1.0 B 0.33 0.5
All three in A are approved, so A's selection rate is 1.0, and both of A's repayers are approved: 1.0. In B one of three is approved, 0.33. B has two people who repay and one of them is approved, so its true positive rate is 0.5. The two rates have different denominators: everyone in the group, and only those who repay.
Write a sentence or two, then compare it with a good answer in the lesson.
In code: group rates, thresholds and what a sample can show
The data is made up so the truth is known. The helpers make and rates are in the setup, so each cell shows only what it adds.
Rates for each group
Write group_rates(y_true, y_pred, group). The three lists line up: y_true holds 1 for people who really are positive, y_pred holds 1 for people the model selected. Return a dictionary from each group to a dictionary with "selection" (the share of the group selected), "tpr" (selected, among the group's real positives) and "fpr" (selected, among the group's real negatives). Where a rate has nothing to divide by, use None.
Uses the setup above, 19 lines.
def group_rates(y_true, y_pred, group): return {} print( group_rates([1, 1, 0, 1, 1, 0], [1, 1, 1, 1, 0, 0], ["A", "A", "A", "B", "B", "B"]) )
You can't have them all
The obvious fix is to give each group its own threshold. The cell tries two versions. One lowers B's threshold until B is approved as often as A. The other lowers it until B's true positive rate matches A's. The last column is the lender's result, counting +1 for a loan repaid and −3 for a default.
Uses the setup above, 19 lines.
group, repay, score = make(6000, 2000, 0) def threshold_for(scores, wanted, of=None): """The threshold, to two decimals, whose approval rate is closest to wanted. The rate is taken over everyone, or only over the people in 'of'.""" best = None for t in np.arange(0.20, 0.81, 0.01): got = (scores >= t).mean() if of is None else (scores >= t)[of].mean() if best is None or abs(got - wanted) < best[0]: best = (abs(got - wanted), round(float(t), 2)) return best[1] def report(name, t_a, t_b): approve = np.where(group == "A", score >= t_a, score >= t_b) a, b = rates(group, repay, approve, "A"), rates(group, repay, approve, "B") gaps = " ".join( f"{a[k] - b[k]:+.3f}" for k in ("approved", "tpr", "fpr", "precision") ) # +1 for a loan repaid, -3 for a default profit = int((approve & repay).sum() - 3 * (approve & ~repay).sum()) print(name.ljust(26), "|", f"{t_a:.2f} {t_b:.2f}", "|", gaps, "|", profit) b = group == "B" a_now = rates(group, repay, score >= 0.5, "A") print( "policy".ljust(26), "| thresholds | gaps A - B: approved, tpr, fpr, precision | lender's profit", ) report("one threshold for all", 0.5, 0.5) report("equal approval rates", 0.5, threshold_for(score[b], a_now["approved"])) report( "equal true positive rates", 0.5, threshold_for(score[b], a_now["tpr"], of=repay[b]), )
Each policy closes the gap it was aimed at, and opens another.
Equal approval rates brings the first gap to +0.008. To get there, B's threshold drops to 0.40, and the gap in false positive rates goes from −0.113 to −0.270. The extra approvals in B go disproportionately to people who won't repay. A loan someone can't repay harms them too. The lender's profit falls from 2,996 to 2,641.
Equal true positive rates does much the same from a threshold of 0.41.
In both, the precision gap grows, to about +0.25: an approval now means something quite different in the two groups.
This is not a failure of threshold-picking. It was proved in 2016 that when groups have different base rates, and the model isn't perfect, equal precision, equal true positive rates and equal false positive rates can't all hold at once. Every system chooses which to give up, whether or not anyone decided.
So the choice is a policy decision. Which error is worse here, and for whom? That belongs to the people accountable for the system, often with lawyers in the room. The engineer's part is the table above: every option, each gap, and the cost.
Show the answer
The true positive rate
"Qualified candidates get through equally often" is a statement about the real positives in each group, which is the true positive rate. That is equal opportunity. Equal selection rates would pass the same share of each group whether or not they were qualified, and equal precision would make a pass mean the same thing in each group.
The four-fifths screen
Write impact_ratio(selection). selection maps each group to its selection rate. Return a tuple (ratio, lowest, passes): the lowest rate divided by the highest, the name of the group with the lowest rate (the alphabetically first, if several tie), and True when the ratio is at least 0.8. If the highest rate is 0, nobody was selected: return a ratio of 1.0, and it passes.
Uses the setup above, 19 lines.
def impact_ratio(selection): return (1.0, None, True) print(impact_ratio({"A": 0.648, "B": 0.504}))
How sure are you, and what to fix
A gap measured on a test set is an estimate, and for a small group it is a poor one. The first half of the cell draws 200 different samples of applicants and measures the true positive rate gap on each. Only the size of group B changes.
The second half tries the fix that thresholds couldn't deliver: a model that is as good for B as it is for A.
Uses the setup above, 19 lines.
group, repay, score = make(6000, 2000, 0) def tpr_gap(n_b, seed, noise_b=0.22): g, y, s = make(6000, n_b, seed, noise_b) approve = s >= 0.5 return rates(g, y, approve, "A")["tpr"] - rates(g, y, approve, "B")["tpr"] print("the gap in true positive rates, measured on 200 different samples of applicants") for n_b in (60, 200, 2000): gaps = np.array([tpr_gap(n_b, 100 + seed) for seed in range(200)]) print( "group B has", str(n_b).rjust(4), "people | 95% of the measured gaps fall between", f"{np.percentile(gaps, 2.5):+.3f}", "and", f"{np.percentile(gaps, 97.5):+.3f}", ) print() print( "the same check after the model is as good for B as for A (more of B's data to " "learn from):" ) group, repay, score = make(6000, 2000, 0, noise_b=0.14) approve = score >= 0.5 a, b = rates(group, repay, approve, "A"), rates(group, repay, approve, "B") for key in ("approved", "tpr", "fpr", "precision"): print( key.ljust(9), "| A", f"{a[key]:.3f}", "| B", f"{b[key]:.3f}", "| gap", f"{a[key] - b[key]:+.3f}", ) print( "lender's profit, with one threshold for all:", int((approve & repay).sum() - 3 * (approve & ~repay).sum()), )
With 2,000 people in B, the measured gap lands between +0.081 and +0.135: clearly there. With 60 people, it lands anywhere from −0.020 to +0.237. One sample would show a huge gap, and another would show B doing slightly better. Sixty people contain about thirty who repay, so each of them moves the rate by three points.
This cuts both ways. A small group can't prove a gap, and it can't prove there isn't one. The honest report says "too few to tell", with the count, and asks for more data.
Now the second half. When the model's scores are as reliable for B as for A, the true positive rate gap falls from +0.105 to +0.007, and the false positive rate gap to −0.014, with one threshold for everyone. The lender's profit rises from 2,996 to 3,388. That gap was never about thresholds. It was a worse model for the group with less data, and the fix was data.
Two gaps remain: approvals, at +0.144, and precision. Those come from the base rates, which no model changes. Whether a difference in the world should be carried through into decisions is again a question for people. What you can now tell them is which part of the gap was the model's fault.
for positives in (20, 2000): print(positives, round(100 / positives, 2), "points")
Show the answer
20 5.0 points 2000 0.05 points
A rate out of 20 people moves in steps of 5 percentage points: one more approval takes it from 75% to 80%. Out of 2,000 the step is a twentieth of a point. A gap of 8 points between a large group and a group with 20 positives is one or two people, which is no evidence of anything.
Show the answer
A model that is as accurate for B as for A
The gap came from scores that were noisier for B, so that more of B's good applicants fell below the line. Making the model as reliable for B, in practice by collecting more of B's data, fixed the cause. A lower threshold treats the symptom and raises false approvals. Removing the group from the test data only removes the ability to see.
The four rates as grouped bars, from the audit table:
Uses the setup above, 19 lines.
import matplotlib.pyplot as plt measures = ["approved", "tpr", "fpr", "precision"] a = [0.648, 0.869, 0.130, 0.940] b = [0.504, 0.764, 0.243, 0.760] xs = range(4) plt.bar([x - 0.2 for x in xs], a, width=0.4, label="group A") plt.bar([x + 0.2 for x in xs], b, width=0.4, label="group B") plt.xticks(xs, measures) plt.ylabel("rate") plt.legend() plt.show()
Worked example: an audit that knows what it can't say
An audit function prints each group's true and false positive rates beside the largest group's, and gives a verdict on each gap only where there are at least 100 people to measure it on. Here the model has been given a flaw: it marks down new customers in group B. The audit runs three times: by group, by group and customer type, and on a smaller sample.
Uses the setup above, 19 lines.
group, repay, score = make(6000, 2000, 0) def audit(group, repay, approve, min_count=100, limit=0.05): """Per-group rates, with a verdict on each gap where there are enough people.""" names = sorted(set(group)) table = {g: rates(group, repay, approve, g) for g in names} biggest = max(names, key=lambda g: table[g]["people"]) for g in names: r = table[g] counts = { "tpr": int(repay[group == g].sum()), "fpr": int((~repay[group == g]).sum()), } line = [g.ljust(12), str(r["people"]).rjust(5)] for key in ("tpr", "fpr"): gap = r[key] - table[biggest][key] if g == biggest: verdict = "reference" elif counts[key] < min_count: verdict = "too few to say (" + str(counts[key]) + ")" else: verdict = "GAP " + f"{gap:+.3f}" if abs( gap ) > limit else "ok " + f"{gap:+.3f}" line.append(key + " " + f"{r[key]:.3f}" + " " + verdict.ljust(22)) print(" | ".join(line)) rng = np.random.default_rng(5) # a second attribute: is this a new customer? new_customer = rng.random(len(group)) < 0.25 # the model marks new B customers down score2 = np.where((group == "B") & new_customer, np.clip(score - 0.12, 0, 1), score) approve = score2 >= 0.5 print("by group") audit(group, repay, approve) print() print("by group and customer type") both = np.array([g + (", new" if n else ", old") for g, n in zip(group, new_customer)]) audit(both, repay, approve) print() print("a group of 150, split the same way") keep = np.concatenate( [np.where(group == "A")[0][:2000], np.where(group == "B")[0][:150]] ) audit(both[keep], repay[keep], approve[keep])
By group, B's true positive rate is 14 points below A's, which is already a finding. Splitting by customer type shows where it lives: new customers in B are approved 57.3% of the time when they would repay, a gap of almost 30 points, while new customers in A are no different from old ones. A gap can hide inside a group, and the combinations are where to look. The third table is the same audit on a sample with 150 people from B. The rate for new B customers reads 0.412, which looks even worse, and it rests on 17 people. The function declines to call it, and says how many it had. An audit that printed "GAP" there would be right by luck, and one that printed "ok" for the old customers, on 57 people, would be wrong. Reporting the count beside every rate is what makes the rest of the table believable.
Write a sentence or two, then compare it with a good answer in the lesson.
On the job: Fairlearn, a model card and who decides
Your part is the audit: every option, each gap and the cost. The choice belongs to the people accountable.
Where it comes up in a week
A reviewer asks for the fairness numbers before a launch. You run the audit by group and by one more attribute, mark any group too small to tell, and attach the table to the model card. If the gaps come from a worse model for a smaller group, the fix is data. If they come from base rates, that is a policy question, and you bring the options with their costs.
The fairness section of a model card
The loan model's audit table is done. Write the section of its model card that a reviewer and a regulator would read: what you measured, what you found, and what you are leaving to the people accountable for the system.
It must
- Gives at least two numbers from the audit table
- Says which gaps come from the model and which from base rates
- Names a group or attribute too small to judge
Go further
- Includes the four-fifths ratio and says it is a screen, not a verdict
The template
## What was measured The groups, the evaluation data and the four measures. ## What was found Two gaps with their numbers, and which part of each is the model's. ## Limits What the audit cannot show, including any group too small to tell. ## Decisions for the owners The tradeoff someone must choose, and the cost of each option.
Goes deeper in
- Privacy engineering: what to protect while you collect the group labels an audit needs.
- What regulation asks of an engineer: the model card as a required artifact.
- A model registry: slice gates in promotion.
Write a sentence or two, then compare it with a good answer in the lesson.
Practice what you just read
- 6 code examples you can edit and run
- a practice exercise that checks your code
- a 5-question quiz
- 12 review cards for spaced repetition
Try a quiz question
A model has the same true positive rate and the same false positive rate in two groups. One group has many more real positives than the other. What must be true?
Before this lesson
Where this lesson leads.
575 lessons in thirteen phases. 176 are free without an account, and a free account opens the rest. See every lesson.
Phase 11 · about 34 hours of study
Professional practice
Design, ship and run real systems, work well in a team, and get hired
43 lessons 1 free
- System design fundamentals FreeRequirements, rough numbers, the standard parts and where systems break.40 min
- Caches, queues and background jobsSpeed reads with caches, and move slow work to queues that retry safely.40 min
- Designing AI systemsLatency, cost, routing, fallbacks and guardrails for features built on models.40 min
- Containers and deploymentPackage an app as an image, configure it with the environment, and roll out safely.45 min
- CI/CD pipelinesTest every change automatically, and ship with gates, flags and rollbacks.40 min
- Docker ComposeSeveral containers together: service names, volumes, secrets and start order.40 min
- Kubernetes basicsDesired state, Deployments and Services, labels, probes and rolling updates.40 min
- Cloud for AI engineersAccess policies and least privilege, storage, compute choices, managed model endpoints, secrets and budgets.30 min
- Metrics, dashboards and alert rulesCounters, gauges and histograms, a rate from a counter, a percentile from buckets, a dashboard panel and an alert rule that fires on a sustained condition.40 min
- Reliability and incidentsSLOs, error budgets, alerts, on-call and blameless postmortems.40 min
- Incident lab: from the page to the postmortemOne incident run from the page to the all-clear: decide the mitigation, work out the impact in failed requests and error budget, and write a blameless postmortem, a status update and a runbook entry.40 min
- Measuring an AI feature from its logsTyped views over event logs, metric specs, cost from a price table, and rollups you can rerun.40 min
- A/B testing a model changeHash assignment, per-user results from raw logs, sample ratio checks, guardrails and a decision rule.40 min
- Columnar files and a local warehouseRows against columns, the encodings that shrink a column, Parquet's row groups and the ranges that let a query skip them, and SQL over files with DuckDB.40 min
- Data quality checks and contractsSix properties of a delivery you can test, each as a count of rows that break it, a contract with the producer, and when to block, quarantine or warn.40 min
- Transformations: staging, models and testsRaw, staging, facts and marts as SQL models run in dependency order in DuckDB, tests as queries that count the rows that break a rule, an incremental load that is safe to rerun, and the same models as dbt files.45 min
- Streams and windowsEvents that never stop arriving: event time against arrival time, tumbling, sliding and session windows, a watermark for late events, and counting each event once.40 min
- Labels and annotator agreementLabels as measurements: agreement and Cohen's kappa, the pair of classes annotators confuse, votes and gold questions, and what wrong labels do to a model and a test score.45 min
- Experiment tracking and reproducible runsA run record of settings, seed, data and code behind every score, a run log in a dozen lines, and comparing settings across seeds.40 min
- Hyperparameter searchWhat a grid costs, random search on a log scale, stopping weak settings early, and why the winner's validation score flatters it.40 min
- Versioning data and modelsFiles stored under the hash of their contents, a manifest that names a whole dataset, and lineage from data and code to each model.35 min
- A model registry and promotionVersions under one name, an alias for the live one, gates a candidate must pass, group by group, and a rollback that moves a pointer.40 min
- Feature pipelines and training-serving skewWhy a model that passes offline fails live: statistics refit at serving, two code paths, and features from the future, with the fix for each.40 min
- Batch and online inferencePredict ahead of time or on request: when the inputs exist, latency near a server's limit, batching what waits, cost, and a hybrid with a freshness limit.40 min
- Monitoring a model: drift and decayWhy a model worsens with no code change: data drift and concept drift, a measure of how far inputs have moved, thresholds set from noise, and labels that arrive late.45 min
- Retraining and pipelinesWhen to retrain and on which weeks, training as steps in dependency order, a cache that reruns only what changed, and guards before and after.45 min
- Designing ML systemsOne method for an ML design, goal and metric to retraining, with the sums that size it: requests a second at peak, a year of feature storage, a label delay and a latency budget.45 min
- Datasheets: documenting a datasetThe document that travels with a dataset: a profile by column and by group, claims a program can check, splits that don't leak, a scan for personal data, and the license and limits only people can write.45 min
- Curation, active learning and synthetic dataThree levers on a training set: removing copies and broken rows, letting a model choose which rows to label, and manufacturing rows, which helps only when their source knows more than you do.45 min
- Measuring fairnessBreak a model's results down by group: selection rate, true and false positive rates and precision, why they can't all be equal, what each fix costs, and when a group is too small to judge.40 min
- Privacy engineeringWhy removing names isn't anonymity, how to hold less (minimization, retention, keyed pseudonyms), the idea behind differential privacy and its budget, what a model remembers, and a deletion that reaches every store.40 min
- A red-team labAttack a simulated assistant methodically: eight kinds of attack run as a suite, success rates beside ordinary requests blocked, fresh attacks to test the fixes, where a defense belongs, and findings kept as release tests.45 min
- What regulation asks of an engineerRisk tiers that follow a system's use, the requirements for high-risk systems as artifacts an engineer builds, evidence tied to a model version, a model card a program can check, and legal dates kept as data.35 min
- Git for teamsBranches, pull requests, rebasing and how a three-way merge finds conflicts.40 min
- Writing for engineersREADMEs, design docs, commit messages and bug reports people can act on.35 min
- Scoping and estimating workA vague request cut into tickets with acceptance criteria and a first shippable slice, estimates as ranges, an honest update for a slip, and a ticket linked to its pull request.30 min
- Roles, résumé and the searchWhat AI, ML and data engineering postings ask for, résumé bullets that carry a number, a 100-word project summary, a take-home as a portfolio piece and how to read an offer.35 min
- Interviews and your portfolioShow your work, and solve interview problems the way professionals do.40 min
- Code reviewWhat to look for, how to give and take feedback, and automating the routine checks.35 min
- Reading unfamiliar codeFind your way around a big codebase fast, with tools that map it for you.35 min
- Working with AI coding toolsGive agents good context, keep tasks small, and check every change they make.35 min
- Capstone: ship a production AI serviceAn AI-backed service with validation, caching, rate limits, fallbacks, logging and evals.45 min
- Capstone: take a model to productionA churn model through a data contract, a tracked run, a registry gate by slice, hybrid serving, a PSI monitor and one retrain, shipped with a datasheet, a model card, a fairness audit, a design doc and a runbook.45 min
Never trained an AI? Teach a small one to write, right in your browser. It takes about 20 seconds and needs no account. Teach it to write