Index Ai Academy
Phase 11 · Responsible AI

Measuring fairness

Break a model's results down by group: selection rate, true and false positive rates and precision, why they can't all be equal, what each fix costs, and when a group is too small to judge.

About 40 minutes · A free account opens the whole lesson.
Already have an account? Sign in to open this lesson

What it covers7

Why it matters

A model that decides who gets a loan, an interview or a medical referral can work well on average and badly for some of the people it judges. Nobody will tell you: the dashboard shows one accuracy number and it looks fine. Measuring fairness is the engineering part of a larger question. It can't say what is fair, which is for people and the law to decide. It can say, with numbers, who gets which errors and what each fix would cost, and without those numbers that decision gets made blind.

What you'll be able to do

  • Break a model's results down by group, and see what an overall number hides
  • Compute selection rate, true positive rate, false positive rate and precision for each group
  • Name the fairness definition each of those measures belongs to
  • Show why the definitions can't all hold at once, and what each choice costs
  • Tell a real gap from noise in a small group, and report the difference honestly
Quick check
A guess before you start. A loan model is right for 84% of applicants. Can you conclude that it works about equally well for every group of applicants?
  1. Yes: 84% applies to each applicant in the same way
  2. Yes, as long as the model wasn't given the group
  3. No: an average can hide a group that it serves badly
  4. No: accuracy can never be measured for a group
Show the answer

No: an average can hide a group that it serves badly

An overall number is an average weighted by how many people are in each group. If three quarters of applicants are in one group, the model can be much worse for the rest and the average barely moves. The only way to know is to compute the numbers again, group by group.

The idea: one number hides the groups

Every fairness measure is a rate computed per group, the kind of count-and-divide you did in Describing data. The recall check below uses the true positive rate.

One number hides the groups

The data for this lesson is made up, so that the truth is known. Eight thousand people applied for a loan: 6,000 in group A and 2,000 in group B. For each, repay says what really happened, and score is a model's guess from 0 to 1. The lender approves a score of 0.5 or more.

Two things differ between the groups, both by construction. People in A repay more often. And the model's scores are noisier for B, as a model's often are for a group it saw less of in training.

Setup (runs before every cell) · 19 lines

The lesson runs this once, before its first cell, and every cell, drill and example starts with what it defines. A cell that changes a setup object changes it for the cells after it, until the page reloads.

import numpy as np


def make(n_a, n_b, seed, noise_b=0.22):
    """Two groups of applicants: repay is the truth, score is the model's guess."""
    rng = np.random.default_rng(seed)
    group = np.array(["A"] * n_a + ["B"] * n_b)
    repay = rng.random(n_a + n_b) < np.where(group == "A", 0.70, 0.50)
    spread = np.where(group == "A", 0.14, noise_b)     # less sure about group B
    noise = rng.normal(0, 1, n_a + n_b) * spread
    score = np.clip(0.35 + 0.30 * repay + noise, 0, 1)
    return group, repay, score


def rates(group, repay, approve, g):
    mine = group == g
    y, p = repay[mine], approve[mine]
    return {"people": int(mine.sum()), "repay": y.mean(), "approved": p.mean(),
            "tpr": p[y].mean(), "fpr": p[~y].mean(), "precision": y[p].mean()}
group, repay, score = make(6000, 2000, 0)

approve = score >= 0.5
print("accuracy, everyone:", round(float((approve == repay).mean()), 3))
print()
print("group | people | really repay | approved | accuracy")
for g in "AB":
    r = rates(group, repay, approve, g)
    print(
        g.ljust(5), "|", str(r["people"]).rjust(6), "|",
        str(round(float(r["repay"]), 3)).ljust(12), "|",
        str(round(float(r["approved"]), 3)).ljust(8), "|",
        round(float((approve == repay)[group == g].mean()), 3),
    )

The model is right for 84.2% of applicants. Split by group, that is 86.9% for A and 76.0% for B. The overall number sits close to A's, since A is three quarters of the people.

Two other columns differ as well. A is approved 64.8% of the time and B 50.4%. And A really does repay more often: 70.1% against 50.1%. So some of the difference in approvals reflects a difference in the world, and some reflects a model that is worse for B. Telling those apart is most of this lesson.

To measure anything by group you need each person's group in the evaluation data. That is separate from whether the model uses it. A model that isn't given the group can still learn it from a ZIP code or a first name, so leaving it out guarantees nothing, and makes the checking impossible.

Quick check
From Precision, recall and F1: a classifier flags people who will repay. What is its recall?
  1. Of the people it flags, the share who really repay
  2. Of the people who don't repay, the share it flags
  3. Of the people who really repay, the share it flags
  4. Of all the people, the share it gets right
Show the answer

Of the people who really repay, the share it flags

Recall asks about the real positives: how many did the model find? In fairness work the same number is called the true positive rate. Here it means: of the people who would repay, what share were approved. The second option is precision, the third accuracy, and the fourth the false positive rate.

Three meanings of fair

"Fair" has more than one precise meaning, and each compares a different number between groups.

Compare between groupsThe numberThe idea
Demographic paritySelection rate: the share approvedEach group is approved equally often
Equal opportunityTrue positive rate: approved, of those who repayPeople who deserve a yes get one equally often
Equalized oddsTrue positive rate and false positive rateBoth kinds of error are shared equally
Predictive parityPrecision: repay, of those approvedA yes means the same thing in each group

Uses the setup above, 19 lines.

group, repay, score = make(6000, 2000, 0)

approve = score >= 0.5
a, b = rates(group, repay, approve, "A"), rates(group, repay, approve, "B")
names = {
    "approved": "approved (selection rate)", "tpr": "approved, of those who repay",
    "fpr": "approved, of those who don't", "precision": "repay, of those approved",
}
print("measure".ljust(30), "|   A   |   B   | gap A - B")
for key, name in names.items():
    print(
        name.ljust(30), "|", f"{a[key]:.3f}", "|", f"{b[key]:.3f}", "|",
        f"{a[key] - b[key]:+.3f}",
    )
print()
print(
    "approval rate of B as a share of A's:",
    round(float(b["approved"] / a["approved"]), 2),
)

By every one of these measures the two groups are treated differently.

The second row is the one a person would feel first. Of the people in B who would have repaid, 76.4% were approved. In A it was 86.9%. One creditworthy applicant in four from group B was turned away.

The third row is the opposite error. Among people who went on not to repay, the model approved 24.3% in B and 13.0% in A. B gets more wrong approvals as well as more wrong refusals, which is what a noisier score does.

The last line gives a ratio that has a rule of thumb attached. In United States employment guidance, a group selected at less than four fifths of the highest group's rate is treated as a sign of adverse impact. Here B is approved at 0.78 of A's rate, just under that line. It is a screening rule, and not a verdict or a universal law.

Try it now

The audit table from the cell above. Lit: where B is treated worse on errors.

Quick check
Six applicants, three in each group. What does this print: each group's selection rate, then each group's true positive rate?
group   = ["A", "A", "A", "B", "B", "B"]
repay   = [1, 1, 0, 1, 1, 0]
approve = [1, 1, 1, 1, 0, 0]
for g in "AB":
    rows = [(r, a) for gg, r, a in zip(group, repay, approve) if gg == g]
    print(
        g, round(sum(a for r, a in rows) / len(rows), 2),
        round(sum(a for r, a in rows if r) / sum(r for r, a in rows), 2),
    )
  1. A 1.0 1.0 B 0.33 0.33
  2. A 1.0 0.67 B 0.33 0.5
  3. A 1.0 1.0 B 0.33 0.5
  4. A 0.67 1.0 B 0.67 0.5
Show the answer

A 1.0 1.0 B 0.33 0.5

All three in A are approved, so A's selection rate is 1.0, and both of A's repayers are approved: 1.0. In B one of three is approved, 0.33. B has two people who repay and one of them is approved, so its true positive rate is 0.5. The two rates have different denominators: everyone in the group, and only those who repay.

In your own words
Why can a model's overall accuracy be high while it works badly for one group?

Write a sentence or two, then compare it with a good answer in the lesson.

In code: group rates, thresholds and what a sample can show

The data is made up so the truth is known. The helpers make and rates are in the setup, so each cell shows only what it adds.

Drill

Rates for each group

Write group_rates(y_true, y_pred, group). The three lists line up: y_true holds 1 for people who really are positive, y_pred holds 1 for people the model selected. Return a dictionary from each group to a dictionary with "selection" (the share of the group selected), "tpr" (selected, among the group's real positives) and "fpr" (selected, among the group's real negatives). Where a rate has nothing to divide by, use None.

Uses the setup above, 19 lines.

def group_rates(y_true, y_pred, group):
    return {}


print(
    group_rates([1, 1, 0, 1, 1, 0], [1, 1, 1, 1, 0, 0], ["A", "A", "A", "B", "B", "B"])
)
In the lesson, your code is checked as you write it.

You can't have them all

The obvious fix is to give each group its own threshold. The cell tries two versions. One lowers B's threshold until B is approved as often as A. The other lowers it until B's true positive rate matches A's. The last column is the lender's result, counting +1 for a loan repaid and −3 for a default.

Uses the setup above, 19 lines.

group, repay, score = make(6000, 2000, 0)

def threshold_for(scores, wanted, of=None):
    """The threshold, to two decimals, whose approval rate is closest to wanted.
    The rate is taken over everyone, or only over the people in 'of'."""
    best = None
    for t in np.arange(0.20, 0.81, 0.01):
        got = (scores >= t).mean() if of is None else (scores >= t)[of].mean()
        if best is None or abs(got - wanted) < best[0]:
            best = (abs(got - wanted), round(float(t), 2))
    return best[1]

def report(name, t_a, t_b):
    approve = np.where(group == "A", score >= t_a, score >= t_b)
    a, b = rates(group, repay, approve, "A"), rates(group, repay, approve, "B")
    gaps = "  ".join(
        f"{a[k] - b[k]:+.3f}" for k in ("approved", "tpr", "fpr", "precision")
    )
    # +1 for a loan repaid, -3 for a default
    profit = int((approve & repay).sum() - 3 * (approve & ~repay).sum())
    print(name.ljust(26), "|", f"{t_a:.2f} {t_b:.2f}", "|", gaps, "|", profit)

b = group == "B"
a_now = rates(group, repay, score >= 0.5, "A")
print(
    "policy".ljust(26),
    "| thresholds | gaps A - B: approved, tpr, fpr, precision | lender's profit",
)
report("one threshold for all", 0.5, 0.5)
report("equal approval rates", 0.5, threshold_for(score[b], a_now["approved"]))
report(
    "equal true positive rates", 0.5,
    threshold_for(score[b], a_now["tpr"], of=repay[b]),
)

Each policy closes the gap it was aimed at, and opens another.

Equal approval rates brings the first gap to +0.008. To get there, B's threshold drops to 0.40, and the gap in false positive rates goes from −0.113 to −0.270. The extra approvals in B go disproportionately to people who won't repay. A loan someone can't repay harms them too. The lender's profit falls from 2,996 to 2,641.

Equal true positive rates does much the same from a threshold of 0.41.

In both, the precision gap grows, to about +0.25: an approval now means something quite different in the two groups.

This is not a failure of threshold-picking. It was proved in 2016 that when groups have different base rates, and the model isn't perfect, equal precision, equal true positive rates and equal false positive rates can't all hold at once. Every system chooses which to give up, whether or not anyone decided.

So the choice is a policy decision. Which error is worse here, and for whom? That belongs to the people accountable for the system, often with lawyers in the room. The engineer's part is the table above: every option, each gap, and the cost.

Quick check
A hiring screen passes candidates on to interview. The team decides that what matters most is that qualified candidates get an interview equally often, whichever group they are in. Which measure should they equalize?
  1. The overall accuracy
  2. The true positive rate
  3. The selection rate
  4. The precision
Show the answer

The true positive rate

"Qualified candidates get through equally often" is a statement about the real positives in each group, which is the true positive rate. That is equal opportunity. Equal selection rates would pass the same share of each group whether or not they were qualified, and equal precision would make a pass mean the same thing in each group.

Drill

The four-fifths screen

Write impact_ratio(selection). selection maps each group to its selection rate. Return a tuple (ratio, lowest, passes): the lowest rate divided by the highest, the name of the group with the lowest rate (the alphabetically first, if several tie), and True when the ratio is at least 0.8. If the highest rate is 0, nobody was selected: return a ratio of 1.0, and it passes.

Uses the setup above, 19 lines.

def impact_ratio(selection):
    return (1.0, None, True)


print(impact_ratio({"A": 0.648, "B": 0.504}))
In the lesson, your code is checked as you write it.

How sure are you, and what to fix

A gap measured on a test set is an estimate, and for a small group it is a poor one. The first half of the cell draws 200 different samples of applicants and measures the true positive rate gap on each. Only the size of group B changes.

The second half tries the fix that thresholds couldn't deliver: a model that is as good for B as it is for A.

Uses the setup above, 19 lines.

group, repay, score = make(6000, 2000, 0)

def tpr_gap(n_b, seed, noise_b=0.22):
    g, y, s = make(6000, n_b, seed, noise_b)
    approve = s >= 0.5
    return rates(g, y, approve, "A")["tpr"] - rates(g, y, approve, "B")["tpr"]

print("the gap in true positive rates, measured on 200 different samples of applicants")
for n_b in (60, 200, 2000):
    gaps = np.array([tpr_gap(n_b, 100 + seed) for seed in range(200)])
    print(
        "group B has", str(n_b).rjust(4),
        "people | 95% of the measured gaps fall between",
        f"{np.percentile(gaps, 2.5):+.3f}", "and", f"{np.percentile(gaps, 97.5):+.3f}",
    )

print()
print(
    "the same check after the model is as good for B as for A (more of B's data to "
    "learn from):"
)
group, repay, score = make(6000, 2000, 0, noise_b=0.14)
approve = score >= 0.5
a, b = rates(group, repay, approve, "A"), rates(group, repay, approve, "B")
for key in ("approved", "tpr", "fpr", "precision"):
    print(
        key.ljust(9), "| A", f"{a[key]:.3f}", "| B", f"{b[key]:.3f}", "| gap",
        f"{a[key] - b[key]:+.3f}",
    )
print(
    "lender's profit, with one threshold for all:",
    int((approve & repay).sum() - 3 * (approve & ~repay).sum()),
)

With 2,000 people in B, the measured gap lands between +0.081 and +0.135: clearly there. With 60 people, it lands anywhere from −0.020 to +0.237. One sample would show a huge gap, and another would show B doing slightly better. Sixty people contain about thirty who repay, so each of them moves the rate by three points.

This cuts both ways. A small group can't prove a gap, and it can't prove there isn't one. The honest report says "too few to tell", with the count, and asks for more data.

Now the second half. When the model's scores are as reliable for B as for A, the true positive rate gap falls from +0.105 to +0.007, and the false positive rate gap to −0.014, with one threshold for everyone. The lender's profit rises from 2,996 to 3,388. That gap was never about thresholds. It was a worse model for the group with less data, and the fix was data.

Two gaps remain: approvals, at +0.144, and precision. Those come from the base rates, which no model changes. Whether a difference in the world should be carried through into decisions is again a question for people. What you can now tell them is which part of the gap was the model's fault.

Quick check
A group has 20 people who really repay. What does this print: how far one person's result moves that group's true positive rate, and the same for a group with 2,000?
for positives in (20, 2000):
    print(positives, round(100 / positives, 2), "points")
  1. 20 0.2 points 2000 0.0 points
  2. 20 5.0 points 2000 0.05 points
  3. 20 5.0 points 2000 5.0 points
  4. 20 20.0 points 2000 2000.0 points
Show the answer

20 5.0 points 2000 0.05 points

A rate out of 20 people moves in steps of 5 percentage points: one more approval takes it from 75% to 80%. Out of 2,000 the step is a twentieth of a point. A gap of 8 points between a large group and a group with 20 positives is one or two people, which is no evidence of anything.

Quick check
In the cell above, what removed the gap in true positive rates and raised the lender's profit as well?
  1. Removing the group from the test data
  2. A lower threshold for group B
  3. A larger test set for group A
  4. A model that is as accurate for B as for A
Show the answer

A model that is as accurate for B as for A

The gap came from scores that were noisier for B, so that more of B's good applicants fell below the line. Making the model as reliable for B, in practice by collecting more of B's data, fixed the cause. A lower threshold treats the symptom and raises false approvals. Removing the group from the test data only removes the ability to see.

The four rates as grouped bars, from the audit table:

Uses the setup above, 19 lines.

import matplotlib.pyplot as plt

measures = ["approved", "tpr", "fpr", "precision"]
a = [0.648, 0.869, 0.130, 0.940]
b = [0.504, 0.764, 0.243, 0.760]
xs = range(4)
plt.bar([x - 0.2 for x in xs], a, width=0.4, label="group A")
plt.bar([x + 0.2 for x in xs], b, width=0.4, label="group B")
plt.xticks(xs, measures)
plt.ylabel("rate")
plt.legend()
plt.show()
Worked example

Worked example: an audit that knows what it can't say

An audit function prints each group's true and false positive rates beside the largest group's, and gives a verdict on each gap only where there are at least 100 people to measure it on. Here the model has been given a flaw: it marks down new customers in group B. The audit runs three times: by group, by group and customer type, and on a smaller sample.

Uses the setup above, 19 lines.

group, repay, score = make(6000, 2000, 0)

def audit(group, repay, approve, min_count=100, limit=0.05):
    """Per-group rates, with a verdict on each gap where there are enough people."""
    names = sorted(set(group))
    table = {g: rates(group, repay, approve, g) for g in names}
    biggest = max(names, key=lambda g: table[g]["people"])
    for g in names:
        r = table[g]
        counts = {
            "tpr": int(repay[group == g].sum()), "fpr": int((~repay[group == g]).sum()),
        }
        line = [g.ljust(12), str(r["people"]).rjust(5)]
        for key in ("tpr", "fpr"):
            gap = r[key] - table[biggest][key]
            if g == biggest:
                verdict = "reference"
            elif counts[key] < min_count:
                verdict = "too few to say (" + str(counts[key]) + ")"
            else:
                verdict = "GAP " + f"{gap:+.3f}" if abs(
                    gap
                ) > limit else "ok " + f"{gap:+.3f}"
            line.append(key + " " + f"{r[key]:.3f}" + " " + verdict.ljust(22))
        print(" | ".join(line))

rng = np.random.default_rng(5)
# a second attribute: is this a new customer?
new_customer = rng.random(len(group)) < 0.25
# the model marks new B customers down
score2 = np.where((group == "B") & new_customer, np.clip(score - 0.12, 0, 1), score)
approve = score2 >= 0.5

print("by group")
audit(group, repay, approve)
print()
print("by group and customer type")
both = np.array([g + (", new" if n else ", old") for g, n in zip(group, new_customer)])
audit(both, repay, approve)
print()
print("a group of 150, split the same way")
keep = np.concatenate(
    [np.where(group == "A")[0][:2000], np.where(group == "B")[0][:150]]
)
audit(both[keep], repay[keep], approve[keep])
In the lesson you can change this and run it.

By group, B's true positive rate is 14 points below A's, which is already a finding. Splitting by customer type shows where it lives: new customers in B are approved 57.3% of the time when they would repay, a gap of almost 30 points, while new customers in A are no different from old ones. A gap can hide inside a group, and the combinations are where to look. The third table is the same audit on a sample with 150 people from B. The rate for new B customers reads 0.412, which looks even worse, and it rests on 17 people. The function declines to call it, and says how many it had. An audit that printed "GAP" there would be right by luck, and one that printed "ok" for the old customers, on 57 people, would be wrong. Reporting the count beside every rate is what makes the rest of the table believable.

In your own words
Lowering one group's threshold made approval rates equal. Why isn't that simply the end of the problem?

Write a sentence or two, then compare it with a good answer in the lesson.

On the job: Fairlearn, a model card and who decides

Your part is the audit: every option, each gap and the cost. The choice belongs to the people accountable.

Where it comes up in a week

A reviewer asks for the fairness numbers before a launch. You run the audit by group and by one more attribute, mark any group too small to tell, and attach the table to the model card. If the gaps come from a worse model for a smaller group, the fix is data. If they come from base rates, that is a policy question, and you bring the options with their costs.

Written brief · Model card section

The fairness section of a model card

The loan model's audit table is done. Write the section of its model card that a reviewer and a regulator would read: what you measured, what you found, and what you are leaving to the people accountable for the system.

It must

  • Gives at least two numbers from the audit table
  • Says which gaps come from the model and which from base rates
  • Names a group or attribute too small to judge

Go further

  • Includes the four-fifths ratio and says it is a screen, not a verdict

The template

## What was measured
The groups, the evaluation data and the four measures.

## What was found
Two gaps with their numbers, and which part of each is the model's.

## Limits
What the audit cannot show, including any group too small to tell.

## Decisions for the owners
The tradeoff someone must choose, and the cost of each option.
In the lesson, you write over it, 120 to 280 words, and the tutor can mark it.

Goes deeper in

In your own words
You are asked to check a model for fairness before it ships. What do you report, and what do you leave for others to decide?

Write a sentence or two, then compare it with a good answer in the lesson.

Practice what you just read

  • 6 code examples you can edit and run
  • a practice exercise that checks your code
  • a 5-question quiz
  • 12 review cards for spaced repetition

Try a quiz question

A model has the same true positive rate and the same false positive rate in two groups. One group has many more real positives than the other. What must be true?

  1. Its precision differs between the two groups
  2. Its precision is equal in the two groups too
  3. Its selection rate is equal in the two groups
  4. Its accuracy is higher in the smaller group

Answer it at the end of the lesson to see if you're right, and why.

Before this lesson

Keep learning

Where this lesson leads.

575 lessons in thirteen phases. 176 are free without an account, and a free account opens the rest. See every lesson.

Phase 11 · about 34 hours of study

Professional practice

Design, ship and run real systems, work well in a team, and get hired 43 lessons 1 free
  1. System design fundamentals FreeRequirements, rough numbers, the standard parts and where systems break.40 min
  2. Caches, queues and background jobsSpeed reads with caches, and move slow work to queues that retry safely.40 min
  3. Designing AI systemsLatency, cost, routing, fallbacks and guardrails for features built on models.40 min
  4. Containers and deploymentPackage an app as an image, configure it with the environment, and roll out safely.45 min
  5. CI/CD pipelinesTest every change automatically, and ship with gates, flags and rollbacks.40 min
  6. Docker ComposeSeveral containers together: service names, volumes, secrets and start order.40 min
  7. Kubernetes basicsDesired state, Deployments and Services, labels, probes and rolling updates.40 min
  8. Cloud for AI engineersAccess policies and least privilege, storage, compute choices, managed model endpoints, secrets and budgets.30 min
  9. Metrics, dashboards and alert rulesCounters, gauges and histograms, a rate from a counter, a percentile from buckets, a dashboard panel and an alert rule that fires on a sustained condition.40 min
  10. Reliability and incidentsSLOs, error budgets, alerts, on-call and blameless postmortems.40 min
  11. Incident lab: from the page to the postmortemOne incident run from the page to the all-clear: decide the mitigation, work out the impact in failed requests and error budget, and write a blameless postmortem, a status update and a runbook entry.40 min
  12. Measuring an AI feature from its logsTyped views over event logs, metric specs, cost from a price table, and rollups you can rerun.40 min
  13. A/B testing a model changeHash assignment, per-user results from raw logs, sample ratio checks, guardrails and a decision rule.40 min
  14. Columnar files and a local warehouseRows against columns, the encodings that shrink a column, Parquet's row groups and the ranges that let a query skip them, and SQL over files with DuckDB.40 min
  15. Data quality checks and contractsSix properties of a delivery you can test, each as a count of rows that break it, a contract with the producer, and when to block, quarantine or warn.40 min
  16. Transformations: staging, models and testsRaw, staging, facts and marts as SQL models run in dependency order in DuckDB, tests as queries that count the rows that break a rule, an incremental load that is safe to rerun, and the same models as dbt files.45 min
  17. Streams and windowsEvents that never stop arriving: event time against arrival time, tumbling, sliding and session windows, a watermark for late events, and counting each event once.40 min
  18. Labels and annotator agreementLabels as measurements: agreement and Cohen's kappa, the pair of classes annotators confuse, votes and gold questions, and what wrong labels do to a model and a test score.45 min
  19. Experiment tracking and reproducible runsA run record of settings, seed, data and code behind every score, a run log in a dozen lines, and comparing settings across seeds.40 min
  20. Hyperparameter searchWhat a grid costs, random search on a log scale, stopping weak settings early, and why the winner's validation score flatters it.40 min
  21. Versioning data and modelsFiles stored under the hash of their contents, a manifest that names a whole dataset, and lineage from data and code to each model.35 min
  22. A model registry and promotionVersions under one name, an alias for the live one, gates a candidate must pass, group by group, and a rollback that moves a pointer.40 min
  23. Feature pipelines and training-serving skewWhy a model that passes offline fails live: statistics refit at serving, two code paths, and features from the future, with the fix for each.40 min
  24. Batch and online inferencePredict ahead of time or on request: when the inputs exist, latency near a server's limit, batching what waits, cost, and a hybrid with a freshness limit.40 min
  25. Monitoring a model: drift and decayWhy a model worsens with no code change: data drift and concept drift, a measure of how far inputs have moved, thresholds set from noise, and labels that arrive late.45 min
  26. Retraining and pipelinesWhen to retrain and on which weeks, training as steps in dependency order, a cache that reruns only what changed, and guards before and after.45 min
  27. Designing ML systemsOne method for an ML design, goal and metric to retraining, with the sums that size it: requests a second at peak, a year of feature storage, a label delay and a latency budget.45 min
  28. Datasheets: documenting a datasetThe document that travels with a dataset: a profile by column and by group, claims a program can check, splits that don't leak, a scan for personal data, and the license and limits only people can write.45 min
  29. Curation, active learning and synthetic dataThree levers on a training set: removing copies and broken rows, letting a model choose which rows to label, and manufacturing rows, which helps only when their source knows more than you do.45 min
  30. Measuring fairnessBreak a model's results down by group: selection rate, true and false positive rates and precision, why they can't all be equal, what each fix costs, and when a group is too small to judge.40 min
  31. Privacy engineeringWhy removing names isn't anonymity, how to hold less (minimization, retention, keyed pseudonyms), the idea behind differential privacy and its budget, what a model remembers, and a deletion that reaches every store.40 min
  32. A red-team labAttack a simulated assistant methodically: eight kinds of attack run as a suite, success rates beside ordinary requests blocked, fresh attacks to test the fixes, where a defense belongs, and findings kept as release tests.45 min
  33. What regulation asks of an engineerRisk tiers that follow a system's use, the requirements for high-risk systems as artifacts an engineer builds, evidence tied to a model version, a model card a program can check, and legal dates kept as data.35 min
  34. Git for teamsBranches, pull requests, rebasing and how a three-way merge finds conflicts.40 min
  35. Writing for engineersREADMEs, design docs, commit messages and bug reports people can act on.35 min
  36. Scoping and estimating workA vague request cut into tickets with acceptance criteria and a first shippable slice, estimates as ranges, an honest update for a slip, and a ticket linked to its pull request.30 min
  37. Roles, résumé and the searchWhat AI, ML and data engineering postings ask for, résumé bullets that carry a number, a 100-word project summary, a take-home as a portfolio piece and how to read an offer.35 min
  38. Interviews and your portfolioShow your work, and solve interview problems the way professionals do.40 min
  39. Code reviewWhat to look for, how to give and take feedback, and automating the routine checks.35 min
  40. Reading unfamiliar codeFind your way around a big codebase fast, with tools that map it for you.35 min
  41. Working with AI coding toolsGive agents good context, keep tasks small, and check every change they make.35 min
  42. Capstone: ship a production AI serviceAn AI-backed service with validation, caching, rate limits, fallbacks, logging and evals.45 min
  43. Capstone: take a model to productionA churn model through a data contract, a tracked run, a registry gate by slice, hybrid serving, a PSI monitor and one retrain, shipped with a datasheet, a model card, a fairness audit, a design doc and a runbook.45 min

Never trained an AI? Teach a small one to write, right in your browser. It takes about 20 seconds and needs no account. Teach it to write