Index Ai Academy
Phase 11 · Responsible AI

Privacy engineering

Why removing names isn't anonymity, how to hold less (minimization, retention, keyed pseudonyms), the idea behind differential privacy and its budget, what a model remembers, and a deletion that reaches every store.

About 40 minutes · A free account opens the whole lesson.
Already have an account? Sign in to open this lesson

What it covers6

Why it matters

Every AI system runs on data about people: the logs of what they asked, the tickets they wrote, the records a model was trained on. Privacy law says what you owe those people. Privacy engineering is how you actually deliver it, and most of it is unglamorous: fewer columns, shorter retention, identifiers that can't be reversed, and knowing where every copy lives. This lesson measures the main risks with small experiments, so that "we removed the names" never again sounds like enough.

What you'll be able to do

  • Tell direct identifiers from quasi-identifiers, and show how a few ordinary columns single a person out
  • Reduce what a system holds: collect less, keep it for less time, and replace identifiers with keyed pseudonyms
  • Measure how well a table hides people with the size of its smallest group
  • Explain the idea behind differential privacy: noise sized to hide any one person, and a budget that is spent
  • Describe how a trained model can leak who was in its training data, and plan a deletion across every store
Quick check
A guess before you start. A hospital shares patient records for research. It removes names and ID numbers and keeps each patient's ZIP code, date of birth and sex. Are the records anonymous?
  1. No: research data may never be shared at all
  2. Yes: without a name, nobody can be identified
  3. No: those three together single out most people
  4. Yes: ZIP codes and birth dates are public anyway
Show the answer

No: those three together single out most people

Each of the three columns is shared by thousands of people. Together they are nearly a fingerprint: very few people in one ZIP code share a birth date and a sex. Anyone holding another list with the same three columns and names, such as a voter register, can put the names back. That the columns are public is what makes the attack possible.

The idea: data that points at a person

Uniqueness is a counting argument: more combinations of details than people. It uses Counting. The first check recalls why free text needs a scan.

Data that points at a person

It helps to sort the columns of any table into three kinds.

  • A direct identifier names one person by itself: an email address, a phone number, an account id.
  • A quasi-identifier doesn't, and narrows things down: a ZIP code, a birth date, a job title. Several together can be as good as a name.
  • A sensitive attribute is what the person would not want attached to them: a diagnosis, a salary.

Removing the direct identifiers is the obvious first step, and it is the one people stop at. The cell builds a city of 20,000 people and counts how many are the only person with their combination of quasi-identifiers.

import numpy as np

rng = np.random.default_rng(0)
N = 20000                                        # everyone in a small city
zip_code = rng.integers(0, 400, N)               # 400 ZIP codes, in 20 areas of 20
year = rng.integers(1940, 2006, N)               # year of birth
day = rng.integers(0, 365, N)                    # day of the year
sex = rng.integers(0, 2, N)

def groups(*columns):
    """For each person, how many people share their combination of these columns."""
    keys = np.stack(columns, axis=1)
    _, where, counts = np.unique(keys, axis=0, return_inverse=True, return_counts=True)
    return counts[where.ravel()]

print("columns kept".ljust(32), "| people who are the only one | smallest group")
for name, columns in (("ZIP code, birth date, sex", (zip_code, year, day, sex)),
                      ("ZIP code, birth year, sex", (zip_code, year, sex)),
                      ("ZIP area, birth year, sex", (zip_code // 20, year, sex)),
                      ("ZIP area, decade, sex", (zip_code // 20, year // 10, sex))):
    size = groups(*columns)
    print(
        name.ljust(32), "|", f"{100 * (size == 1).mean():5.1f}%".rjust(27), "|",
        int(size.min()),
    )

# A hospital shares 2,000 patients' records with the names removed. The city's public
# register has everyone's
# name, ZIP code, birth date and sex.
patients = rng.choice(N, 2000, replace=False)
alone = groups(zip_code, year, day, sex) == 1
print()
print(
    "patients in the shared table:", len(patients),
    "| matched to exactly one named person in the register:",
    int(alone[patients].sum()),
)
It runs right here in your browser, with nothing to install.

With ZIP code, full birth date and sex, 99.9% of the city is unique. That is why the hospital's table fails: of 2,000 patients, 1,999 match exactly one named person in the public register. Their diagnoses now have names on them.

Blurring helps, slowly. The birth year alone still leaves 68.8% of people unique. Widening the ZIP code to an area gets that near zero, and yet the smallest group is still 1: somebody remains alone. Only at area, decade and sex does every person share their combination with at least 31 others.

That last number has a name. A table is k-anonymous when every combination of quasi-identifiers is shared by at least k people. The last row is 32-anonymous. It is a useful measure with a known weakness: if all 32 people in a group have the same diagnosis, finding the group is finding the diagnosis.

Try it now

Each row keeps fewer details than the one above. k is the size of the smallest group.

Quick check
From Datasheets: support tickets have a free-text field. Why does a dataset built from them need a scan for personal data, even if no column is meant to hold any?
  1. Ticket text is public, so it has to be labeled as such
  2. A scan makes the dataset smaller and faster to load
  3. Free text is always encrypted, so it must be decoded first
  4. People type names, emails and phone numbers into free text
Show the answer

People type names, emails and phone numbers into free text

Personal data arrives uninvited: a customer pastes their email address or phone number into the message. A dataset built from that text carries it along, into training sets and logs. A scan with patterns finds the kinds that have a recognizable shape, and the datasheet records what was found and what was done about it.

In your own words
Why isn't a table anonymous once the names, emails and ID numbers have been removed?

Write a sentence or two, then compare it with a good answer in the lesson.

In code: blurring, keyed hashes, noise and memorization

Each cell runs a small attack and its defense, with the numbers printed, so you can see what a protection buys.

Drill

How well does this table hide people?

Write smallest_group(rows, keys). rows is a list of dictionaries and keys is the list of quasi-identifier columns. Group the rows by their values in those columns, and return the size of the smallest group: the k of k-anonymity. Return 0 for a table with no rows. With no keys at all, every row is in one group.

def smallest_group(rows, keys):
    return 0


rows = [
    {"zip": "10001", "decade": 1980, "diagnosis": "flu"},
    {"zip": "10001", "decade": 1980, "diagnosis": "asthma"},
    {"zip": "10001", "decade": 1990, "diagnosis": "flu"},
]
print(smallest_group(rows, ["zip", "decade"]))
In the lesson, your code is checked as you write it.

Collect less, keep less, blur the rest

The strongest protections are the plainest.

  • Minimization. Don't collect a column unless a feature needs it. Data you don't hold can't leak, be subpoenaed or be misused. Ask of every field: what breaks without it?
  • Retention. Give every store a lifetime, and delete on schedule. A debug log kept forever is a growing copy of everything users typed.
  • Coarser values. Keep the birth decade, the ZIP area, the day and not the second.
  • Pseudonyms. Where rows must still be linked to the same person, replace the identifier with a stand-in.

That last one has a classic mistake. The cell replaces email addresses with a hash, which looks irreversible, and then attacks it.

import hashlib, hmac

# addresses an attacker could list
addresses = ["user" + str(i) + "@example.com" for i in range(5000)]
customers = addresses[:300]

# Attempt 1: replace each address with its SHA-256 hash.
stored = [hashlib.sha256(a.encode()).hexdigest() for a in customers]
print("stored:", stored[0][:24] + "...")
# the attacker hashes every address they can think of
table = {hashlib.sha256(a.encode()).hexdigest(): a for a in addresses}
print(
    "plain hash | addresses recovered:", sum(h in table for h in stored), "of",
    len(stored),
)

# Attempt 2: a keyed hash. The key lives in a secrets manager, away from the data.
KEY = b"kept out of the data store"
def pseudonym(address):
    return "u_" + hmac.new(
        KEY, address.lower().encode(), hashlib.sha256,
    ).hexdigest()[:12]

stored = [pseudonym(a) for a in customers]
guesses = {"u_" + hashlib.sha256(a.encode()).hexdigest()[:12] for a in addresses}
print(
    "keyed hash | addresses recovered:", sum(p in guesses for p in stored), "of",
    len(stored),
)
print(
    "the same person always gets the same pseudonym:", pseudonym("Sam@Example.com"),
    pseudonym("sam@example.com"),
)

A plain hash hid nothing: all 300 addresses were recovered. The attacker didn't reverse the hash. They hashed every address they could list and looked for matches. Email addresses, phone numbers and names are guessable, so hashing them only obscures them from honest readers.

A keyed hash fixes that. Without the key the guesses don't match, and 0 of 300 are recovered. The same person still gets the same pseudonym every time, so counts and joins keep working. The key lives apart from the data, in a secrets manager, and whoever holds both can link pseudonyms back to people.

For that reason pseudonymized data is still personal data, in law and in fact: it is safer to hold, and it still needs protecting. Only data that can't be linked back by anyone counts as anonymous, and the last part showed how hard that is to reach.

Quick check
A table stores each customer's phone number as a SHA-256 hash. There are about ten billion possible phone numbers in the country. What does this print: can an attacker with the table find a customer's number?
import hashlib

stored = hashlib.sha256(b"5550142").hexdigest()
found = [n for n in range(5550000, 5560000)
         if hashlib.sha256(str(n).encode()).hexdigest() == stored]
print(found)
  1. [5550000]
  2. []
  3. [5550142]
  4. An error: a hash can't be compared
Show the answer

[5550142]

The attacker hashes every candidate and compares. Here ten thousand candidates take a moment, and every phone number in a country is within reach of one ordinary computer. A hash is only hard to undo when the input is hard to guess, and phone numbers aren't. A keyed hash, with the key held elsewhere, is what stops this.

Quick check
A sign-up form asks for a date of birth. The product only needs to know that the user is an adult. What is the best change?
  1. Store the date of birth in a separate database
  2. Ask "are you 18 or over?" and store only the answer
  3. Store the date of birth, and hide it in the interface
  4. Store the date of birth as a SHA-256 hash
Show the answer

Ask "are you 18 or over?" and store only the answer

The feature needs one bit, so collect one bit. A date of birth that was never collected can't be leaked, matched against another list, or asked for in a deletion request. A hash of a birth date is trivially reversed, since there are only a few tens of thousands of possible dates, and hiding or moving the column leaves it just as collected.

Answers leak too

Suppose nobody ever sees a row. People can only ask for totals. That still leaks, by subtraction: ask for the count over everyone, then for the count over everyone but one person.

Differential privacy is the idea that answers this. Add random noise to each answer, sized so that the answer would look much the same whether or not any one person were in the data. Then no answer can reveal much about anyone. The amount of noise is set by a number called epsilon: small epsilon, more noise, more privacy.

import numpy as np

rng = np.random.default_rng(1)
# 500 employees, and a private fact about each
has_condition = rng.random(500) < 0.3
# the person an attacker wants to know about
target = int(np.argmax(has_condition))

everyone = int(has_condition.sum())
all_but_one = int(np.delete(has_condition, target).sum())
print(
    "How many have the condition?", everyone, "| How many, leaving out one person?",
    all_but_one, "| so that person:", everyone - all_but_one,
)

print()
print("the same two questions, answered with noise added:")
for epsilon in (0.1, 0.5, 1.0, 5.0):
    # 2,000 tries at the attack
    a = everyone + rng.laplace(0, 1 / epsilon, 2000)
    b = all_but_one + rng.laplace(0, 1 / epsilon, 2000)
    right = ((a - b) > 0.5).mean()
    print(
        "epsilon", str(epsilon).ljust(3), "| attacker right",
        f"{100 * right:5.1f}% of the time", "| the count is off by",
        f"{np.abs(a - everyone).mean():4.1f}", "on average, out of", everyone,
    )

print()
print("asking the same two questions again and again, at epsilon 0.5:")
for times in (1, 10, 100, 1000):
    a = everyone + rng.laplace(0, 2, times)
    b = all_but_one + rng.laplace(0, 2, times)
    print(
        str(times).rjust(4), "times | averaged answers differ by",
        f"{a.mean() - b.mean():+.2f}", "| privacy budget spent:", 2 * times * 0.5,
    )

With exact answers, two questions give the person away: 159 minus 158 is 1.

With noise at epsilon 0.1 the attacker is right 52.3% of the time, barely better than a coin, and the count is off by about 10 out of 159. At epsilon 5 the count is nearly exact, off by 0.2, and the attacker is right 91.5% of the time. Privacy and accuracy are traded against each other, and epsilon is the dial. Notice that a count over more people would tolerate the same noise far better: being off by 10 matters little in 159,000.

The last four lines show why noise alone isn't enough. Ask the same two questions a thousand times and average, and the noise washes out: the difference comes back as +1.03. So every answer spends some privacy, the epsilons add up, and a system must keep a privacy budget and refuse to answer once it is gone. One question at epsilon 0.5 is a strong protection. A thousand of them is none.

Quick check
A dashboard answers each query with noise at epsilon 0.2, and the team has set a total privacy budget of 3.0 for this dataset. What does this print: how many queries can be answered, and how much budget is then left?
budget, each = 3.0, 0.2
answered = 0
while budget >= each - 1e-9:
    budget -= each
    answered += 1
print(answered, round(abs(budget), 2))
  1. 15 0.0
  2. 15 3.0
  3. 150 0.0
  4. 6 0.2
Show the answer

15 0.0

Each answer spends 0.2, and the epsilons add, so 3.0 buys 15 answers and nothing is left. The sixteenth query has to be refused, or answered from a cached earlier answer, which costs nothing more. A budget that is never enforced is the same as having none.

Drill

Noise sized to hide one person

The noise used above comes from the Laplace distribution, and can be made from one uniform random number u between 0 and 1. Write laplace_noise(u, epsilon, sensitivity). Let scale = sensitivity / epsilon. If u is below 0.5, return scale * log(2 * u). Otherwise return -scale * log(2 * (1 - u)). Raise a ValueError if epsilon isn't above 0. (sensitivity is how much one person can change the true answer: 1 for a count.)

import math


def laplace_noise(u, epsilon, sensitivity):
    return 0.0


print(round(laplace_noise(0.25, 0.5, 1), 3), round(laplace_noise(0.75, 0.5, 1), 3))
In the lesson, your code is checked as you write it.

The attacker's success against epsilon, on a log axis:

import matplotlib.pyplot as plt

epsilons = [0.1, 0.5, 1.0, 5.0]
attacker = [52.3, 56.2, 62.7, 91.5]
plt.plot(epsilons, attacker, marker="o")
plt.axhline(50, linestyle=":", color="gray")
plt.xscale("log")
plt.xlabel("epsilon (log scale)")
plt.ylabel("attacker right (%)")
plt.show()

The curve stays near the coin-flip line until epsilon passes 1, and then climbs fast.

In your own words
Why must a system that adds noise to its answers also limit how many questions it will answer?

Write a sentence or two, then compare it with a good answer in the lesson.

Models remember

A trained model is made from its training data, and can give some of it back. One measurable form of this is a membership inference attack: given a person's record, decide whether the model was trained on it. That alone can be the secret, if the training set was "patients with this condition".

The attack needs nothing clever. A model is usually more confident on examples it was trained on. The cell trains the same small model on more and more people, and lets an attacker guess membership from confidence alone.

import numpy as np

def train(X, y, steps=300, rate=0.5):
    w = np.zeros(X.shape[1])
    for step in range(steps):
        w -= rate * X.T @ (1 / (1 + np.exp(-X @ w)) - y) / len(y)
    return w

def loss(X, y, w):
    p = np.clip(1 / (1 + np.exp(-X @ w)), 1e-9, 1 - 1e-9)
    return -(y * np.log(p) + (1 - y) * np.log(1 - p))

rng = np.random.default_rng(2)
truth = rng.normal(0, 1, 100) * (rng.random(100) < 0.2)

def people(n):
    # 100 facts about each person
    X = rng.normal(0, 1, (n, 100))
    return X, (rng.random(n) < 1 / (1 + np.exp(-X @ truth))).astype(float)

# people the model never saw
X_out, y_out = people(1000)
print(
    "trained on | accuracy on them | on new people | attacker says who was in the "
    "training data"
)
for n in (50, 200, 1000, 10000):
    X_in, y_in = people(n)
    w = train(X_in, y_in)
    inside, outside = loss(X_in[:1000], y_in[:1000], w), loss(X_out, y_out, w)
    # the attack: "the model is very sure about this person, so it was probably trained
    # on them"
    cuts = np.quantile(np.concatenate([inside, outside]), np.linspace(0.02, 0.98, 49))
    attack = max(((inside < cut).mean() + (outside >= cut).mean()) / 2 for cut in cuts)
    print(
        str(n).rjust(10), "|", f"{((X_in @ w > 0) == y_in).mean():.2f}".rjust(16), "|",
        f"{((X_out @ w > 0) == y_out).mean():.2f}".rjust(13), "|",
        f"right {100 * attack:.0f}% of the time",
    )

Trained on 50 people, the model is perfect on them and poor on anyone new, and the attacker is right 87% of the time. That gap between training and new data is the leak: the model has memorized individuals. Trained on 10,000 people it does as well on new people as on its own, and the attacker is at 52%, which is guessing. (Picking the best cut-off after the fact is worth a point or two above 50.)

So the defenses are the ones that reduce overfitting: more data, a smaller model, regularization. There is a version of training with differential privacy built in, which adds noise to the gradients. Large language models have the same weakness in another form: text that appears often in training can be recited back.

The practical consequences reach beyond the model.

  • Don't train on what you couldn't publish, unless you have measured what comes back out.
  • Prompts and outputs are personal data when a person typed them. They follow the same rules of minimization and retention as any log.
  • Deletion has to reach every copy. A model can't forget one row. When someone asks to be erased, the honest plan is to delete the rows everywhere and retrain on schedule.
Quick check
A model scores 99% on its training data and 71% on new data. What does that gap suggest about privacy?
  1. Nothing: accuracy and privacy are unrelated measures
  2. It is private, since new data is handled differently
  3. It was trained with differential privacy already
  4. It has memorized examples, so membership can likely be inferred
Show the answer

It has memorized examples, so membership can likely be inferred

A large gap means the model treats the examples it trained on very differently from others: it is far more confident about them. An attacker can use exactly that difference to tell who was in the training set. Closing the gap, with more data or regularization, closes most of the leak.

On the job: deletion across stores, logs and the tools

In a real system the work is finding every copy of a person's data, and keeping new logs from becoming one more copy.

Worked example

Worked example: "please delete my data"

A user asks to be erased. The product keeps things in five stores, some keyed by email address and some by pseudonym, each with its own lifetime. The code finds every row that belongs to the person and prints what has to happen to it.

import hmac, hashlib

KEY = b"kept out of the data store"
def pseudonym(address):
    return "u_" + hmac.new(
        KEY, address.lower().encode(), hashlib.sha256,
    ).hexdigest()[:12]

person = "sam@example.com"
pid = pseudonym(person)
ana_id = pseudonym("ana@example.com")

# Where one product keeps things. Each store: what a row is keyed by, how long it is
# kept, and its rows.
stores = {
    "accounts":     {
        "key": "email", "kept": "until deleted",
        "rows": [{"email": "sam@example.com"}, {"email": "ana@example.com"}],
    },
    "events":       {"key": "pseudonym", "kept": "13 months",
                     "rows": [{"pseudonym": pid}] * 412 + [{"pseudonym": ana_id}] * 77},
    "tickets":      {
        "key": "email", "kept": "3 years",
        "rows": [{"email": "sam@example.com"}] * 3 + [{"email": "ana@example.com"}],
    },
    "training set": {"key": "pseudonym", "kept": "until rebuilt",
                     "rows": [{"pseudonym": pid}] * 3 + [{"pseudonym": ana_id}]},
    "backups":      {
        "key": "email", "kept": "30 days", "rows": [{"email": "sam@example.com"}],
    },
}
ACTIONS = {"accounts": "delete now", "events": "delete now", "tickets": "delete now",
           "training set": "delete, and mark the set as changed",
           "backups": "can't edit: gone when the backup expires"}

found = 0
for name, store in stores.items():
    wanted = person if store["key"] == "email" else pid
    rows = [r for r in store["rows"] if r[store["key"]] == wanted]
    found += len(rows)
    print(
        name.ljust(12), "|", str(len(rows)).rjust(3), "rows | kept",
        store["kept"].ljust(13), "|", ACTIONS[name],
    )
print("rows found:", found, "in", len(stores), "stores")
print(
    "model fine-tuned on the training set | no rows to delete | retrain without them "
    "at the next scheduled build"
)
In the lesson you can change this and run it.

One person, 420 rows, five stores. Three stores can be cleaned at once. Two things are worth noticing. The events and the training set hold no email address, and the rows are still found, since the pseudonym can be recomputed from the address with the key. Pseudonyms made the stores safer to hold without making the person unfindable, and that is the reason pseudonymized data still counts as personal data. Then the two awkward cases. The backup can't be edited, so the answer is its lifetime: thirty days and it is gone, which is only true if backups really do expire. And the model was fine-tuned on three of this person's tickets. There is no row to delete in a model. The honest plan is the one printed: remove the rows from the training set, mark it changed, and retrain at the next scheduled build. None of this works without the inventory in the first place. A team that can't list where personal data lives can't delete it.

Where it comes up in a week

A product manager wants every user message kept "in case we need it". You ask what breaks without it, propose a retention period for the log, redact the obvious identifiers on the way in, and key the pseudonyms with a secret held in a secrets manager. When a user asks to be erased, the deletion job runs against every store, and the training set is marked changed so the next scheduled build leaves them out.

Goes deeper in

In your own words
You are designing the data side of a new AI feature that uses customer messages. What do you decide before the first message is stored?

Write a sentence or two, then compare it with a good answer in the lesson.

Practice what you just read

  • 6 code examples you can edit and run
  • a practice exercise that checks your code
  • a 5-question quiz
  • 13 review cards for spaced repetition

Try a quiz question

Why is a quasi-identifier a privacy risk, when it doesn't identify anyone by itself?

  1. Several together can be unique, and can be matched to a named list
  2. It is always encrypted, which draws attention to it
  3. It can't be removed from a table once collected
  4. It reveals the sensitive attribute directly

Answer it at the end of the lesson to see if you're right, and why.

Before this lesson

Keep learning

Where this lesson leads.

575 lessons in thirteen phases. 176 are free without an account, and a free account opens the rest. See every lesson.

Phase 11 · about 34 hours of study

Professional practice

Design, ship and run real systems, work well in a team, and get hired 43 lessons 1 free
  1. System design fundamentals FreeRequirements, rough numbers, the standard parts and where systems break.40 min
  2. Caches, queues and background jobsSpeed reads with caches, and move slow work to queues that retry safely.40 min
  3. Designing AI systemsLatency, cost, routing, fallbacks and guardrails for features built on models.40 min
  4. Containers and deploymentPackage an app as an image, configure it with the environment, and roll out safely.45 min
  5. CI/CD pipelinesTest every change automatically, and ship with gates, flags and rollbacks.40 min
  6. Docker ComposeSeveral containers together: service names, volumes, secrets and start order.40 min
  7. Kubernetes basicsDesired state, Deployments and Services, labels, probes and rolling updates.40 min
  8. Cloud for AI engineersAccess policies and least privilege, storage, compute choices, managed model endpoints, secrets and budgets.30 min
  9. Metrics, dashboards and alert rulesCounters, gauges and histograms, a rate from a counter, a percentile from buckets, a dashboard panel and an alert rule that fires on a sustained condition.40 min
  10. Reliability and incidentsSLOs, error budgets, alerts, on-call and blameless postmortems.40 min
  11. Incident lab: from the page to the postmortemOne incident run from the page to the all-clear: decide the mitigation, work out the impact in failed requests and error budget, and write a blameless postmortem, a status update and a runbook entry.40 min
  12. Measuring an AI feature from its logsTyped views over event logs, metric specs, cost from a price table, and rollups you can rerun.40 min
  13. A/B testing a model changeHash assignment, per-user results from raw logs, sample ratio checks, guardrails and a decision rule.40 min
  14. Columnar files and a local warehouseRows against columns, the encodings that shrink a column, Parquet's row groups and the ranges that let a query skip them, and SQL over files with DuckDB.40 min
  15. Data quality checks and contractsSix properties of a delivery you can test, each as a count of rows that break it, a contract with the producer, and when to block, quarantine or warn.40 min
  16. Transformations: staging, models and testsRaw, staging, facts and marts as SQL models run in dependency order in DuckDB, tests as queries that count the rows that break a rule, an incremental load that is safe to rerun, and the same models as dbt files.45 min
  17. Streams and windowsEvents that never stop arriving: event time against arrival time, tumbling, sliding and session windows, a watermark for late events, and counting each event once.40 min
  18. Labels and annotator agreementLabels as measurements: agreement and Cohen's kappa, the pair of classes annotators confuse, votes and gold questions, and what wrong labels do to a model and a test score.45 min
  19. Experiment tracking and reproducible runsA run record of settings, seed, data and code behind every score, a run log in a dozen lines, and comparing settings across seeds.40 min
  20. Hyperparameter searchWhat a grid costs, random search on a log scale, stopping weak settings early, and why the winner's validation score flatters it.40 min
  21. Versioning data and modelsFiles stored under the hash of their contents, a manifest that names a whole dataset, and lineage from data and code to each model.35 min
  22. A model registry and promotionVersions under one name, an alias for the live one, gates a candidate must pass, group by group, and a rollback that moves a pointer.40 min
  23. Feature pipelines and training-serving skewWhy a model that passes offline fails live: statistics refit at serving, two code paths, and features from the future, with the fix for each.40 min
  24. Batch and online inferencePredict ahead of time or on request: when the inputs exist, latency near a server's limit, batching what waits, cost, and a hybrid with a freshness limit.40 min
  25. Monitoring a model: drift and decayWhy a model worsens with no code change: data drift and concept drift, a measure of how far inputs have moved, thresholds set from noise, and labels that arrive late.45 min
  26. Retraining and pipelinesWhen to retrain and on which weeks, training as steps in dependency order, a cache that reruns only what changed, and guards before and after.45 min
  27. Designing ML systemsOne method for an ML design, goal and metric to retraining, with the sums that size it: requests a second at peak, a year of feature storage, a label delay and a latency budget.45 min
  28. Datasheets: documenting a datasetThe document that travels with a dataset: a profile by column and by group, claims a program can check, splits that don't leak, a scan for personal data, and the license and limits only people can write.45 min
  29. Curation, active learning and synthetic dataThree levers on a training set: removing copies and broken rows, letting a model choose which rows to label, and manufacturing rows, which helps only when their source knows more than you do.45 min
  30. Measuring fairnessBreak a model's results down by group: selection rate, true and false positive rates and precision, why they can't all be equal, what each fix costs, and when a group is too small to judge.40 min
  31. Privacy engineeringWhy removing names isn't anonymity, how to hold less (minimization, retention, keyed pseudonyms), the idea behind differential privacy and its budget, what a model remembers, and a deletion that reaches every store.40 min
  32. A red-team labAttack a simulated assistant methodically: eight kinds of attack run as a suite, success rates beside ordinary requests blocked, fresh attacks to test the fixes, where a defense belongs, and findings kept as release tests.45 min
  33. What regulation asks of an engineerRisk tiers that follow a system's use, the requirements for high-risk systems as artifacts an engineer builds, evidence tied to a model version, a model card a program can check, and legal dates kept as data.35 min
  34. Git for teamsBranches, pull requests, rebasing and how a three-way merge finds conflicts.40 min
  35. Writing for engineersREADMEs, design docs, commit messages and bug reports people can act on.35 min
  36. Scoping and estimating workA vague request cut into tickets with acceptance criteria and a first shippable slice, estimates as ranges, an honest update for a slip, and a ticket linked to its pull request.30 min
  37. Roles, résumé and the searchWhat AI, ML and data engineering postings ask for, résumé bullets that carry a number, a 100-word project summary, a take-home as a portfolio piece and how to read an offer.35 min
  38. Interviews and your portfolioShow your work, and solve interview problems the way professionals do.40 min
  39. Code reviewWhat to look for, how to give and take feedback, and automating the routine checks.35 min
  40. Reading unfamiliar codeFind your way around a big codebase fast, with tools that map it for you.35 min
  41. Working with AI coding toolsGive agents good context, keep tasks small, and check every change they make.35 min
  42. Capstone: ship a production AI serviceAn AI-backed service with validation, caching, rate limits, fallbacks, logging and evals.45 min
  43. Capstone: take a model to productionA churn model through a data contract, a tracked run, a registry gate by slice, hybrid serving, a PSI monitor and one retrain, shipped with a datasheet, a model card, a fairness audit, a design doc and a runbook.45 min

Never trained an AI? Teach a small one to write, right in your browser. It takes about 20 seconds and needs no account. Teach it to write