Index Ai Academy
Phase 11 · Career

Capstone: take a model to production

A churn model through a data contract, a tracked run, a registry gate by slice, hybrid serving, a PSI monitor and one retrain, shipped with a datasheet, a model card, a fairness audit, a design doc and a runbook.

About 45 minutes · A free account opens the whole lesson.
Already have an account? Sign in to open this lesson

What it covers9

Why it matters

Earlier lessons built each part of a production model on its own: a tracked run, a registry, features, serving, monitoring, retraining. This capstone puts them in one line for a churn model, and gives you one thing the parts cannot give you alone: the experience of a damaged model that every simple check passes and one careful gate stops. At the end you hold a working pipeline and the documents a reviewer asks for.

What you'll be able to do

  • Assemble a model's path to production from the parts of the ML in production lessons
  • Gate a candidate model by slice, and explain why a data contract alone cannot catch damaged labels
  • Serve scores from a table when fresh and live otherwise, and watch inputs with PSI
  • Ship a datasheet, a model card and a fairness audit, with a design doc and a runbook
Quick check
A guess before you start. A new churn model scores higher overall than the one in production, and passes every range check on its data. What else must be true before it goes live?
  1. It must be larger than the old model
  2. It must not be worse for any group of subscribers that matters
  3. Nothing: a higher score is enough
  4. It must have been trained on more rows than before
Show the answer

It must not be worse for any group of subscribers that matters

An average can rise while one group falls. The gate scores each slice separately.

The brief: one churn model, taken all the way

A subscription shop wants to know which subscribers will cancel next month, so that its retention team can call them. You will take a churn model through the whole path, in order: a data contract at the edge, a tracked training run, a registry gate that scores by slice, hybrid serving, a PSI monitor, and one retraining decision. It is the path of Experiment tracking, Versioning data and models, A model registry, Batch and online inference, Monitoring a model and Retraining and pipelines, in one place, with the checks of Data quality checks and contracts at the front.

The data is made up so the truth is known. Subscribers have a tenure, a monthly spend and a count of support tickets. New subscribers (under 9 months) churn by a different rule from long-standing ones, and they are the slice that matters most. The setup has the helpers; each cell shows only what it adds.

The picture follows two versions of the model through the system. Step through both.

Try it now

Pick a version and step through the system. The contract catches broken rows; only the gate catches a model that learned the wrong thing.

Quick check
From A model registry: why is a model's score on the fixed evaluation set, by slice, the thing a promotion gate reads?
  1. It is faster than scoring live traffic
  2. The slice scores are always higher
  3. Every candidate is measured against the same yardstick, and a slice cannot hide behind the average
  4. It lets the candidate train on the evaluation rows, so its score is as high as can be
Show the answer

Every candidate is measured against the same yardstick, and a slice cannot hide behind the average

The same fixed set makes scores comparable, and scoring slices separately shows a group that fell.

What done means

The project is done when all of these hold.

  • A batch that breaks the contract is refused before training.
  • Every trained version has a record: parameters, seed, a data hash and a code commit.
  • A candidate is promoted only if its overall score gains and no slice drops by more than 0.02.
  • Serving reads a stored score when it is fresh and runs the model live when it is not.
  • A weekly check compares each input with training and raises an alert past a PSI of 0.2.
  • You can say when to retrain, and the retrain goes through the same gate.
  • The project has a datasheet, a model card, a fairness audit, a design doc and a runbook.
In your own words
In your own words: why does a data contract not stop a model trained on damaged labels, and what does?

Write a sentence or two, then compare it with a good answer in the lesson.

Build it: the pieces, in order

1. The contract. A batch of subscribers is checked for nulls and for values outside the range each column may take. The cell checks a clean batch and one with 30 negative spends and 10 missing tenures.

Setup (runs before every cell) · 120 lines

The lesson runs this once, before its first cell, and every cell, drill and example starts with what it defines. A cell that changes a setup object changes it for the cells after it, until the page reloads.

import hashlib
import json
import math

import numpy as np
from sklearn.linear_model import LogisticRegression

RANGES = {"tenure": (0, 120), "spend": (0, 500), "tickets": (0, 30)}
NAMES = ["tenure", "spend", "tickets", "new", "new_x_tickets"]
BASE, ALL = [0, 1, 2], [0, 1, 2, 3, 4]


def make_data(n, seed, drift=0.0):
    """Subscribers: tenure in months, monthly spend, support tickets, and churn."""
    rng = np.random.default_rng(seed)
    tenure = rng.integers(1, 61, n).astype(float)
    spend = np.clip(rng.normal(60 + 40 * drift, 20, n), 0, 500)
    tickets = rng.poisson(1.5 + 2 * drift, n).astype(float)
    new = tenure < 9
    z = -0.03 * tenure + 0.25 * tickets - 0.01 * spend + 0.2
    z = z + new * (-4.0 + 2.6 * tickets)
    y = (rng.random(n) < 1 / (1 + np.exp(-z))).astype(int)
    return np.column_stack([tenure, spend, tickets, new, new * tickets]), y


def contract_check(X):
    problems = []
    if np.isnan(X).any():
        problems.append(f"nulls: {int(np.isnan(X).any(axis=1).sum())} rows")
    for j, name in enumerate(NAMES[:3]):
        low, high = RANGES[name]
        bad = int(((X[:, j] < low) | (X[:, j] > high)).sum())
        if bad:
            problems.append(f"{name} out of range: {bad} rows")
    return problems


def data_hash(X, y):
    return hashlib.sha256(X.tobytes() + y.tobytes()).hexdigest()[:12]


def train(X, y, cols, seed=0, C=1.0):
    return LogisticRegression(C=C, max_iter=2000, random_state=seed).fit(X[:, cols], y)


def slices(model, cols, X, y):
    new = X[:, 3] == 1
    return {"all": round(float(model.score(X[:, cols], y)), 3),
            "new": round(float(model.score(X[new][:, cols], y[new])), 3),
            "tenured": round(float(model.score(X[~new][:, cols], y[~new])), 3)}


class Registry:
    def __init__(self):
        self.versions, self.aliases, self.history = {}, {}, []

    def register(self, model, record):
        number = len(self.versions) + 1
        self.versions[number] = {"model": model, "record": record}
        return number

    def set_alias(self, alias, version, note):
        self.history.append({"alias": alias, "from": self.aliases.get(alias),
                             "to": version, "note": note})
        self.aliases[alias] = version

    def get(self, ref):
        number = self.aliases[ref] if isinstance(ref, str) else ref
        return self.versions[number]


def gate(candidate, champion, min_gain=0.005, max_drop=0.02):
    reasons = []
    if not candidate["data"] or not candidate["code"]:
        reasons.append("no lineage")
    if candidate["scores"]["all"] - champion["scores"]["all"] < min_gain:
        reasons.append("gain too small")
    for name, score in champion["scores"].items():
        if score - candidate["scores"][name] > max_drop:
            reasons.append(name + " dropped")
    return reasons


def psi(expected, actual, bins=5):
    edges = np.quantile(expected, np.linspace(0, 1, bins + 1))
    edges[0], edges[-1] = -np.inf, np.inf
    e = np.histogram(expected, edges)[0] / len(expected)
    a = np.histogram(actual, edges)[0] / len(actual)
    e, a = np.clip(e, 1e-4, None), np.clip(a, 1e-4, None)
    return float(np.sum((a - e) * np.log(a / e)))


def should_retrain(psi_value, weeks_since, max_psi=0.2, max_weeks=8):
    return psi_value > max_psi or weeks_since >= max_weeks


Xt, yt = make_data(4000, 1)       # what the first champion learns from
Xe, ye = make_data(3000, 9)       # the fixed evaluation set
X2, y2 = make_data(8000, 2)       # a bigger extract, with its labels


def first_registry():
    reg = Registry()
    model = train(Xt, yt, BASE)
    record = {"params": {"C": 1.0}, "seed": 0, "data": data_hash(Xt, yt),
              "code": "a41c9e7", "scores": slices(model, BASE, Xe, ye)}
    reg.register(model, record)
    reg.set_alias("champion", 1, "first champion")
    return reg


def registry_after_gate():
    """The registry once version 3, trained on repaired labels, is champion."""
    reg = first_registry()
    model = train(X2, y2, ALL)
    record = {"data": data_hash(X2, y2), "code": "c07be11",
              "scores": slices(model, ALL, Xe, ye)}
    number = reg.register(model, record)
    reg.set_alias("champion", number, "passed the gates")
    return reg
X, y = make_data(2000, 0)
print(contract_check(X))
broken = X.copy()
broken[:30, 1] = -5
broken[30:40, 0] = np.nan
print(contract_check(broken))

The clean batch has no problems. The broken one reports 10 rows with nulls and 30 with a spend out of range, so it would be refused before training.

Quick check
What does this print: the number of rows with a spend below 0, when 30 of 2,000 rows have spend −5?
import numpy as np
spend = np.array([60.0] * 1970 + [-5.0] * 30)
print(int((spend < 0).sum()))
  1. 2000
  2. 30
  3. −5
  4. 0
Show the answer

30

The contract counts the rows that break the range rule, and 30 rows do.

2. A tracked run and a registered version. The first champion uses the three base columns. Its record holds the settings, the seed, a hash of the data it saw, the code commit and its scores on a fixed evaluation set, by slice.

Uses the setup above, 120 lines.

reg = Registry()
m1 = train(Xt, yt, BASE)
record = {"params": {"C": 1.0}, "seed": 0, "data": data_hash(Xt, yt), "code": "a41c9e7",
          "scores": slices(m1, BASE, Xe, ye)}
v1 = reg.register(m1, record)
reg.set_alias("champion", v1, "first champion")
print(v1, record["data"], record["scores"])

Version 1 is registered with data hash d5f6172ed7a2. On the evaluation set it scores 0.736 overall, 0.823 for new subscribers and 0.724 for tenured ones. Those three numbers are what every later candidate is measured against.

3. The gate. Two candidates arrive from a bigger extract of 8,000 rows. In the first, a join bug has set the label to 0 for most new subscribers. The second is trained on the repaired labels.

Uses the setup above, 120 lines.

reg = first_registry()
bug = (X2[:, 3] == 1) & (np.random.default_rng(3).random(8000) < 0.7)
for name, labels in (("bugged labels", np.where(bug, 0, y2)), ("repaired labels", y2)):
    model = train(X2, labels, ALL)
    cand = {
        "data": data_hash(X2, labels), "code": "c07be11",
        "scores": slices(model, ALL, Xe, ye),
    }
    number = reg.register(model, cand)
    reasons = gate(cand, reg.get("champion")["record"])
    print("version", number, name, cand["scores"], "->", reasons or "promoted")
    if not reasons:
        reg.set_alias("champion", number, "passed the gates")
print(reg.aliases, len(reg.history))

Version 2 is refused with three reasons: its gain is too small, the overall score dropped (0.71 against 0.736) and the new-subscriber slice fell from 0.823 to 0.586. Its record was complete, and its data passed the contract: the damage was in the labels. Version 3 gains overall (0.744), raises the new slice to 0.872 and holds the tenured slice at 0.726, so the alias moves. The history has two lines: the first champion and this promotion.

Quick check
What does this print: the reasons a candidate is refused, when its overall score is 0.710 against a champion at 0.736 and its new slice is 0.586 against 0.823 (min gain 0.005, max drop 0.02)?
champ = {'all': 0.736, 'new': 0.823}
cand = {'all': 0.710, 'new': 0.586}
reasons = []
if cand['all'] - champ['all'] < 0.005:
    reasons.append('gain too small')
for name, score in champ.items():
    if score - cand[name] > 0.02:
        reasons.append(name + ' dropped')
print(reasons)
  1. ['new dropped']
  2. []
  3. ['gain too small', 'all dropped', 'new dropped']
  4. ['gain too small', 'all dropped', 'new']
Show the answer

['gain too small', 'all dropped', 'new dropped']

The overall score fell, so the gain is below the minimum and the 'all' slice dropped by 0.026. The new slice dropped by 0.237.

4. Hybrid serving. Scores for subscribers are stored by a nightly job. A request reads the stored score if it is fresh, no more than 24 hours old, and runs the model live if it is stale or missing. A lookup takes 2 ms and a live score 40 ms.

Uses the setup above, 120 lines.

table = {"c1": {"score": 0.31, "at": 0}, "c2": {"score": 0.82, "at": 20}}


def source(customer, now, max_age=24):
    entry = table.get(customer)
    return "lookup" if entry is not None and now - entry["at"] <= max_age else "live"


sources = [source(c, 30) for c in ("c1", "c2", "c3")]
print(
    sources, round(sum(2 if s == "lookup" else 40 for s in sources) / 3, 1),
    "ms on average",
)
It runs right here in your browser, with nothing to install.

At hour 30, subscriber c1's score is 30 hours old, so it is computed live. c2's is 10 hours old and is looked up. c3 has no stored score, so it is live as well. The average is 27.3 ms: two live calls and one lookup. The freshness limit is a decision for the owners of the product.

5. The monitor. A weekly check compares each input with its training values using the population stability index, PSI, from Monitoring a model: drift and decay. The cell shows four weeks of growing drift.

Uses the setup above, 120 lines.

base = Xt[:, 2]
for drift in (0.0, 0.1, 0.3, 0.6):
    Xd, yd = make_data(3000, 20 + int(drift * 10), drift)
    print(
        drift, "tickets PSI", round(psi(base, Xd[:, 2]), 3), "| spend PSI",
        round(psi(Xt[:, 1], Xd[:, 1]), 3),
    )

With no drift both are 0.001. At drift 0.3 the spend input is at 0.36, over the 0.2 line, while tickets are still at 0.145. At 0.6 both are well over. Different inputs cross the line at different times, so the alert names the input, not just the model.

The same four weeks as a plot:

Uses the setup above, 120 lines.

import matplotlib.pyplot as plt

drifts = [0.0, 0.1, 0.3, 0.6]
tickets = [0.001, 0.025, 0.145, 0.449]
spend = [0.001, 0.038, 0.36, 1.28]
plt.plot(drifts, tickets, marker="o", label="tickets")
plt.plot(drifts, spend, marker="o", label="spend")
plt.axhline(0.2, linestyle=":", color="gray")
plt.xlabel("how far the inputs have drifted")
plt.ylabel("PSI against training")
plt.legend()
plt.show()
Quick check
What does this print: whether a retrain is due at a PSI of 0.449 after 3 weeks, and at a PSI of 0.05 after 9 weeks (alert at 0.2, at most 8 weeks apart)?
def should_retrain(psi_value, weeks_since, max_psi=0.2, max_weeks=8):
    return psi_value > max_psi or weeks_since >= max_weeks


print(should_retrain(0.449, 3), should_retrain(0.05, 9))
  1. False True
  2. False False
  3. True True
  4. True False
Show the answer

True True

The first has drifted past 0.2. The second has not drifted, but the schedule floor of 8 weeks has passed.

6. One retrain. The alert fired, so the pipeline trains a candidate on fresh data from the drifted world and puts it through the same gate, scored on a fresh evaluation set from that world.

Uses the setup above, 120 lines.

reg = registry_after_gate()
Xr, yr = make_data(4000, 5, drift=0.6)
Xde, yde = make_data(3000, 12, drift=0.6)
champion = reg.get("champion")
cand = train(Xr, yr, ALL)
champ_scores = slices(champion["model"], ALL, Xde, yde)
record4 = {
    "data": data_hash(Xr, yr), "code": "c07be11", "scores": slices(cand, ALL, Xde, yde),
}
print("champion on the drifted world:", champ_scores)
print(
    "candidate:", record4["scores"], "->",
    gate(record4, {"scores": champ_scores}) or "promoted",
)

The champion scores 0.761 on the drifted world and the retrained candidate 0.759. The gate refuses the candidate: its gain is too small. The inputs drifted, but the rule that links them to churn did not, so the model in production is still right. Drift in the inputs is a reason to look, not always a reason to replace. The champion stays, and the failed candidate is recorded.

Drill

Lookup or live

Write serve_source(table, customer, now, max_age). table maps a customer to {"score": ..., "at": hour}. Return "lookup" when the customer has a stored score that is at most max_age hours old at now, and "live" otherwise (missing, or older than max_age).

Uses the setup above, 120 lines.

def serve_source(table, customer, now, max_age):
    return "live"


print(serve_source({"c1": {"score": 0.3, "at": 0}}, "c1", 10, 24))
In the lesson, your code is checked as you write it.
Drill

Alert on sustained drift

Write drift_alert(psis, limit, needed). psis is the weekly PSI of one input, in order. Return the index of the week at which the PSI has been above limit for needed weeks in a row, or None if that never happens.

Uses the setup above, 120 lines.

def drift_alert(psis, limit, needed):
    return None


print(drift_alert([0.01, 0.3, 0.4, 0.5], 0.2, 3))
In the lesson, your code is checked as you write it.
In your own words
The retrained candidate was refused with "gain too small". Was the retrain a waste?

Write a sentence or two, then compare it with a good answer in the lesson.

Ship it: the documents a reviewer asks for

A model in production comes with its paperwork. The worked example below computes the facts for a datasheet, a model card and a fairness audit from the run itself, so none of them can disagree with the data. The people still write the rest, as in Datasheets, What regulation asks of an engineer and Measuring fairness.

Worked example

Worked example: three documents from one run

Compute the evaluation set's composition, the model card's claims and the fairness audit by slice, and print them as one document. The fairness audit uses the champion, version 3.

Uses the setup above, 120 lines.

Xe, ye = make_data(3000, 9)
X2, y2 = make_data(8000, 2)
champion = train(X2, y2, ALL)
new = Xe[:, 3] == 1
pred = champion.predict(Xe[:, ALL])
print("# Datasheet: churn evaluation set")
print(
    "rows:", len(ye), "| churn rate:", round(float(ye.mean()), 3), "| new subscribers:",
    round(float(new.mean()), 3),
)
print("data hash:", data_hash(Xe, ye))
print("# Model card: version 3")
print("claims:", slices(champion, ALL, Xe, ye))
print("# Fairness audit: new against tenured")
for name, mask in (("new", new), ("tenured", ~new)):
    p, t = pred[mask], ye[mask]
    print(
        name.ljust(8), int(mask.sum()), "rows | flagged", round(float(p.mean()), 3),
        "| caught", round(float(p[t == 1].mean()), 3), "| false alarms",
        round(float(p[t == 0].mean()), 3),
    )
In the lesson you can change this and run it.

The evaluation set has 3,000 rows, 29.5% of whom churn and 12.2% of whom are new subscribers. The model card's claims, 0.744 overall, 0.872 for new and 0.726 for tenured, come from the same run. The audit shows the finding to write down: the model catches 84.4% of the churn among new subscribers but only 6.3% among tenured ones. It flags only 3.3% of tenured subscribers at all. The model is good where the retention team wants it most, and nearly blind for long-standing customers. That is a limit for the card's "Limits" section, not a surprise to find in production.

Quick check
In the audit, what share of the churning new subscribers does the model catch?
  1. About 43%
  2. About 84%
  3. About 11%
  4. About 6%
Show the answer

About 84%

The model flags 84.4% of new subscribers who go on to churn. 6.3% is the figure for tenured subscribers.

Portfolio brief

Do it for real: add a slice to the gate

The gate checks three slices. The retention team says low-spend subscribers matter most to them. Extend the pipeline so the scores include a low spend slice (spend under 40) and a high spend slice (40 or more), and show the gate refusing a candidate that drops either by more than 0.02.

It must

  • Scores include two new slices computed from the spend column
  • The champion's record holds all five slices
  • A candidate that drops a spend slice by more than 0.02 is refused, with the reason named
  • A candidate that holds every slice is promoted and the history records it
  • The run prints the alias and the history at the end

Go further

  • Report each slice's size, and mark a slice with fewer than 100 rows as too small to judge
  • Plot the slices for the champion and the candidate

Uses the setup above, 120 lines.

reg = Registry()
Xt, yt = make_data(4000, 1)
Xe, ye = make_data(3000, 9)
model = train(Xt, yt, BASE)
record = {
    "data": data_hash(Xt, yt), "code": "a41c9e7", "scores": slices(model, BASE, Xe, ye),
}
reg.register(model, record)
reg.set_alias("champion", 1, "first champion")
print(reg.aliases, record["scores"])

# your code: score a candidate by one more slice (spend under 40, and 40 or more),
# add it to the record's scores, and make the gate refuse a candidate that drops it
In the lesson, you build it here and the tutor can review it.
Written brief · Design doc

The design doc for the churn service

Your team will review the churn service before it goes live. Write the one-page design doc a reviewer would read first: what it is for, the numbers it must meet, how it is built from the parts of this capstone, and what could go wrong.

It must

  • Names a metric and a slice that must not drop
  • Gives at least three sized numbers
  • Lists a risk with the signal that would reveal it

Go further

  • Says what would make you roll back

The template

## Goal and metric
What it is for, with the metric and a guardrail.

## Design
The parts, in order, in five sentences or fewer.

## Numbers
Requests a second at peak, latency budget, cost, label delay.

## Risks
Three things that could go wrong and how you would notice.
In the lesson, you write over it, 160 to 340 words, and the tutor can mark it.
Written brief · Runbook

A runbook for a drift alert

The PSI monitor has raised an alert at 2 a.m. for the spend input. Write the runbook entry for the person on call, who has not seen this service before.

It must

  • Names the metric and its threshold
  • Says how to tell drift from a broken upstream job
  • Names a first action that is safe at night

Go further

  • Says when a retrain is the answer and when it is not

The template

## Symptom
What the alert says.

## Confirm
How to check it is real.

## First step
The first safe action.

## Escalate
Who to call, and when.
In the lesson, you write over it, 80 to 200 words, and the tutor can mark it.
In your own words
The audit shows the model catches 6% of churn among tenured subscribers. What goes in the model card, and what do you tell the retention team?

Write a sentence or two, then compare it with a good answer in the lesson.

Where it comes up in a week

A reviewer opens the design doc, runs the pipeline from the repository and reads the model card against the latest run. If the card's numbers match, the review is short. If you have kept the gate, the history and the runbook, the question "what happens at 2 a.m." has an answer.

Goes deeper in

In your own words
What would you show a reviewer first, and why?

Write a sentence or two, then compare it with a good answer in the lesson.

Practice what you just read

  • 8 code examples you can edit and run
  • a practice exercise that checks your code
  • a 5-question quiz
  • 9 review cards for spaced repetition

Try a quiz question

Version 2's data passed the contract, yet its model was refused. Why?

  1. Its labels were wrong but valid numbers, so only a slice score showed the harm
  2. The contract was broken
  3. The gate is stricter than it should be for this case, so it refuses good models
  4. Version 2 had too many rows

Answer it at the end of the lesson to see if you're right, and why.

Before this lesson

Keep learning

Where this lesson leads.

575 lessons in thirteen phases. 176 are free without an account, and a free account opens the rest. See every lesson.

Phase 11 · about 34 hours of study

Professional practice

Design, ship and run real systems, work well in a team, and get hired 43 lessons 1 free
  1. System design fundamentals FreeRequirements, rough numbers, the standard parts and where systems break.40 min
  2. Caches, queues and background jobsSpeed reads with caches, and move slow work to queues that retry safely.40 min
  3. Designing AI systemsLatency, cost, routing, fallbacks and guardrails for features built on models.40 min
  4. Containers and deploymentPackage an app as an image, configure it with the environment, and roll out safely.45 min
  5. CI/CD pipelinesTest every change automatically, and ship with gates, flags and rollbacks.40 min
  6. Docker ComposeSeveral containers together: service names, volumes, secrets and start order.40 min
  7. Kubernetes basicsDesired state, Deployments and Services, labels, probes and rolling updates.40 min
  8. Cloud for AI engineersAccess policies and least privilege, storage, compute choices, managed model endpoints, secrets and budgets.30 min
  9. Metrics, dashboards and alert rulesCounters, gauges and histograms, a rate from a counter, a percentile from buckets, a dashboard panel and an alert rule that fires on a sustained condition.40 min
  10. Reliability and incidentsSLOs, error budgets, alerts, on-call and blameless postmortems.40 min
  11. Incident lab: from the page to the postmortemOne incident run from the page to the all-clear: decide the mitigation, work out the impact in failed requests and error budget, and write a blameless postmortem, a status update and a runbook entry.40 min
  12. Measuring an AI feature from its logsTyped views over event logs, metric specs, cost from a price table, and rollups you can rerun.40 min
  13. A/B testing a model changeHash assignment, per-user results from raw logs, sample ratio checks, guardrails and a decision rule.40 min
  14. Columnar files and a local warehouseRows against columns, the encodings that shrink a column, Parquet's row groups and the ranges that let a query skip them, and SQL over files with DuckDB.40 min
  15. Data quality checks and contractsSix properties of a delivery you can test, each as a count of rows that break it, a contract with the producer, and when to block, quarantine or warn.40 min
  16. Transformations: staging, models and testsRaw, staging, facts and marts as SQL models run in dependency order in DuckDB, tests as queries that count the rows that break a rule, an incremental load that is safe to rerun, and the same models as dbt files.45 min
  17. Streams and windowsEvents that never stop arriving: event time against arrival time, tumbling, sliding and session windows, a watermark for late events, and counting each event once.40 min
  18. Labels and annotator agreementLabels as measurements: agreement and Cohen's kappa, the pair of classes annotators confuse, votes and gold questions, and what wrong labels do to a model and a test score.45 min
  19. Experiment tracking and reproducible runsA run record of settings, seed, data and code behind every score, a run log in a dozen lines, and comparing settings across seeds.40 min
  20. Hyperparameter searchWhat a grid costs, random search on a log scale, stopping weak settings early, and why the winner's validation score flatters it.40 min
  21. Versioning data and modelsFiles stored under the hash of their contents, a manifest that names a whole dataset, and lineage from data and code to each model.35 min
  22. A model registry and promotionVersions under one name, an alias for the live one, gates a candidate must pass, group by group, and a rollback that moves a pointer.40 min
  23. Feature pipelines and training-serving skewWhy a model that passes offline fails live: statistics refit at serving, two code paths, and features from the future, with the fix for each.40 min
  24. Batch and online inferencePredict ahead of time or on request: when the inputs exist, latency near a server's limit, batching what waits, cost, and a hybrid with a freshness limit.40 min
  25. Monitoring a model: drift and decayWhy a model worsens with no code change: data drift and concept drift, a measure of how far inputs have moved, thresholds set from noise, and labels that arrive late.45 min
  26. Retraining and pipelinesWhen to retrain and on which weeks, training as steps in dependency order, a cache that reruns only what changed, and guards before and after.45 min
  27. Designing ML systemsOne method for an ML design, goal and metric to retraining, with the sums that size it: requests a second at peak, a year of feature storage, a label delay and a latency budget.45 min
  28. Datasheets: documenting a datasetThe document that travels with a dataset: a profile by column and by group, claims a program can check, splits that don't leak, a scan for personal data, and the license and limits only people can write.45 min
  29. Curation, active learning and synthetic dataThree levers on a training set: removing copies and broken rows, letting a model choose which rows to label, and manufacturing rows, which helps only when their source knows more than you do.45 min
  30. Measuring fairnessBreak a model's results down by group: selection rate, true and false positive rates and precision, why they can't all be equal, what each fix costs, and when a group is too small to judge.40 min
  31. Privacy engineeringWhy removing names isn't anonymity, how to hold less (minimization, retention, keyed pseudonyms), the idea behind differential privacy and its budget, what a model remembers, and a deletion that reaches every store.40 min
  32. A red-team labAttack a simulated assistant methodically: eight kinds of attack run as a suite, success rates beside ordinary requests blocked, fresh attacks to test the fixes, where a defense belongs, and findings kept as release tests.45 min
  33. What regulation asks of an engineerRisk tiers that follow a system's use, the requirements for high-risk systems as artifacts an engineer builds, evidence tied to a model version, a model card a program can check, and legal dates kept as data.35 min
  34. Git for teamsBranches, pull requests, rebasing and how a three-way merge finds conflicts.40 min
  35. Writing for engineersREADMEs, design docs, commit messages and bug reports people can act on.35 min
  36. Scoping and estimating workA vague request cut into tickets with acceptance criteria and a first shippable slice, estimates as ranges, an honest update for a slip, and a ticket linked to its pull request.30 min
  37. Roles, résumé and the searchWhat AI, ML and data engineering postings ask for, résumé bullets that carry a number, a 100-word project summary, a take-home as a portfolio piece and how to read an offer.35 min
  38. Interviews and your portfolioShow your work, and solve interview problems the way professionals do.40 min
  39. Code reviewWhat to look for, how to give and take feedback, and automating the routine checks.35 min
  40. Reading unfamiliar codeFind your way around a big codebase fast, with tools that map it for you.35 min
  41. Working with AI coding toolsGive agents good context, keep tasks small, and check every change they make.35 min
  42. Capstone: ship a production AI serviceAn AI-backed service with validation, caching, rate limits, fallbacks, logging and evals.45 min
  43. Capstone: take a model to productionA churn model through a data contract, a tracked run, a registry gate by slice, hybrid serving, a PSI monitor and one retrain, shipped with a datasheet, a model card, a fairness audit, a design doc and a runbook.45 min
Part of a free course 11 lessons

Free MLOps course: running machine learning in production

Take a model from a notebook to a system that stays right: track and reproduce every run, search settings honestly, version data and models, promote through gates, keep training and serving in step, choose batch or online, catch drift, and retrain through a pipeline. Every number is measured by the lesson's own code.

Never trained an AI? Teach a small one to write, right in your browser. It takes about 20 seconds and needs no account. Teach it to write