Index Ai Academy
Phase 11 · ML in production

Monitoring a model: drift and decay

Why a model worsens with no code change: data drift and concept drift, a measure of how far inputs have moved, thresholds set from noise, and labels that arrive late.

About 45 minutes · A free account opens the whole lesson.
Already have an account? Sign in to open this lesson

What it covers7

Why it matters

Ordinary software keeps working until someone changes it. A model can get worse while every file stays the same, because it was fitted to a world that has since moved: customers change, prices change, fraudsters adapt. Nothing crashes and no test fails. The predictions just become less right, a little each week, until a person notices the business numbers. Monitoring is how you notice first. This lesson measures three kinds of change on a real model, including one that the usual monitor is blind to and one that looks alarming and does no harm.

What you'll be able to do

  • Tell data drift from concept drift, and say which of them a monitor on the inputs can and can't see
  • Measure how far a feature has moved from its training distribution with the population stability index
  • Set a drift threshold from the noise in your own window size, not from a rule of thumb alone
  • Monitor accuracy when labels arrive late, and say how far behind that monitor runs
Quick check
A guess before you start. A loan model was trained last year. Since then, the incomes and debts of applicants look exactly as they did in training: every chart of the inputs is unchanged. Can the model's accuracy have fallen?
  1. No: a model fed the same kind of inputs gives the same quality of answers
  2. Yes, but only if the server it runs on has become slower
  3. Yes: what the inputs mean for repayment can change while the inputs stay put
  4. No: accuracy can only fall when the code or the weights are changed
Show the answer

Yes: what the inputs mean for repayment can change while the inputs stay put

A model learns a relationship between inputs and outcomes. If the relationship shifts, say a downturn makes the same income a weaker sign of repayment, the same inputs deserve different answers. This lesson simulates exactly that: the input charts stay flat and accuracy falls from 0.79 to 0.54.

The idea: the inputs move, or the rule does

This lesson rests on Surprise, entropy and cross-entropy (PSI is a distance between two distributions, built from the same log ratio) and Samples, the law of large numbers and the central limit theorem (a small window is noisy even when nothing changed). A model is trained once; the world keeps moving.

Two ways the world moves

A model is trained on rows that pair inputs with outcomes. In production, two different things can change.

Data drift is a change in the inputs: who is applying, what they look like, how the values are spread. The rule that connects inputs to outcomes is the same, but the model is now being asked about cases unlike the ones it learned from.

Concept drift is a change in the rule itself. The same applicant, with the same income and debt, now has a different chance of repaying.

The model in this lesson decides loan applications from two inputs, income and debt, in thousands. It is trained once and never touched again. loans makes a week of applications and can move the world in three ways.

Setup (runs before every cell) · 26 lines

The lesson runs this once, before its first cell, and every cell, drill and example starts with what it defines. A cell that changes a setup object changes it for the cells after it, until the page reloads.

import numpy as np
from sklearn.linear_model import LogisticRegression

def loans(n, seed, income_up=0.0, debt_down=0.0, rule_change=0.0):
    """A batch of applications: income and debt in thousands,
    and whether each loan was repaid."""
    rng = np.random.default_rng(seed)
    income = rng.normal(50 + income_up, 12, size=n)
    debt = rng.normal(20 - debt_down, 6, size=n)
    earn = income - 50
    signal = (0.08 * earn - 0.006 * np.maximum(earn - 10, 0) ** 2
              - 0.15 * (debt - 20) - rule_change * 0.16 * earn)
    return (np.column_stack([income, debt]),
            (signal + rng.normal(0, 1, n) > 0).astype(int))

def shares(values, edges):
    counts = np.bincount(np.searchsorted(edges, values, side="right"),
                         minlength=len(edges) + 1)
    return counts / counts.sum()

def psi(expected, actual):
    e, a = np.maximum(expected, 1e-4), np.maximum(actual, 1e-4)
    return float(np.sum((a - e) * np.log(a / e)))

X_train, y_train = loans(4000, 0)
model = LogisticRegression().fit(X_train, y_train)
X_train, y_train = loans(4000, 0)
model = LogisticRegression().fit(X_train, y_train)
X_test, y_test = loans(2000, 1)
print("at launch                | accuracy",
      round(float(model.score(X_test, y_test)), 3))

for name, change in (("incomes rise by 12      ", {"income_up": 12}),
                     ("debts fall by 6         ", {"debt_down": 6}),
                     ("the rule changes        ", {"rule_change": 1.0})):
    X, y = loans(1000, 7, **change)
    print(name, "| accuracy", round(float(model.score(X, y)), 3))

Three changes, three different results.

  • Incomes rise and accuracy slips from 0.789 to 0.746. In this world, repayment stops improving at high incomes. The training data held few such applicants, so the model never learned that, and it approves them too readily. This is data drift that hurts.
  • Debts fall and accuracy goes up, to 0.808. The inputs moved just as far, but into territory where what the model learned still holds, and low-debt cases are easy ones. This is data drift that does no harm.
  • The rule changes, so income no longer predicts repayment the way it did, and accuracy collapses to 0.544, barely better than a coin. The inputs didn't move at all. This is concept drift.

Hold on to the second line. A change in the inputs is a reason to look, not proof of damage.

Quick check
A demand-forecasting model was trained before a new public holiday was created. On that day, shops see twice the usual sales for the same weather, weekday and prices. What is this?
  1. Concept drift: the same inputs now lead to a different outcome
  2. A broken pipeline: a feature is arriving with the wrong values
  3. Nothing: one unusual day can't be told apart from noise
  4. Data drift: the inputs have moved away from the training data
Show the answer

Concept drift: the same inputs now lead to a different outcome

The weather, weekday and prices are ones the model has seen many times. What changed is what they lead to: on this date, twice the sales. The relationship between inputs and outcome moved, which is concept drift. The remedy is a new input, a holiday flag, and retraining.

Measuring how far a feature has moved

To monitor the inputs you need one number per feature that says how different this week's values are from training. The most common is the population stability index, or PSI. It works on shares:

  1. Cut the feature's training values into ten bins holding a tenth of the rows each. The bin edges are fixed from then on.
  2. For a new window of data, find the share of rows that falls in each of those same bins.
  3. Compare the two lists of shares.
PSI=∑i(ai−ei)×ln⁡aiei\text{PSI} = \sum_i (a_i - e_i) \times \ln\frac{a_i}{e_i}

In plain words: for each bin, eie_i is the share of training rows in it (expected) and aia_i is the share of this window's rows (actual). Take the difference of the shares, multiply by the natural log of their ratio, and add up the bins. A bin whose share didn't change adds nothing. A bin that gained or lost rows adds a positive amount either way.

If you remember KL divergence from the cross-entropy lesson, this is the same idea applied in both directions and added together: a distance between two distributions that is 0 when they match.

Uses the setup above, 26 lines.

import numpy as np

def psi(expected, actual):
    # a tiny floor, so an empty bin can't divide by zero
    e, a = np.maximum(expected, 1e-4), np.maximum(actual, 1e-4)
    return float(np.sum((a - e) * np.log(a / e)))

training = np.array([0.25, 0.25, 0.25, 0.25])
print("no change:   ", round(psi(training, np.array([0.25, 0.25, 0.25, 0.25])), 3))
print("small change:", round(psi(training, np.array([0.22, 0.25, 0.25, 0.28])), 3))
large = np.array([0.05, 0.15, 0.30, 0.50])
print("large change:", round(psi(training, large), 3))
print("each bin adds:", np.round((large - training) * np.log(large / training), 3))
It runs right here in your browser, with nothing to install.

Now watch it on the loan model while incomes creep up by 1.5 thousand a week:

Uses the setup above, 26 lines.

# nine edges make ten bins of training income
edges = np.quantile(X_train[:, 0], np.linspace(0.1, 0.9, 9))
base = shares(X_train[:, 0], edges)

for week in range(1, 9):
    X, y = loans(1000, 100 + week, income_up=1.5 * week)
    print("week", week, "| PSI of income", round(psi(base, shares(X[:, 0], edges)), 2),
          "| share approved", round(float(model.predict(X).mean()), 2),
          "| accuracy", round(float(model.score(X, y)), 3))

The PSI climbs every week, from 0.02 to 0.95. The share of applications the model approves climbs with it, from 0.48 to 0.73, and accuracy drifts down from 0.789 to 0.731.

Look at which columns you could actually have in production. The first two need no labels: they are computed from the requests and the model's own answers, the same day. The third needs to know who repaid, which nobody will know for months. That is why drift monitors exist: they are the part of the picture you can see today.

Try it now

Step through three frames: the expected shares, the actual shares, and what each bin adds to the PSI.

Quick check
What does this print?
import numpy as np

def psi(expected, actual):
    e, a = np.maximum(expected, 1e-4), np.maximum(actual, 1e-4)
    return float(np.sum((a - e) * np.log(a / e)))

train = np.array([0.5, 0.5])
print(round(psi(train, np.array([0.5, 0.5])), 3),
      round(psi(train, np.array([0.8, 0.2])), 3),
      round(psi(train, np.array([0.2, 0.8])), 3))
  1. 0.0 0.416 0.416
  2. 0.0 0.416 -0.416
  3. 1.0 0.416 0.416
  4. 0.0 0.3 0.3
Show the answer

0.0 0.416 0.416

Identical shares give 0. Moving 0.3 of the rows from one bin to the other gives 0.3 × ln(1.6) + 0.3 × ln(2.5) ≈ 0.416, and it gives the same value whichever bin gained: every bin's term is positive, so PSI measures how far the shares moved, not in which direction.

In your own words
What is the difference between data drift and concept drift, and what does a PSI of 0 mean?

Write a sentence or two, then compare it with a good answer in the lesson.

In code: PSI by week, noise, patience and late labels

The monitor is a few lines on top of the model you already have. The cells use the same loans world, set up once above them, and they show what the input monitor sees, what it misses and how late the truth arrives.

Drill

The population stability index

Write psi(expected, actual) for two lists of shares of equal length. For each bin add (a - e) * math.log(a / e). Before using a share, raise it to at least 0.0001, so an empty bin can't cause a division by zero or a log of zero. Return the total rounded to 4 places.

Uses the setup above, 26 lines.

import math


def psi(expected, actual):
    return 0.0


print(psi([0.25, 0.25, 0.25, 0.25], [0.05, 0.15, 0.30, 0.50]))
In the lesson, your code is checked as you write it.

What an input monitor can't see

Run the same monitor on all three changes from the start of the lesson, with a PSI for each feature:

Uses the setup above, 26 lines.

edges = [np.quantile(X_train[:, k], np.linspace(0.1, 0.9, 9)) for k in (0, 1)]
base = [shares(X_train[:, k], edges[k]) for k in (0, 1)]

for name, change in (("incomes rise by 12", {"income_up": 12}),
                     ("debts fall by 6   ", {"debt_down": 6}),
                     ("the rule changes  ", {"rule_change": 1.0})):
    X, y = loans(1000, 7, **change)
    print(name, "| PSI income", round(psi(base[0], shares(X[:, 0], edges[0])), 2),
          "| PSI debt", round(psi(base[1], shares(X[:, 1], edges[1])), 2),
          "| accuracy", round(float(model.score(X, y)), 3))

Read it as a table of what the alarm said against what was true.

ChangeThe input monitor saysWhat happened to accuracy
Incomes riseIncome has moved a long way: 0.92Down, to 0.746
Debts fallDebt has moved a long way: 0.94Up, to 0.808
The rule changesNothing has moved: 0.02 and 0.01Down to 0.544

The first row is the monitor working. The second is a false alarm: the same size of PSI, and the model is fine. The third is the dangerous one. The worst failure of the three produced no signal at all, because concept drift leaves the inputs alone.

So an input monitor is a smoke detector. When it sounds, someone should look. When it's quiet, that proves only that the inputs haven't moved. The only thing that can see concept drift is the outcome: comparing what the model predicted with what really happened. And outcomes are usually late.

Quick check
From the feature pipelines lesson: to compare this week's feature values with the training values, which values should the monitor read?
  1. The training table itself, sampled again with a new random seed
  2. The features the service logged as it actually fed them to the model
  3. The raw request fields, before any feature code has run on them
  4. The features recomputed later from the warehouse by the training code
Show the answer

The features the service logged as it actually fed them to the model

The logged features are the only record of what the model really saw. Recomputing them with the training code would hide any difference between the two paths, which is one of the things the monitor is there to catch.

When is a PSI big enough to act on?

A widely repeated rule of thumb says a PSI under 0.1 means no real change, 0.1 to 0.25 a moderate one, and over 0.25 a large one. It comes from credit scoring, where windows hold many thousands of applicants. Before borrowing it, measure what PSI you get when nothing has changed: draw fresh windows from the training distribution itself, many times, at your own window size.

Uses the setup above, 26 lines.

import numpy as np

rng = np.random.default_rng(0)
train = rng.normal(50, 12, size=4000)
edges = np.quantile(train, np.linspace(0.1, 0.9, 9))

def shares(values):
    counts = np.bincount(np.searchsorted(edges, values, side="right"), minlength=10)
    return counts / counts.sum()

def psi(expected, actual):
    e, a = np.maximum(expected, 1e-4), np.maximum(actual, 1e-4)
    return float(np.sum((a - e) * np.log(a / e)))

base = shares(train)
for n in (100, 300, 1000, 3000):
    values = [psi(base, shares(rng.normal(50, 12, size=n))) for trial in range(300)]
    print(n, "rows in a window | typical PSI", round(float(np.median(values)), 3),
          "| 1 window in 20 is above", round(float(np.quantile(values, 0.95)), 3))

With a hundred rows in a window, the typical PSI is 0.092 when the distribution hasn't changed at all, and one window in twenty is above 0.175. A threshold of 0.1 would fire on nearly half of those windows. With three thousand rows the typical value is 0.006, and 0.1 is a strong signal.

The rule that follows: set the threshold above your own noise, from a run like this one, and prefer larger windows to smaller ones. A second defense is patience: alert only when the value has stayed above the threshold for several windows in a row, since noise seldom repeats and real drift usually persists.

Quick check
A monitor alerts when a daily value has been above the threshold for 3 days in a row. What does this print?
values = [0.05, 0.30, 0.08, 0.22, 0.26, 0.31, 0.29]
threshold, patience = 0.2, 3

run, fired = 0, []
for day, value in enumerate(values):
    run = run + 1 if value > threshold else 0
    if run == patience:
        fired.append(day)
print(fired)
  1. [3]
  2. [5]
  3. [5, 6]
  4. [1, 3, 4, 5, 6]
Show the answer

[5]

Day 1 is above the threshold but day 2 falls back, so the run resets. Days 3, 4 and 5 are all above it, and the third of them, day 5, fires the alert. Day 6 extends the run to 4, which is not equal to 3, so the same episode doesn't fire twice.

Drill

Alerts that wait for the signal to persist

Write alerts(values, threshold, patience). Return the list of indexes at which an alert fires. An alert fires at the moment a value has been above the threshold for patience windows in a row. It doesn't fire again while the values stay above. Once a value drops to the threshold or below, the count starts over, and a later run can fire a new alert.

Uses the setup above, 26 lines.

def alerts(values, threshold, patience):
    return [i for i, v in enumerate(values) if v > threshold]


print(alerts([0.05, 0.30, 0.08, 0.22, 0.26, 0.31, 0.29], 0.2, 3))
In the lesson, your code is checked as you write it.
Worked example

Plot PSI, approvals and accuracy by week

The weekly cell above, as three lines instead of eight rows. Incomes creep up by 1.5 a week.

Uses the setup above, 26 lines.

import matplotlib.pyplot as plt

edges = np.quantile(X_train[:, 0], np.linspace(0.1, 0.9, 9))
base = shares(X_train[:, 0], edges)
weeks = list(range(1, 9))
values = {"PSI of income": [], "share approved": [], "accuracy": []}
for week in weeks:
    X, y = loans(1000, 100 + week, income_up=1.5 * week)
    values["PSI of income"].append(psi(base, shares(X[:, 0], edges)))
    values["share approved"].append(float(model.predict(X).mean()))
    values["accuracy"].append(float(model.score(X, y)))
print({k: round(v[-1], 2) for k, v in values.items()})

for name, series in values.items():
    plt.plot(weeks, series, marker="o", label=name)
plt.xlabel("week")
plt.legend()
plt.show()
In the lesson you can change this and run it.

By week 8 the PSI is 0.95, the model approves 0.73 of applications and accuracy has slid to 0.73. The first line rises quickly and visibly, and the accuracy line moves less: the input monitor shouts early, but accuracy is the line that says whether it matters.

Labels arrive late

The honest measure of a model is its accuracy on what really happened. In a notebook the labels are in the next column. In production they come from the future. A loan's outcome is known months after the decision. A churn label takes thirty days by definition. So an accuracy monitor always describes the past, and the question is how far back.

Here the rule starts changing on day 20, and each loan's outcome is known 30 days after the decision.

Uses the setup above, 26 lines.

# one entry per day: (day, predictions that were right, predictions made)
log = []
for day in range(80):
    X, y = loans(300, 1000 + day, rule_change=min(1.0, max(0, day - 20) / 30))
    log.append((day, int((model.predict(X) == y).sum()), len(y)))

def accuracy(days):
    right = sum(r for d, r, n in log if d in days)
    total = sum(n for d, r, n in log if d in days)
    return round(right / total, 3)

delay = 30
for today in (35, 45, 55, 65, 75):
    newest = today - delay                 # the latest day whose outcomes are known
    print("day", today, "| newest known outcome: day", newest,
          "| the monitor shows", accuracy(range(newest - 9, newest + 1)),
          "| the truth today", accuracy(range(today - 9, today + 1)))

On day 55 the monitor reads 0.774 and looks healthy. The model is in fact at 0.505, no better than a coin, and has been getting worse for five weeks. The monitor isn't wrong. It is reporting on days 16 to 25, the newest it can know about, and it will be day 85 before it shows what is true today.

Because the inputs never moved, the input monitor is silent through all of it. This is the hardest case in monitoring, and there are only partial answers:

  • Find an earlier signal of the outcome. A missed first payment arrives long before a default. A customer going quiet arrives before a cancellation. It is a noisier label, but it is weeks sooner.
  • Label a small sample by hand, every week, where a person can judge the answer.
  • Watch the business numbers the model drives, which react before the labels do.
  • Know your blind interval. If outcomes take 30 days, say so on the dashboard: "accuracy as of 30 days ago". A stale number presented as current is how the day-55 reading fools a team.
Quick check
A churn model's labels are known 30 days after each prediction. Today is day 100. The accuracy chart's newest point is 0.91. What does that point describe?
  1. The model's accuracy on the predictions it made today
  2. Predictions made on day 100, checked against day 70's labels
  3. Predictions made on or before day 70, and nothing more recent
  4. The average accuracy over all 100 days so far
Show the answer

Predictions made on or before day 70, and nothing more recent

A prediction can be scored only once its label exists, 30 days later. On day 100 the newest predictions with labels are from day 70. Anything the model has done in the last 30 days is not in the chart yet.

Quick check
The input monitor for a fraud model has been quiet for a month. Which conclusion is justified?
  1. The inputs look like the training data; the model may still be wrong
  2. Monitoring of outcomes can be switched off to save cost
  3. The model is as accurate as it was when it was launched
  4. There has been no fraud of any new kind this month
Show the answer

The inputs look like the training data; the model may still be wrong

A quiet input monitor says the inputs haven't moved. Concept drift, such as fraudsters changing tactics inside ordinary-looking payments, leaves the inputs alone. Only outcomes can show it.

Worked example

Worked example: a weekly monitoring report

Put the pieces in one report: a PSI for each feature against a threshold set from noise, the share of approvals, and accuracy for the newest week whose outcomes are known, labeled with its age.

Uses the setup above, 26 lines.

import numpy as np
from sklearn.linear_model import LogisticRegression

def loans(n, seed, income_up=0.0, debt_down=0.0, rule_change=0.0):
    rng = np.random.default_rng(seed)
    income = rng.normal(50 + income_up, 12, size=n)
    debt = rng.normal(20 - debt_down, 6, size=n)
    earn = income - 50
    signal = (0.08 * earn - 0.006 * np.maximum(earn - 10, 0) ** 2
              - 0.15 * (debt - 20) - rule_change * 0.16 * earn)
    return (np.column_stack([income, debt]),
            (signal + rng.normal(0, 1, n) > 0).astype(int))

def shares(values, edges):
    counts = np.bincount(np.searchsorted(edges, values, side="right"),
                         minlength=len(edges) + 1)
    return counts / counts.sum()

def psi(expected, actual):
    e, a = np.maximum(expected, 1e-4), np.maximum(actual, 1e-4)
    return float(np.sum((a - e) * np.log(a / e)))

X_train, y_train = loans(4000, 0)
model = LogisticRegression().fit(X_train, y_train)
edges = [np.quantile(X_train[:, k], np.linspace(0.1, 0.9, 9)) for k in (0, 1)]
base = [shares(X_train[:, k], edges[k]) for k in (0, 1)]
threshold = 0.05                        # windows of 1,000 rows: noise stays near 0.02

weeks = [loans(1000, 300 + w, income_up=2.0 * max(0, w - 3)) for w in range(1, 9)]
label_delay = 4                         # outcomes are known four weeks later
for w, (X, y) in enumerate(weeks, start=1):
    drift = [round(psi(base[k], shares(X[:, k], edges[k])), 2) for k in (0, 1)]
    flags = [name for name, value in zip(("income", "debt"), drift)
             if value > threshold]
    known = w - label_delay
    past = ("accuracy week %d: %.3f" % (known, model.score(*weeks[known - 1]))
            if known >= 1 else "no outcomes yet")
    print("week", w, "| PSI", drift,
          "| approved", round(float(model.predict(X).mean()), 2),
          "|", past, "|", "LOOK AT " + ", ".join(flags) if flags else "quiet")
In the lesson you can change this and run it.

Incomes start rising in week 4. The income PSI crosses the threshold in week 5 and the approval share starts to climb the same week, both visible at once with no labels. The accuracy column can't help yet: it first speaks in week 5, about week 1, and by week 8 it has only reached week 4, the last normal-looking week. A team reading only accuracy would see nothing wrong for another month. A team reading the whole row has a named feature to investigate in week 5.

In your own words
Why can't you copy a drift threshold from a textbook, and why does the accuracy monitor lag?

Write a sentence or two, then compare it with a good answer in the lesson.

On the job: the weekly report and the blind interval

In a week, this lesson is a standing report. It lists the PSI of each important feature, the share of predictions in each class, the label-based accuracy, and a plain line that says how far back the accuracy goes: "accuracy as of 30 days ago". When the PSI alert fires, nobody retrains at once: you look at the model's own predictions, at the business numbers it drives, and at the freshest labels you can get. Only the last of those can say whether the rule changed.

In your own words
You own a model whose labels take six weeks to arrive. What do you put on its monitoring page?

Write a sentence or two, then compare it with a good answer in the lesson.

Practice what you just read

  • 8 code examples you can edit and run
  • a practice exercise that checks your code
  • a 5-question quiz
  • 12 review cards for spaced repetition

Try a quiz question

A model's inputs look exactly as they did in training, yet its accuracy has dropped sharply. What has most likely happened?

  1. Concept drift: the relationship between inputs and outcome changed
  2. Data drift: the inputs moved away from the training distribution
  3. Overfitting: the model memorized its training rows too closely
  4. Nothing: accuracy always varies by that much from week to week

Answer it at the end of the lesson to see if you're right, and why.

Before this lesson

Keep learning

Where this lesson leads.

575 lessons in thirteen phases. 176 are free without an account, and a free account opens the rest. See every lesson.

Phase 11 · about 34 hours of study

Professional practice

Design, ship and run real systems, work well in a team, and get hired 43 lessons 1 free
  1. System design fundamentals FreeRequirements, rough numbers, the standard parts and where systems break.40 min
  2. Caches, queues and background jobsSpeed reads with caches, and move slow work to queues that retry safely.40 min
  3. Designing AI systemsLatency, cost, routing, fallbacks and guardrails for features built on models.40 min
  4. Containers and deploymentPackage an app as an image, configure it with the environment, and roll out safely.45 min
  5. CI/CD pipelinesTest every change automatically, and ship with gates, flags and rollbacks.40 min
  6. Docker ComposeSeveral containers together: service names, volumes, secrets and start order.40 min
  7. Kubernetes basicsDesired state, Deployments and Services, labels, probes and rolling updates.40 min
  8. Cloud for AI engineersAccess policies and least privilege, storage, compute choices, managed model endpoints, secrets and budgets.30 min
  9. Metrics, dashboards and alert rulesCounters, gauges and histograms, a rate from a counter, a percentile from buckets, a dashboard panel and an alert rule that fires on a sustained condition.40 min
  10. Reliability and incidentsSLOs, error budgets, alerts, on-call and blameless postmortems.40 min
  11. Incident lab: from the page to the postmortemOne incident run from the page to the all-clear: decide the mitigation, work out the impact in failed requests and error budget, and write a blameless postmortem, a status update and a runbook entry.40 min
  12. Measuring an AI feature from its logsTyped views over event logs, metric specs, cost from a price table, and rollups you can rerun.40 min
  13. A/B testing a model changeHash assignment, per-user results from raw logs, sample ratio checks, guardrails and a decision rule.40 min
  14. Columnar files and a local warehouseRows against columns, the encodings that shrink a column, Parquet's row groups and the ranges that let a query skip them, and SQL over files with DuckDB.40 min
  15. Data quality checks and contractsSix properties of a delivery you can test, each as a count of rows that break it, a contract with the producer, and when to block, quarantine or warn.40 min
  16. Transformations: staging, models and testsRaw, staging, facts and marts as SQL models run in dependency order in DuckDB, tests as queries that count the rows that break a rule, an incremental load that is safe to rerun, and the same models as dbt files.45 min
  17. Streams and windowsEvents that never stop arriving: event time against arrival time, tumbling, sliding and session windows, a watermark for late events, and counting each event once.40 min
  18. Labels and annotator agreementLabels as measurements: agreement and Cohen's kappa, the pair of classes annotators confuse, votes and gold questions, and what wrong labels do to a model and a test score.45 min
  19. Experiment tracking and reproducible runsA run record of settings, seed, data and code behind every score, a run log in a dozen lines, and comparing settings across seeds.40 min
  20. Hyperparameter searchWhat a grid costs, random search on a log scale, stopping weak settings early, and why the winner's validation score flatters it.40 min
  21. Versioning data and modelsFiles stored under the hash of their contents, a manifest that names a whole dataset, and lineage from data and code to each model.35 min
  22. A model registry and promotionVersions under one name, an alias for the live one, gates a candidate must pass, group by group, and a rollback that moves a pointer.40 min
  23. Feature pipelines and training-serving skewWhy a model that passes offline fails live: statistics refit at serving, two code paths, and features from the future, with the fix for each.40 min
  24. Batch and online inferencePredict ahead of time or on request: when the inputs exist, latency near a server's limit, batching what waits, cost, and a hybrid with a freshness limit.40 min
  25. Monitoring a model: drift and decayWhy a model worsens with no code change: data drift and concept drift, a measure of how far inputs have moved, thresholds set from noise, and labels that arrive late.45 min
  26. Retraining and pipelinesWhen to retrain and on which weeks, training as steps in dependency order, a cache that reruns only what changed, and guards before and after.45 min
  27. Designing ML systemsOne method for an ML design, goal and metric to retraining, with the sums that size it: requests a second at peak, a year of feature storage, a label delay and a latency budget.45 min
  28. Datasheets: documenting a datasetThe document that travels with a dataset: a profile by column and by group, claims a program can check, splits that don't leak, a scan for personal data, and the license and limits only people can write.45 min
  29. Curation, active learning and synthetic dataThree levers on a training set: removing copies and broken rows, letting a model choose which rows to label, and manufacturing rows, which helps only when their source knows more than you do.45 min
  30. Measuring fairnessBreak a model's results down by group: selection rate, true and false positive rates and precision, why they can't all be equal, what each fix costs, and when a group is too small to judge.40 min
  31. Privacy engineeringWhy removing names isn't anonymity, how to hold less (minimization, retention, keyed pseudonyms), the idea behind differential privacy and its budget, what a model remembers, and a deletion that reaches every store.40 min
  32. A red-team labAttack a simulated assistant methodically: eight kinds of attack run as a suite, success rates beside ordinary requests blocked, fresh attacks to test the fixes, where a defense belongs, and findings kept as release tests.45 min
  33. What regulation asks of an engineerRisk tiers that follow a system's use, the requirements for high-risk systems as artifacts an engineer builds, evidence tied to a model version, a model card a program can check, and legal dates kept as data.35 min
  34. Git for teamsBranches, pull requests, rebasing and how a three-way merge finds conflicts.40 min
  35. Writing for engineersREADMEs, design docs, commit messages and bug reports people can act on.35 min
  36. Scoping and estimating workA vague request cut into tickets with acceptance criteria and a first shippable slice, estimates as ranges, an honest update for a slip, and a ticket linked to its pull request.30 min
  37. Roles, résumé and the searchWhat AI, ML and data engineering postings ask for, résumé bullets that carry a number, a 100-word project summary, a take-home as a portfolio piece and how to read an offer.35 min
  38. Interviews and your portfolioShow your work, and solve interview problems the way professionals do.40 min
  39. Code reviewWhat to look for, how to give and take feedback, and automating the routine checks.35 min
  40. Reading unfamiliar codeFind your way around a big codebase fast, with tools that map it for you.35 min
  41. Working with AI coding toolsGive agents good context, keep tasks small, and check every change they make.35 min
  42. Capstone: ship a production AI serviceAn AI-backed service with validation, caching, rate limits, fallbacks, logging and evals.45 min
  43. Capstone: take a model to productionA churn model through a data contract, a tracked run, a registry gate by slice, hybrid serving, a PSI monitor and one retrain, shipped with a datasheet, a model card, a fairness audit, a design doc and a runbook.45 min
Part of a free course 11 lessons

Free MLOps course: running machine learning in production

Take a model from a notebook to a system that stays right: track and reproduce every run, search settings honestly, version data and models, promote through gates, keep training and serving in step, choose batch or online, catch drift, and retrain through a pipeline. Every number is measured by the lesson's own code.

Never trained an AI? Teach a small one to write, right in your browser. It takes about 20 seconds and needs no account. Teach it to write