Index Ai Academy
Phase 11 · ML in production

Retraining and pipelines

When to retrain and on which weeks, training as steps in dependency order, a cache that reruns only what changed, and guards before and after.

About 45 minutes · A free account opens the whole lesson.
Already have an account? Sign in to open this lesson

What it covers6

Why it matters

Monitoring a model: drift and decay ended with a model that had quietly got worse. The cure is to train it again on newer data, and then again next month, for as long as the model lives. Done by hand, that is a person running a notebook from memory at intervals, and it fails the first time they are on holiday. Done properly, it is a pipeline: the same steps, in the same order, started by a rule, checked at both ends, and safe to run twice. This lesson measures what retraining is worth, then builds the machinery that makes it routine.

What you'll be able to do

  • Decide when to retrain and on which data, and show with numbers what each choice is worth
  • Describe training as a pipeline of steps that depend on each other, and run them in a valid order
  • Skip the steps whose inputs haven't changed, and make a rerun safe
  • Put a data check in front of training and a gate behind it, so a retrain can't ship a worse model unseen
Quick check
A guess before you start. The rule a model learned has shifted, and the team retrains every month on all the data they have ever collected. The old data outnumbers the new many times over. What will the retrained model mostly learn?
  1. An exact half-and-half blend of the old rule and the new one
  2. The old rule, since most of its training rows still follow it
  3. Nothing at all, since the two rules cancel each other out
  4. The new rule, since the newest rows are the last ones it sees
Show the answer

The old rule, since most of its training rows still follow it

A model fits the bulk of its data. If years of rows follow the old rule and a few weeks follow the new one, retraining on everything gives a model that is still mostly about the past. This lesson measures it: retraining on all history recovers far less accuracy than retraining on recent weeks alone.

The idea: when to retrain, on what, and in what order

A retrain is a pipeline of steps with a dependency order, the same ready-step loop as in Trees and graphs and in CI/CD pipelines. The accuracy sums use Describing data.

When to retrain, and on what

Three kinds of rule can start a retrain.

  • A schedule: every week, every month. Simple and predictable, and it retrains whether or not anything changed.
  • A signal: the drift or accuracy monitor from Monitoring a model crosses its threshold. It reacts to need, and it is only as good as the monitor.
  • New data: enough fresh labeled rows have arrived to be worth learning from.

Most teams combine them: a schedule as the floor, a signal that can bring a retrain forward.

The second decision is which rows to train on. All history uses everything. A recent window uses only the last few weeks and forgets the rest. The loan model from Monitoring a model can settle the question. Here the rule that links income to repayment starts shifting in week 5 and settles in week 16, and each loan's outcome is known four weeks after the decision.

import numpy as np
from sklearn.linear_model import LogisticRegression

def loans(n, seed, rule_change=0.0):
    rng = np.random.default_rng(seed)
    income = rng.normal(50, 12, size=n)
    debt = rng.normal(20, 6, size=n)
    earn = income - 50
    signal = (0.08 * earn - 0.006 * np.maximum(earn - 10, 0) ** 2
              - 0.15 * (debt - 20) - rule_change * 0.16 * earn)
    return (np.column_stack([income, debt]),
            (signal + rng.normal(0, 1, n) > 0).astype(int))

def world(changing):
    """Week 0 is the launch data. In a changing world the rule shifts
    from week 5 and settles in week 16."""
    data = {0: loans(4000, 0)}
    for w in range(1, 25):
        data[w] = loans(500, 100 + w,
                        rule_change=min(1.0, max(0, w - 4) / 12) if changing else 0.0)
    return data

def fit(data, weeks):
    X = np.vstack([data[w][0] for w in weeks])
    y = np.concatenate([data[w][1] for w in weeks])
    return LogisticRegression().fit(X, y)

def run(data, window=None, every=None, delay=4):
    """Weekly accuracy of a model retrained every `every` weeks
    on the last `window` labeled weeks."""
    model, scores = fit(data, [0]), []
    for w in range(1, 25):
        # outcomes arrive `delay` weeks late
        labeled = [v for v in range(w) if v + delay <= w]
        if every and w % every == 0 and labeled:
            model = fit(data, labeled[-window:] if window else labeled)
        scores.append(float(model.score(*data[w])))
    return scores

data = world(changing=True)
plans = {"never retrain                ": {},
         "every 4 weeks, on all history": {"every": 4},
         "every 4 weeks, on the last 8 ": {"every": 4, "window": 8},
         "every week, on the last 4    ": {"every": 1, "window": 4}}
for name, plan in plans.items():
    s = run(data, **plan)
    print(name, "| mean, weeks 5 to 24:", round(float(np.mean(s[4:])), 3),
          "| weeks 12, 16, 20, 24:", [round(s[w - 1], 2) for w in (12, 16, 20, 24)])
It runs right here in your browser, with nothing to install.

Read the last column of each row, the model's accuracy at four moments.

  • Never retrained, it sinks to about 0.52 and stays there. Averaged over weeks 5 to 24: 0.584.
  • Retrained on all history, it does a little better, 0.626. The 4,000 launch rows follow the old rule and outvote the new weeks every time. This is the opening question's answer.
  • Retrained on the last 8 labeled weeks, it recovers to 0.79 once the rule settles: 0.688 on average.
  • Retrained every week on the last 4, it adapts fastest and averages 0.734.

So when the world has changed, recent data beats more data, and retraining often beats retraining rarely. Before taking that as a law, run the same plans in a world that doesn't change:

import numpy as np
from sklearn.linear_model import LogisticRegression

def loans(n, seed, rule_change=0.0):
    rng = np.random.default_rng(seed)
    income = rng.normal(50, 12, size=n)
    debt = rng.normal(20, 6, size=n)
    earn = income - 50
    signal = (0.08 * earn - 0.006 * np.maximum(earn - 10, 0) ** 2
              - 0.15 * (debt - 20) - rule_change * 0.16 * earn)
    return (np.column_stack([income, debt]),
            (signal + rng.normal(0, 1, n) > 0).astype(int))

def world(changing):
    data = {0: loans(4000, 0)}
    for w in range(1, 25):
        data[w] = loans(500, 100 + w,
                        rule_change=min(1.0, max(0, w - 4) / 12) if changing else 0.0)
    return data

def fit(data, weeks):
    return LogisticRegression().fit(np.vstack([data[w][0] for w in weeks]),
                                    np.concatenate([data[w][1] for w in weeks]))

def run(data, window=None, every=None, delay=4):
    model, scores = fit(data, [0]), []
    for w in range(1, 25):
        labeled = [v for v in range(w) if v + delay <= w]
        if every and w % every == 0 and labeled:
            model = fit(data, labeled[-window:] if window else labeled)
        scores.append(float(model.score(*data[w])))
    return round(float(np.mean(scores[4:])), 3)

for name, data in (("the rule changes", world(True)),
                   ("nothing changes ", world(False))):
    print(name, "| never retrain", run(data),
          "| weekly on the last 4", run(data, window=4, every=1),
          "| the same, with labels at once", run(data, window=4, every=1, delay=0))

In the world where nothing changes, never retraining scores 0.779 and weekly retraining 0.78. Fifty-two retrains a year would buy nothing, and each one is a chance to ship a model trained on a bad week of data. Retraining is a response to change, not a ritual.

The last column prices something else. If outcomes were known at once, the weekly plan would score 0.766 in the changing world. With a four-week wait it scores 0.734. A retrained model is always fitted to the world as it was when its newest labels were made, so slow labels cap how well any retraining plan can do.

Quick check
A model can be retrained on the weeks whose outcomes have arrived, and outcomes arrive 4 weeks late. What does this print?
week, delay, window = 14, 4, 3

labeled = [v for v in range(week) if v + delay <= week]
print(labeled[-window:], len(labeled))
  1. [8, 9, 10] 10
  2. [9, 10, 11] 11
  3. [8, 9, 10] 11
  4. [11, 12, 13] 14
Show the answer

[8, 9, 10] 11

In week 14, a week's outcomes are in if that week plus 4 is no later than 14: weeks 0 to 10, which is 11 weeks. The last three of those are 8, 9 and 10. Weeks 11 to 13 have happened, but nobody knows their outcomes yet.

Quick check
A model predicts next month's electricity demand. Demand follows the seasons, which repeat every year, and nothing else about it has changed in a decade. Which training data makes most sense?
  1. Several full years, so that every season is well represented
  2. Only the last four weeks, since recent data is always best
  3. A random tenth of one year, to keep training fast
  4. Only last summer, since summer has the highest demand
Show the answer

Several full years, so that every season is well represented

A recent window helps when the rule has changed. Here the rule is stable and repeats yearly, so a short window would forget the other seasons and the model would be wrong every time the season turned. Match the window to how the world actually moves.

Training as a pipeline

Retraining is never one command. The data has to be fetched, checked and turned into features, then a model is fitted and evaluated, and each step needs the output of others. Written down, that is a list of steps and, for each one, the steps it needs:

needs = {
    "fetch": [],
    "check": ["fetch"],
    "features": ["check"],
    "train": ["features"],
    "evaluate": ["train", "features"],
    "report": ["evaluate", "check"],
}

done, order = set(), []
while len(order) < len(needs):
    ready = sorted(step for step in needs
                   if step not in done and all(d in done for d in needs[step]))
    order.append(ready[0])
    done.add(ready[0])
print(order)

A step is ready when everything it needs is done. Run a ready step, mark it done, and repeat: the result is an order in which no step runs before its inputs exist. You met this in Trees and graphs as a topological sort, and the picture of steps joined by "needs" arrows is a directed acyclic graph, or DAG: directed because the arrows point one way, acyclic because no step can need itself by any route.

Tools called orchestrators, such as Airflow, Dagster and Prefect, are built around this structure. You declare the steps and what each needs. The orchestrator works out the order, runs independent steps at the same time, retries a step that fails, starts the whole thing on a schedule, and keeps a record of every run. The record is the run log of the experiment tracking lesson, kept for you.

Try it now

Pick a run and step through the steps. Lit steps ran; skipped steps came from the cache.

Quick check
From the trees and graphs lesson: a pipeline's steps and what each needs are below. Which is a valid order to run them in?
needs = {
    "train": ["features", "settings"],
    "features": ["clean"],
    "clean": [],
    "settings": [],
    "evaluate": ["train"],
}
  1. evaluate, train, features, clean, settings
  2. clean, features, train, settings, evaluate
  3. train, features, clean, settings, evaluate
  4. clean, settings, features, train, evaluate
Show the answer

clean, settings, features, train, evaluate

Every step comes after everything it needs: clean and settings need nothing, features needs clean, train needs features and settings, evaluate needs train. In the third option train runs before settings, one of its two inputs, exists.

In your own words
In your own words: when is a recent window of training data better than all of history, and when is it worse?

Write a sentence or two, then compare it with a good answer in the lesson.

In code: a cache, a safe rerun and the guards

The cells below run the plans and the cache for real, on a loan model whose rule shifts in week 5.

Drill

A valid order for the steps

Write run_order(needs). needs maps each step's name to the list of steps it needs. Return a list of every step in an order where each comes after all the steps it needs. When several steps are ready at once, take the one that comes first alphabetically, so the answer is always the same. Raise ValueError if a step needs a name that isn't a step, or if the steps can't all be ordered because some need each other in a loop.

def run_order(needs):
    return sorted(needs)


print(run_order({"fetch": [], "check": ["fetch"], "features": ["check"],
                 "train": ["features"], "evaluate": ["train", "features"]}))
In the lesson, your code is checked as you write it.

Run only what changed, and make reruns safe

A retrain that fetches and cleans a year of data to change one setting is wasting an hour. A pipeline can skip any step whose inputs are the same as last time. Versioning data and models gave you the tool: a fingerprint. Give each step a key made from its name and the keys of everything it needs. If that key is already in the cache, the output is already known.

import hashlib, json

def fingerprint(value):
    return hashlib.sha256(json.dumps(value, sort_keys=True).encode()).hexdigest()[:10]

def run_pipeline(steps, inputs, cache):
    """steps: name -> (the names it needs, a function). inputs: name -> value.
    Returns outputs and a log."""
    keys = {name: fingerprint(value) for name, value in inputs.items()}
    outputs, log, waiting = dict(inputs), [], dict(steps)
    while waiting:
        name = sorted(n for n, (needs, fn) in waiting.items()
                      if all(d in keys for d in needs))[0]
        needs, fn = waiting.pop(name)
        # this step, on exactly these inputs
        keys[name] = fingerprint([name] + [keys[d] for d in needs])
        if keys[name] in cache:
            outputs[name] = cache[keys[name]]
            log.append(name + ": cached")
        else:
            outputs[name] = cache[keys[name]] = fn(*[outputs[d] for d in needs])
            log.append(name + ": ran")
    return outputs, log

steps = {
    "clean": (["rows"], lambda rows: [r for r in rows if r["amount"] is not None]),
    "features": (["clean"], lambda rows: [[r["amount"] / 100, r["age"]] for r in rows]),
    "train": (["features", "settings"],
              lambda X, s: [round(sum(col) / len(col) * s["scale"], 3)
                            for col in zip(*X)]),
    "evaluate": (["train", "features"], lambda model, X: round(sum(model), 3)),
}
rows = [{"amount": 1200, "age": 3}, {"amount": None, "age": 9},
        {"amount": 800, "age": 5}]
cache = {}

print(run_pipeline(steps, {"rows": rows, "settings": {"scale": 1.0}}, cache)[1])
print(run_pipeline(steps, {"rows": rows, "settings": {"scale": 1.0}}, cache)[1])
print(run_pipeline(steps, {"rows": rows, "settings": {"scale": 2.0}}, cache)[1])
print(run_pipeline(steps, {"rows": rows + [{"amount": 400, "age": 1}],
                           "settings": {"scale": 1.0}}, cache)[1])

Four runs, four different amounts of work.

  1. The first run has an empty cache, so every step runs.
  2. The second has the same inputs. Every key matches and nothing runs.
  3. The third changes a setting. Cleaning and features don't depend on it, so their keys are unchanged. Only training and what follows it run again.
  4. The fourth adds a row. That changes the first key in the chain, so everything downstream runs again.

A change flows down the arrows and nowhere else. Real orchestrators also put the version of each step's code into its key, so that editing a step reruns it and everything after it.

A rerun must be safe

Steps fail halfway: a network drops, a machine is taken away. The orchestrator retries, so every step must be safe to run twice. The word for that, from the lessons on automation and on reliability, is idempotent: doing it again leaves things as they were after the first time.

scores_appended, scores_keyed = [], {}

def save_appending(day, scores):
    scores_appended.extend((day, customer, s) for customer, s in scores.items())

def save_keyed(day, scores):
    # the whole day's output, written under the day
    scores_keyed[day] = dict(scores)

today = {"ana": 0.21, "ben": 0.67}
# the step is retried after a failure further down
for attempt in (1, 2):
    save_appending("2026-11-10", today)
    save_keyed("2026-11-10", today)
print("appending:", len(scores_appended), "rows | keyed by day:",
      sum(len(v) for v in scores_keyed.values()), "rows")

The appending step wrote every score twice. The other wrote the day's output under the day, so the retry replaced it with the same thing. The rule: a step writes its whole output to a place named by its inputs, a date, a run id, a key, and never adds to something shared.

Quick check
A pipeline caches each step under a key made from the step's name and its inputs' keys. The steps are fetch → clean → features → train → evaluate, and train also reads a settings file. Only the settings file changes. Which steps run again?
  1. All five: any change to an input reruns the whole pipeline
  2. train alone: evaluate reads the model, not the settings
  3. None: the data is unchanged, so every step is cached
  4. train and evaluate: the only steps downstream of the settings
Show the answer

train and evaluate: the only steps downstream of the settings

A step reruns when its key changes, and its key changes when any input's key does. The settings feed train, so train's key changes. Evaluate reads train's output, so its key changes too. Fetch, clean and features don't depend on the settings and stay cached.

Guards at both ends

An automatic retrain is a machine that turns whatever data arrives into a model and puts it in front of customers. Two guards keep it honest.

Before training: check the data. The monitoring lesson's warning was that a broken upstream job looks like drift, and retraining on it teaches the model the bug. So the first step after fetching is a set of plain checks: enough rows, required columns present, missing values within a limit, values inside sensible ranges. If any fails, the pipeline stops and a person looks.

def check_data(rows, min_rows, max_missing, ranges):
    problems = []
    if len(rows) < min_rows:
        problems.append("too few rows: " + str(len(rows)))
    for column, (low, high) in ranges.items():
        missing = sum(1 for r in rows if r.get(column) is None)
        if rows and missing / len(rows) > max_missing:
            problems.append(column + ": " + str(missing) + " missing")
        outside = sum(1 for r in rows
                      if r.get(column) is not None and not low <= r[column] <= high)
        if outside:
            problems.append(column + ": " + str(outside) + " out of range")
    return problems

good = [{"income": 40 + i % 30, "debt": 10 + i % 20} for i in range(200)]
# a unit change upstream
cents = [{"income": (40 + i % 30) * 100, "debt": 10 + i % 20} for i in range(200)]
gaps = [{"income": None if i % 4 == 0 else 50, "debt": 15} for i in range(200)]

for name, rows in (("a normal week", good), ("income arrives in cents", cents),
                   ("a quarter of incomes missing", gaps), ("a short file", good[:30])):
    print(name, "->",
          check_data(rows, min_rows=100, max_missing=0.05,
                     ranges={"income": (0, 500), "debt": (0, 200)}) or "ok")

None of these checks is clever. Each one catches a failure that would otherwise be trained into a model and discovered by customers.

After training: the new model is a challenger, not a champion. A retrain produces a candidate. It is registered as a new version and has to pass the promotion gates from the registry lesson: a complete record, a real gain over the champion on the fixed evaluation set, no slice much worse. If it fails, the champion stays and nothing is lost. Automating the retrain doesn't mean automating the launch.

Go deeper When the model shapes its own training data

A loan model only learns the outcome of loans it approved. A recommender only sees clicks on what it chose to show. Retrain on that, and each new model inherits the blind spots of the last: applicants it would have refused never appear in the data as people who might have repaid. This is a feedback loop. The usual remedy is to make a small share of decisions outside the model's advice, for example approving a random few of the borderline cases, and to treat those rows as the unbiased evidence. It costs something, and it is the only way to keep learning about the cases the model avoids.

Drill

Should the pipeline retrain today?

Write should_retrain(status, policy). status has "checks_passed" (bool), "new_labels" (int), "accuracy" (a float, or None if no outcomes are known yet), "psi" (float) and "weeks_since" (int). policy has "min_labels", "min_accuracy", "max_psi" and "max_weeks". Return the first of these that applies:

  1. "blocked: data checks failed" if the checks didn't pass.
  2. None if there are fewer new labels than min_labels: there is nothing to learn from yet.
  3. "accuracy" if the accuracy is known and below min_accuracy.
  4. "drift" if the PSI is above max_psi.
  5. "schedule" if weeks_since is at least max_weeks.
  6. None otherwise.
def should_retrain(status, policy):
    return None


policy = {"min_labels": 500, "min_accuracy": 0.75, "max_psi": 0.25, "max_weeks": 8}
print(should_retrain({"checks_passed": True, "new_labels": 2000, "accuracy": 0.71,
                      "psi": 0.05, "weeks_since": 3}, policy))
In the lesson, your code is checked as you write it.
Quick check
An automatic retrain produces a model that scores lower than the champion on the fixed evaluation set. What should the pipeline do?
  1. Rerun training with new seeds until one candidate passes
  2. Keep the champion, record the failed candidate, and alert the team
  3. Promote the new model anyway, since it was trained on newer data
  4. Delete the champion, since the retrain shows it is out of date
Show the answer

Keep the champion, record the failed candidate, and alert the team

A retrain produces a candidate. If it fails the gates the champion stays, and the failure is itself information: the new data may be bad, or the evaluation set may no longer represent the world. Both need a person. Rerunning until something passes is the winner's-luck trap from the search lesson.

The four plans from the first cell, drawn at the four weeks it prints:

import matplotlib.pyplot as plt

weeks = [12, 16, 20, 24]
plans = {
    "never": [0.57, 0.52, 0.51, 0.54],
    "all history": [0.59, 0.58, 0.62, 0.66],
    "last 8 weeks": [0.6, 0.68, 0.79, 0.78],
    "weekly, last 4": [0.64, 0.74, 0.8, 0.79],
}
for name, ys in plans.items():
    plt.plot(weeks, ys, marker="o", label=name)
plt.xlabel("week")
plt.ylabel("accuracy")
plt.legend()
plt.show()

The never-retrained line stays near 0.5, and the two window plans climb once the rule has settled.

Worked example

Worked example: one turn of the loop, end to end

Tie the whole track together on the loan model in week 20 of the changing world: decide, check the data, train a candidate on the recent labeled weeks, and gate it against the champion on the newest labeled week, which neither model trained on.

import numpy as np
from sklearn.linear_model import LogisticRegression

def loans(n, seed, rule_change=0.0):
    rng = np.random.default_rng(seed)
    income = rng.normal(50, 12, size=n)
    debt = rng.normal(20, 6, size=n)
    earn = income - 50
    signal = (0.08 * earn - 0.006 * np.maximum(earn - 10, 0) ** 2
              - 0.15 * (debt - 20) - rule_change * 0.16 * earn)
    return (np.column_stack([income, debt]),
            (signal + rng.normal(0, 1, n) > 0).astype(int))

data = {0: loans(4000, 0)}
for w in range(1, 21):
    data[w] = loans(500, 100 + w, rule_change=min(1.0, max(0, w - 4) / 12))

week, delay = 20, 4
# weeks 0 to 16 have outcomes
labeled = [v for v in range(week) if v + delay <= week]
# judge on week 16, train on weeks 12 to 15
holdout, train_weeks = labeled[-1], labeled[-5:-1]

champion = LogisticRegression().fit(*data[0])
known_accuracy = float(champion.score(*data[holdout]))
print("1 trigger | champion's accuracy on week", holdout,
      "is", round(known_accuracy, 3),
      "->", "retrain" if known_accuracy < 0.75 else "leave it")

X = np.vstack([data[w][0] for w in train_weeks])
y = np.concatenate([data[w][1] for w in train_weeks])
ok = (len(X) >= 1000 and not np.isnan(X).any()
      and X[:, 0].min() > 0 and X[:, 0].max() < 500)
print("2 data    |", len(X), "rows from weeks", train_weeks,
      "->", "ok" if ok else "STOP")

candidate = LogisticRegression().fit(X, y)
gain = float(candidate.score(*data[holdout])) - known_accuracy
print("3 gates   | candidate", round(float(candidate.score(*data[holdout])), 3),
      "| gain", round(gain, 3),
      "->", "promote" if gain >= 0.02 else "keep the champion")
print("4 today   | week 20: champion", round(float(champion.score(*data[20])), 3),
      "| candidate", round(float(candidate.score(*data[20])), 3))
In the lesson you can change this and run it.

Line 1 is the trigger: on the newest week with outcomes, the champion is well under the 0.75 floor. Line 2 is the data check, passed. Line 3 is the gate, and it is honest because week 16 was held out of the candidate's training: the candidate beats the champion there by a wide margin, so it is promoted. Line 4 is the number nobody could see on the day, the two models on week 20 itself, shown here only because this is a simulation. The candidate is far ahead there too. Every piece of the track is in these few lines: a monitor that respects late labels, a data check, a logged training run on a chosen window, a gate against the live model, and a promotion that keeps the old version.

In your own words
Why must a pipeline step be safe to run twice, and how do you make it so?

Write a sentence or two, then compare it with a good answer in the lesson.

On the job: an Airflow DAG and a schedule

In production the pipeline is declared to an orchestrator, which runs it on a trigger, retries a failed step and keeps the run log.

Where it comes up in a week

The monthly retrain ran on Sunday and its gate refused the candidate: the data check found half the usual rows. You open the run log, see which step stopped it, and read the upstream job's failure. Nothing was promoted, the champion kept serving, and the retrain reruns safely once the data is fixed, because every step writes to a place named by its inputs.

Goes deeper in

In your own words
Why should an automated retraining pipeline end with a gate and not with a deployment?

Write a sentence or two, then compare it with a good answer in the lesson.

Practice what you just read

  • 8 code examples you can edit and run
  • a practice exercise that checks your code
  • a 5-question quiz
  • 10 review cards for spaced repetition

Try a quiz question

The relationship a model learned changed three months ago and has been stable since. Which retraining data will give the best model?

  1. The labeled rows from after the change
  2. Every labeled row ever collected
  3. Only the rows from before the change
  4. A random sample of all the rows

Answer it at the end of the lesson to see if you're right, and why.

Before this lesson

Keep learning

Where this lesson leads.

575 lessons in thirteen phases. 176 are free without an account, and a free account opens the rest. See every lesson.

Phase 11 · about 34 hours of study

Professional practice

Design, ship and run real systems, work well in a team, and get hired 43 lessons 1 free
  1. System design fundamentals FreeRequirements, rough numbers, the standard parts and where systems break.40 min
  2. Caches, queues and background jobsSpeed reads with caches, and move slow work to queues that retry safely.40 min
  3. Designing AI systemsLatency, cost, routing, fallbacks and guardrails for features built on models.40 min
  4. Containers and deploymentPackage an app as an image, configure it with the environment, and roll out safely.45 min
  5. CI/CD pipelinesTest every change automatically, and ship with gates, flags and rollbacks.40 min
  6. Docker ComposeSeveral containers together: service names, volumes, secrets and start order.40 min
  7. Kubernetes basicsDesired state, Deployments and Services, labels, probes and rolling updates.40 min
  8. Cloud for AI engineersAccess policies and least privilege, storage, compute choices, managed model endpoints, secrets and budgets.30 min
  9. Metrics, dashboards and alert rulesCounters, gauges and histograms, a rate from a counter, a percentile from buckets, a dashboard panel and an alert rule that fires on a sustained condition.40 min
  10. Reliability and incidentsSLOs, error budgets, alerts, on-call and blameless postmortems.40 min
  11. Incident lab: from the page to the postmortemOne incident run from the page to the all-clear: decide the mitigation, work out the impact in failed requests and error budget, and write a blameless postmortem, a status update and a runbook entry.40 min
  12. Measuring an AI feature from its logsTyped views over event logs, metric specs, cost from a price table, and rollups you can rerun.40 min
  13. A/B testing a model changeHash assignment, per-user results from raw logs, sample ratio checks, guardrails and a decision rule.40 min
  14. Columnar files and a local warehouseRows against columns, the encodings that shrink a column, Parquet's row groups and the ranges that let a query skip them, and SQL over files with DuckDB.40 min
  15. Data quality checks and contractsSix properties of a delivery you can test, each as a count of rows that break it, a contract with the producer, and when to block, quarantine or warn.40 min
  16. Transformations: staging, models and testsRaw, staging, facts and marts as SQL models run in dependency order in DuckDB, tests as queries that count the rows that break a rule, an incremental load that is safe to rerun, and the same models as dbt files.45 min
  17. Streams and windowsEvents that never stop arriving: event time against arrival time, tumbling, sliding and session windows, a watermark for late events, and counting each event once.40 min
  18. Labels and annotator agreementLabels as measurements: agreement and Cohen's kappa, the pair of classes annotators confuse, votes and gold questions, and what wrong labels do to a model and a test score.45 min
  19. Experiment tracking and reproducible runsA run record of settings, seed, data and code behind every score, a run log in a dozen lines, and comparing settings across seeds.40 min
  20. Hyperparameter searchWhat a grid costs, random search on a log scale, stopping weak settings early, and why the winner's validation score flatters it.40 min
  21. Versioning data and modelsFiles stored under the hash of their contents, a manifest that names a whole dataset, and lineage from data and code to each model.35 min
  22. A model registry and promotionVersions under one name, an alias for the live one, gates a candidate must pass, group by group, and a rollback that moves a pointer.40 min
  23. Feature pipelines and training-serving skewWhy a model that passes offline fails live: statistics refit at serving, two code paths, and features from the future, with the fix for each.40 min
  24. Batch and online inferencePredict ahead of time or on request: when the inputs exist, latency near a server's limit, batching what waits, cost, and a hybrid with a freshness limit.40 min
  25. Monitoring a model: drift and decayWhy a model worsens with no code change: data drift and concept drift, a measure of how far inputs have moved, thresholds set from noise, and labels that arrive late.45 min
  26. Retraining and pipelinesWhen to retrain and on which weeks, training as steps in dependency order, a cache that reruns only what changed, and guards before and after.45 min
  27. Designing ML systemsOne method for an ML design, goal and metric to retraining, with the sums that size it: requests a second at peak, a year of feature storage, a label delay and a latency budget.45 min
  28. Datasheets: documenting a datasetThe document that travels with a dataset: a profile by column and by group, claims a program can check, splits that don't leak, a scan for personal data, and the license and limits only people can write.45 min
  29. Curation, active learning and synthetic dataThree levers on a training set: removing copies and broken rows, letting a model choose which rows to label, and manufacturing rows, which helps only when their source knows more than you do.45 min
  30. Measuring fairnessBreak a model's results down by group: selection rate, true and false positive rates and precision, why they can't all be equal, what each fix costs, and when a group is too small to judge.40 min
  31. Privacy engineeringWhy removing names isn't anonymity, how to hold less (minimization, retention, keyed pseudonyms), the idea behind differential privacy and its budget, what a model remembers, and a deletion that reaches every store.40 min
  32. A red-team labAttack a simulated assistant methodically: eight kinds of attack run as a suite, success rates beside ordinary requests blocked, fresh attacks to test the fixes, where a defense belongs, and findings kept as release tests.45 min
  33. What regulation asks of an engineerRisk tiers that follow a system's use, the requirements for high-risk systems as artifacts an engineer builds, evidence tied to a model version, a model card a program can check, and legal dates kept as data.35 min
  34. Git for teamsBranches, pull requests, rebasing and how a three-way merge finds conflicts.40 min
  35. Writing for engineersREADMEs, design docs, commit messages and bug reports people can act on.35 min
  36. Scoping and estimating workA vague request cut into tickets with acceptance criteria and a first shippable slice, estimates as ranges, an honest update for a slip, and a ticket linked to its pull request.30 min
  37. Roles, résumé and the searchWhat AI, ML and data engineering postings ask for, résumé bullets that carry a number, a 100-word project summary, a take-home as a portfolio piece and how to read an offer.35 min
  38. Interviews and your portfolioShow your work, and solve interview problems the way professionals do.40 min
  39. Code reviewWhat to look for, how to give and take feedback, and automating the routine checks.35 min
  40. Reading unfamiliar codeFind your way around a big codebase fast, with tools that map it for you.35 min
  41. Working with AI coding toolsGive agents good context, keep tasks small, and check every change they make.35 min
  42. Capstone: ship a production AI serviceAn AI-backed service with validation, caching, rate limits, fallbacks, logging and evals.45 min
  43. Capstone: take a model to productionA churn model through a data contract, a tracked run, a registry gate by slice, hybrid serving, a PSI monitor and one retrain, shipped with a datasheet, a model card, a fairness audit, a design doc and a runbook.45 min
Part of a free course 11 lessons

Free MLOps course: running machine learning in production

Take a model from a notebook to a system that stays right: track and reproduce every run, search settings honestly, version data and models, promote through gates, keep training and serving in step, choose batch or online, catch drift, and retrain through a pipeline. Every number is measured by the lesson's own code.

Never trained an AI? Teach a small one to write, right in your browser. It takes about 20 seconds and needs no account. Teach it to write