Monitoring a model: drift and decay ended with a model that had quietly got worse. The cure is to train it again on newer data, and then again next month, for as long as the model lives. Done by hand, that is a person running a notebook from memory at intervals, and it fails the first time they are on holiday. Done properly, it is a pipeline: the same steps, in the same order, started by a rule, checked at both ends, and safe to run twice. This lesson measures what retraining is worth, then builds the machinery that makes it routine.
What you'll be able to do
Decide when to retrain and on which data, and show with numbers what each choice is worth
Describe training as a pipeline of steps that depend on each other, and run them in a valid order
Skip the steps whose inputs haven't changed, and make a rerun safe
Put a data check in front of training and a gate behind it, so a retrain can't ship a worse model unseen
Quick check
A guess before you start. The rule a model learned has shifted, and the team retrains every month on all the data they have ever collected. The old data outnumbers the new many times over. What will the retrained model mostly learn?
An exact half-and-half blend of the old rule and the new one
The old rule, since most of its training rows still follow it
Nothing at all, since the two rules cancel each other out
The new rule, since the newest rows are the last ones it sees
Show the answer
The old rule, since most of its training rows still follow it
A model fits the bulk of its data. If years of rows follow the old rule and a few weeks follow the new one, retraining on everything gives a model that is still mostly about the past. This lesson measures it: retraining on all history recovers far less accuracy than retraining on recent weeks alone.
The idea: when to retrain, on what, and in what order
A schedule: every week, every month. Simple and predictable, and it retrains whether or not anything changed.
A signal: the drift or accuracy monitor from Monitoring a model crosses its threshold. It reacts to need, and it is only as good as the monitor.
New data: enough fresh labeled rows have arrived to be worth learning from.
Most teams combine them: a schedule as the floor, a signal that can bring a retrain forward.
The second decision is which rows to train on. All history uses everything. A recent window uses only the last few weeks and forgets the rest. The loan model from Monitoring a model can settle the question. Here the rule that links income to repayment starts shifting in week 5 and settles in week 16, and each loan's outcome is known four weeks after the decision.
import numpy as np
from sklearn.linear_model import LogisticRegression
defloans(n, seed, rule_change=0.0):
rng = np.random.default_rng(seed)
income = rng.normal(50, 12, size=n)
debt = rng.normal(20, 6, size=n)
earn = income - 50
signal = (0.08 * earn - 0.006 * np.maximum(earn - 10, 0) ** 2
- 0.15 * (debt - 20) - rule_change * 0.16 * earn)
return (np.column_stack([income, debt]),
(signal + rng.normal(0, 1, n) > 0).astype(int))
defworld(changing):
"""Week 0 is the launch data. In a changing world the rule shifts
from week 5 and settles in week 16."""
data = {0: loans(4000, 0)}
for w inrange(1, 25):
data[w] = loans(500, 100 + w,
rule_change=min(1.0, max(0, w - 4) / 12) if changing else0.0)
return data
deffit(data, weeks):
X = np.vstack([data[w][0] for w in weeks])
y = np.concatenate([data[w][1] for w in weeks])
return LogisticRegression().fit(X, y)
defrun(data, window=None, every=None, delay=4):
"""Weekly accuracy of a model retrained every `every` weeks
on the last `window` labeled weeks."""
model, scores = fit(data, [0]), []
for w inrange(1, 25):
# outcomes arrive `delay` weeks late
labeled = [v for v inrange(w) if v + delay <= w]
if every and w % every == 0and labeled:
model = fit(data, labeled[-window:] if window else labeled)
scores.append(float(model.score(*data[w])))
return scores
data = world(changing=True)
plans = {"never retrain ": {},
"every 4 weeks, on all history": {"every": 4},
"every 4 weeks, on the last 8 ": {"every": 4, "window": 8},
"every week, on the last 4 ": {"every": 1, "window": 4}}
for name, plan in plans.items():
s = run(data, **plan)
print(name, "| mean, weeks 5 to 24:", round(float(np.mean(s[4:])), 3),
"| weeks 12, 16, 20, 24:", [round(s[w - 1], 2) for w in (12, 16, 20, 24)])
It runs right here in your browser, with nothing to install.
Read the last column of each row, the model's accuracy at four moments.
Never retrained, it sinks to about 0.52 and stays there. Averaged over weeks 5 to 24: 0.584.
Retrained on all history, it does a little better, 0.626. The 4,000 launch rows follow the old rule and outvote the new weeks every time. This is the opening question's answer.
Retrained on the last 8 labeled weeks, it recovers to 0.79 once the rule settles: 0.688 on average.
Retrained every week on the last 4, it adapts fastest and averages 0.734.
So when the world has changed, recent data beats more data, and retraining often beats retraining rarely. Before taking that as a law, run the same plans in a world that doesn't change:
import numpy as np
from sklearn.linear_model import LogisticRegression
defloans(n, seed, rule_change=0.0):
rng = np.random.default_rng(seed)
income = rng.normal(50, 12, size=n)
debt = rng.normal(20, 6, size=n)
earn = income - 50
signal = (0.08 * earn - 0.006 * np.maximum(earn - 10, 0) ** 2
- 0.15 * (debt - 20) - rule_change * 0.16 * earn)
return (np.column_stack([income, debt]),
(signal + rng.normal(0, 1, n) > 0).astype(int))
defworld(changing):
data = {0: loans(4000, 0)}
for w inrange(1, 25):
data[w] = loans(500, 100 + w,
rule_change=min(1.0, max(0, w - 4) / 12) if changing else0.0)
return data
deffit(data, weeks):
return LogisticRegression().fit(np.vstack([data[w][0] for w in weeks]),
np.concatenate([data[w][1] for w in weeks]))
defrun(data, window=None, every=None, delay=4):
model, scores = fit(data, [0]), []
for w inrange(1, 25):
labeled = [v for v inrange(w) if v + delay <= w]
if every and w % every == 0and labeled:
model = fit(data, labeled[-window:] if window else labeled)
scores.append(float(model.score(*data[w])))
returnround(float(np.mean(scores[4:])), 3)
for name, data in (("the rule changes", world(True)),
("nothing changes ", world(False))):
print(name, "| never retrain", run(data),
"| weekly on the last 4", run(data, window=4, every=1),
"| the same, with labels at once", run(data, window=4, every=1, delay=0))
In the world where nothing changes, never retraining scores 0.779 and weekly retraining 0.78. Fifty-two retrains a year would buy nothing, and each one is a chance to ship a model trained on a bad week of data. Retraining is a response to change, not a ritual.
The last column prices something else. If outcomes were known at once, the weekly plan would score 0.766 in the changing world. With a four-week wait it scores 0.734. A retrained model is always fitted to the world as it was when its newest labels were made, so slow labels cap how well any retraining plan can do.
Quick check
A model can be retrained on the weeks whose outcomes have arrived, and outcomes arrive 4 weeks late. What does this print?
week, delay, window = 14, 4, 3
labeled = [v for v inrange(week) if v + delay <= week]
print(labeled[-window:], len(labeled))
[8, 9, 10] 10
[9, 10, 11] 11
[8, 9, 10] 11
[11, 12, 13] 14
Show the answer
[8, 9, 10] 11
In week 14, a week's outcomes are in if that week plus 4 is no later than 14: weeks 0 to 10, which is 11 weeks. The last three of those are 8, 9 and 10. Weeks 11 to 13 have happened, but nobody knows their outcomes yet.
Quick check
A model predicts next month's electricity demand. Demand follows the seasons, which repeat every year, and nothing else about it has changed in a decade. Which training data makes most sense?
Several full years, so that every season is well represented
Only the last four weeks, since recent data is always best
A random tenth of one year, to keep training fast
Only last summer, since summer has the highest demand
Show the answer
Several full years, so that every season is well represented
A recent window helps when the rule has changed. Here the rule is stable and repeats yearly, so a short window would forget the other seasons and the model would be wrong every time the season turned. Match the window to how the world actually moves.
Training as a pipeline
Retraining is never one command. The data has to be fetched, checked and turned into features, then a model is fitted and evaluated, and each step needs the output of others. Written down, that is a list of steps and, for each one, the steps it needs:
needs = {
"fetch": [],
"check": ["fetch"],
"features": ["check"],
"train": ["features"],
"evaluate": ["train", "features"],
"report": ["evaluate", "check"],
}
done, order = set(), []
whilelen(order) < len(needs):
ready = sorted(step for step in needs
if step notin done andall(d in done for d in needs[step]))
order.append(ready[0])
done.add(ready[0])
print(order)
A step is ready when everything it needs is done. Run a ready step, mark it done, and repeat: the result is an order in which no step runs before its inputs exist. You met this in Trees and graphs as a topological sort, and the picture of steps joined by "needs" arrows is a directed acyclic graph, or DAG: directed because the arrows point one way, acyclic because no step can need itself by any route.
Tools called orchestrators, such as Airflow, Dagster and Prefect, are built around this structure. You declare the steps and what each needs. The orchestrator works out the order, runs independent steps at the same time, retries a step that fails, starts the whole thing on a schedule, and keeps a record of every run. The record is the run log of the experiment tracking lesson, kept for you.
Try it now
Pick a run and step through the steps. Lit steps ran; skipped steps came from the cache.
Quick check
From the trees and graphs lesson: a pipeline's steps and what each needs are below. Which is a valid order to run them in?
Every step comes after everything it needs: clean and settings need nothing, features needs clean, train needs features and settings, evaluate needs train. In the third option train runs before settings, one of its two inputs, exists.
In your own words
In your own words: when is a recent window of training data better than all of history, and when is it worse?
Write a sentence or two, then compare it with a good answer in the lesson.
In code: a cache, a safe rerun and the guards
The cells below run the plans and the cache for real, on a loan model whose rule shifts in week 5.
Drill
A valid order for the steps
Write run_order(needs). needs maps each step's name to the list of steps it needs. Return a list of every step in an order where each comes after all the steps it needs. When several steps are ready at once, take the one that comes first alphabetically, so the answer is always the same. Raise ValueError if a step needs a name that isn't a step, or if the steps can't all be ordered because some need each other in a loop.
In the lesson, your code is checked as you write it.
Run only what changed, and make reruns safe
A retrain that fetches and cleans a year of data to change one setting is wasting an hour. A pipeline can skip any step whose inputs are the same as last time. Versioning data and models gave you the tool: a fingerprint. Give each step a key made from its name and the keys of everything it needs. If that key is already in the cache, the output is already known.
import hashlib, json
deffingerprint(value):
return hashlib.sha256(json.dumps(value, sort_keys=True).encode()).hexdigest()[:10]
defrun_pipeline(steps, inputs, cache):
"""steps: name -> (the names it needs, a function). inputs: name -> value.
Returns outputs and a log."""
keys = {name: fingerprint(value) for name, value in inputs.items()}
outputs, log, waiting = dict(inputs), [], dict(steps)
while waiting:
name = sorted(n for n, (needs, fn) in waiting.items()
ifall(d in keys for d in needs))[0]
needs, fn = waiting.pop(name)
# this step, on exactly these inputs
keys[name] = fingerprint([name] + [keys[d] for d in needs])
if keys[name] in cache:
outputs[name] = cache[keys[name]]
log.append(name + ": cached")
else:
outputs[name] = cache[keys[name]] = fn(*[outputs[d] for d in needs])
log.append(name + ": ran")
return outputs, log
steps = {
"clean": (["rows"], lambda rows: [r for r in rows if r["amount"] isnotNone]),
"features": (["clean"], lambda rows: [[r["amount"] / 100, r["age"]] for r in rows]),
"train": (["features", "settings"],
lambda X, s: [round(sum(col) / len(col) * s["scale"], 3)
for col inzip(*X)]),
"evaluate": (["train", "features"], lambda model, X: round(sum(model), 3)),
}
rows = [{"amount": 1200, "age": 3}, {"amount": None, "age": 9},
{"amount": 800, "age": 5}]
cache = {}
print(run_pipeline(steps, {"rows": rows, "settings": {"scale": 1.0}}, cache)[1])
print(run_pipeline(steps, {"rows": rows, "settings": {"scale": 1.0}}, cache)[1])
print(run_pipeline(steps, {"rows": rows, "settings": {"scale": 2.0}}, cache)[1])
print(run_pipeline(steps, {"rows": rows + [{"amount": 400, "age": 1}],
"settings": {"scale": 1.0}}, cache)[1])
Four runs, four different amounts of work.
The first run has an empty cache, so every step runs.
The second has the same inputs. Every key matches and nothing runs.
The third changes a setting. Cleaning and features don't depend on it, so their keys are unchanged. Only training and what follows it run again.
The fourth adds a row. That changes the first key in the chain, so everything downstream runs again.
A change flows down the arrows and nowhere else. Real orchestrators also put the version of each step's code into its key, so that editing a step reruns it and everything after it.
A rerun must be safe
Steps fail halfway: a network drops, a machine is taken away. The orchestrator retries, so every step must be safe to run twice. The word for that, from the lessons on automation and on reliability, is idempotent: doing it again leaves things as they were after the first time.
scores_appended, scores_keyed = [], {}
defsave_appending(day, scores):
scores_appended.extend((day, customer, s) for customer, s in scores.items())
defsave_keyed(day, scores):
# the whole day's output, written under the day
scores_keyed[day] = dict(scores)
today = {"ana": 0.21, "ben": 0.67}
# the step is retried after a failure further downfor attempt in (1, 2):
save_appending("2026-11-10", today)
save_keyed("2026-11-10", today)
print("appending:", len(scores_appended), "rows | keyed by day:",
sum(len(v) for v in scores_keyed.values()), "rows")
The appending step wrote every score twice. The other wrote the day's output under the day, so the retry replaced it with the same thing. The rule: a step writes its whole output to a place named by its inputs, a date, a run id, a key, and never adds to something shared.
Quick check
A pipeline caches each step under a key made from the step's name and its inputs' keys. The steps are fetch → clean → features → train → evaluate, and train also reads a settings file. Only the settings file changes. Which steps run again?
All five: any change to an input reruns the whole pipeline
train alone: evaluate reads the model, not the settings
None: the data is unchanged, so every step is cached
train and evaluate: the only steps downstream of the settings
Show the answer
train and evaluate: the only steps downstream of the settings
A step reruns when its key changes, and its key changes when any input's key does. The settings feed train, so train's key changes. Evaluate reads train's output, so its key changes too. Fetch, clean and features don't depend on the settings and stay cached.
Guards at both ends
An automatic retrain is a machine that turns whatever data arrives into a model and puts it in front of customers. Two guards keep it honest.
Before training: check the data. The monitoring lesson's warning was that a broken upstream job looks like drift, and retraining on it teaches the model the bug. So the first step after fetching is a set of plain checks: enough rows, required columns present, missing values within a limit, values inside sensible ranges. If any fails, the pipeline stops and a person looks.
defcheck_data(rows, min_rows, max_missing, ranges):
problems = []
iflen(rows) < min_rows:
problems.append("too few rows: " + str(len(rows)))
for column, (low, high) in ranges.items():
missing = sum(1for r in rows if r.get(column) isNone)
if rows and missing / len(rows) > max_missing:
problems.append(column + ": " + str(missing) + " missing")
outside = sum(1for r in rows
if r.get(column) isnotNoneandnot low <= r[column] <= high)
if outside:
problems.append(column + ": " + str(outside) + " out of range")
return problems
good = [{"income": 40 + i % 30, "debt": 10 + i % 20} for i inrange(200)]
# a unit change upstream
cents = [{"income": (40 + i % 30) * 100, "debt": 10 + i % 20} for i inrange(200)]
gaps = [{"income": Noneif i % 4 == 0else50, "debt": 15} for i inrange(200)]
for name, rows in (("a normal week", good), ("income arrives in cents", cents),
("a quarter of incomes missing", gaps), ("a short file", good[:30])):
print(name, "->",
check_data(rows, min_rows=100, max_missing=0.05,
ranges={"income": (0, 500), "debt": (0, 200)}) or"ok")
None of these checks is clever. Each one catches a failure that would otherwise be trained into a model and discovered by customers.
After training: the new model is a challenger, not a champion. A retrain produces a candidate. It is registered as a new version and has to pass the promotion gates from the registry lesson: a complete record, a real gain over the champion on the fixed evaluation set, no slice much worse. If it fails, the champion stays and nothing is lost. Automating the retrain doesn't mean automating the launch.
Go deeper When the model shapes its own training data
A loan model only learns the outcome of loans it approved. A recommender only sees clicks on what it chose to show. Retrain on that, and each new model inherits the blind spots of the last: applicants it would have refused never appear in the data as people who might have repaid. This is a feedback loop. The usual remedy is to make a small share of decisions outside the model's advice, for example approving a random few of the borderline cases, and to treat those rows as the unbiased evidence. It costs something, and it is the only way to keep learning about the cases the model avoids.
Drill
Should the pipeline retrain today?
Write should_retrain(status, policy). status has "checks_passed" (bool), "new_labels" (int), "accuracy" (a float, or None if no outcomes are known yet), "psi" (float) and "weeks_since" (int). policy has "min_labels", "min_accuracy", "max_psi" and "max_weeks". Return the first of these that applies:
"blocked: data checks failed" if the checks didn't pass.
None if there are fewer new labels than min_labels: there is nothing to learn from yet.
"accuracy" if the accuracy is known and below min_accuracy.
In the lesson, your code is checked as you write it.
Quick check
An automatic retrain produces a model that scores lower than the champion on the fixed evaluation set. What should the pipeline do?
Rerun training with new seeds until one candidate passes
Keep the champion, record the failed candidate, and alert the team
Promote the new model anyway, since it was trained on newer data
Delete the champion, since the retrain shows it is out of date
Show the answer
Keep the champion, record the failed candidate, and alert the team
A retrain produces a candidate. If it fails the gates the champion stays, and the failure is itself information: the new data may be bad, or the evaluation set may no longer represent the world. Both need a person. Rerunning until something passes is the winner's-luck trap from the search lesson.
The four plans from the first cell, drawn at the four weeks it prints:
The never-retrained line stays near 0.5, and the two window plans climb once the rule has settled.
Worked example
Worked example: one turn of the loop, end to end
Tie the whole track together on the loan model in week 20 of the changing world: decide, check the data, train a candidate on the recent labeled weeks, and gate it against the champion on the newest labeled week, which neither model trained on.
import numpy as np
from sklearn.linear_model import LogisticRegression
defloans(n, seed, rule_change=0.0):
rng = np.random.default_rng(seed)
income = rng.normal(50, 12, size=n)
debt = rng.normal(20, 6, size=n)
earn = income - 50
signal = (0.08 * earn - 0.006 * np.maximum(earn - 10, 0) ** 2
- 0.15 * (debt - 20) - rule_change * 0.16 * earn)
return (np.column_stack([income, debt]),
(signal + rng.normal(0, 1, n) > 0).astype(int))
data = {0: loans(4000, 0)}
for w inrange(1, 21):
data[w] = loans(500, 100 + w, rule_change=min(1.0, max(0, w - 4) / 12))
week, delay = 20, 4# weeks 0 to 16 have outcomes
labeled = [v for v inrange(week) if v + delay <= week]
# judge on week 16, train on weeks 12 to 15
holdout, train_weeks = labeled[-1], labeled[-5:-1]
champion = LogisticRegression().fit(*data[0])
known_accuracy = float(champion.score(*data[holdout]))
print("1 trigger | champion's accuracy on week", holdout,
"is", round(known_accuracy, 3),
"->", "retrain"if known_accuracy < 0.75else"leave it")
X = np.vstack([data[w][0] for w in train_weeks])
y = np.concatenate([data[w][1] for w in train_weeks])
ok = (len(X) >= 1000andnot np.isnan(X).any()
and X[:, 0].min() > 0and X[:, 0].max() < 500)
print("2 data |", len(X), "rows from weeks", train_weeks,
"->", "ok"if ok else"STOP")
candidate = LogisticRegression().fit(X, y)
gain = float(candidate.score(*data[holdout])) - known_accuracy
print("3 gates | candidate", round(float(candidate.score(*data[holdout])), 3),
"| gain", round(gain, 3),
"->", "promote"if gain >= 0.02else"keep the champion")
print("4 today | week 20: champion", round(float(champion.score(*data[20])), 3),
"| candidate", round(float(candidate.score(*data[20])), 3))
In the lesson you can change this and run it.
Line 1 is the trigger: on the newest week with outcomes, the champion is well under the 0.75 floor. Line 2 is the data check, passed. Line 3 is the gate, and it is honest because week 16 was held out of the candidate's training: the candidate beats the champion there by a wide margin, so it is promoted. Line 4 is the number nobody could see on the day, the two models on week 20 itself, shown here only because this is a simulation. The candidate is far ahead there too. Every piece of the track is in these few lines: a monitor that respects late labels, a data check, a logged training run on a chosen window, a gate against the live model, and a promotion that keeps the old version.
In your own words
Why must a pipeline step be safe to run twice, and how do you make it so?
Write a sentence or two, then compare it with a good answer in the lesson.
On the job: an Airflow DAG and a schedule
In production the pipeline is declared to an orchestrator, which runs it on a trigger, retries a failed step and keeps the run log.
Where it comes up in a week
The monthly retrain ran on Sunday and its gate refused the candidate: the data check found half the usual rows. You open the run log, see which step stopped it, and read the upstream job's failure. Nothing was promoted, the champion kept serving, and the retrain reruns safely once the data is fixed, because every step writes to a place named by its inputs.
03Designing AI systemsLatency, cost, routing, fallbacks and guardrails for features built on models.40 min
04Containers and deploymentPackage an app as an image, configure it with the environment, and roll out safely.45 min
05CI/CD pipelinesTest every change automatically, and ship with gates, flags and rollbacks.40 min
06Docker ComposeSeveral containers together: service names, volumes, secrets and start order.40 min
07Kubernetes basicsDesired state, Deployments and Services, labels, probes and rolling updates.40 min
08Cloud for AI engineersAccess policies and least privilege, storage, compute choices, managed model endpoints, secrets and budgets.30 min
09Metrics, dashboards and alert rulesCounters, gauges and histograms, a rate from a counter, a percentile from buckets, a dashboard panel and an alert rule that fires on a sustained condition.40 min
11Incident lab: from the page to the postmortemOne incident run from the page to the all-clear: decide the mitigation, work out the impact in failed requests and error budget, and write a blameless postmortem, a status update and a runbook entry.40 min
13A/B testing a model changeHash assignment, per-user results from raw logs, sample ratio checks, guardrails and a decision rule.40 min
14Columnar files and a local warehouseRows against columns, the encodings that shrink a column, Parquet's row groups and the ranges that let a query skip them, and SQL over files with DuckDB.40 min
15Data quality checks and contractsSix properties of a delivery you can test, each as a count of rows that break it, a contract with the producer, and when to block, quarantine or warn.40 min
16Transformations: staging, models and testsRaw, staging, facts and marts as SQL models run in dependency order in DuckDB, tests as queries that count the rows that break a rule, an incremental load that is safe to rerun, and the same models as dbt files.45 min
17Streams and windowsEvents that never stop arriving: event time against arrival time, tumbling, sliding and session windows, a watermark for late events, and counting each event once.40 min
18Labels and annotator agreementLabels as measurements: agreement and Cohen's kappa, the pair of classes annotators confuse, votes and gold questions, and what wrong labels do to a model and a test score.45 min
19Experiment tracking and reproducible runsA run record of settings, seed, data and code behind every score, a run log in a dozen lines, and comparing settings across seeds.40 min
20Hyperparameter searchWhat a grid costs, random search on a log scale, stopping weak settings early, and why the winner's validation score flatters it.40 min
21Versioning data and modelsFiles stored under the hash of their contents, a manifest that names a whole dataset, and lineage from data and code to each model.35 min
22A model registry and promotionVersions under one name, an alias for the live one, gates a candidate must pass, group by group, and a rollback that moves a pointer.40 min
23Feature pipelines and training-serving skewWhy a model that passes offline fails live: statistics refit at serving, two code paths, and features from the future, with the fix for each.40 min
24Batch and online inferencePredict ahead of time or on request: when the inputs exist, latency near a server's limit, batching what waits, cost, and a hybrid with a freshness limit.40 min
25Monitoring a model: drift and decayWhy a model worsens with no code change: data drift and concept drift, a measure of how far inputs have moved, thresholds set from noise, and labels that arrive late.45 min
26Retraining and pipelinesWhen to retrain and on which weeks, training as steps in dependency order, a cache that reruns only what changed, and guards before and after.45 min
27Designing ML systemsOne method for an ML design, goal and metric to retraining, with the sums that size it: requests a second at peak, a year of feature storage, a label delay and a latency budget.45 min
28Datasheets: documenting a datasetThe document that travels with a dataset: a profile by column and by group, claims a program can check, splits that don't leak, a scan for personal data, and the license and limits only people can write.45 min
29Curation, active learning and synthetic dataThree levers on a training set: removing copies and broken rows, letting a model choose which rows to label, and manufacturing rows, which helps only when their source knows more than you do.45 min
30Measuring fairnessBreak a model's results down by group: selection rate, true and false positive rates and precision, why they can't all be equal, what each fix costs, and when a group is too small to judge.40 min
31Privacy engineeringWhy removing names isn't anonymity, how to hold less (minimization, retention, keyed pseudonyms), the idea behind differential privacy and its budget, what a model remembers, and a deletion that reaches every store.40 min
32A red-team labAttack a simulated assistant methodically: eight kinds of attack run as a suite, success rates beside ordinary requests blocked, fresh attacks to test the fixes, where a defense belongs, and findings kept as release tests.45 min
33What regulation asks of an engineerRisk tiers that follow a system's use, the requirements for high-risk systems as artifacts an engineer builds, evidence tied to a model version, a model card a program can check, and legal dates kept as data.35 min
34Git for teamsBranches, pull requests, rebasing and how a three-way merge finds conflicts.40 min
35Writing for engineersREADMEs, design docs, commit messages and bug reports people can act on.35 min
36Scoping and estimating workA vague request cut into tickets with acceptance criteria and a first shippable slice, estimates as ranges, an honest update for a slip, and a ticket linked to its pull request.30 min
37Roles, résumé and the searchWhat AI, ML and data engineering postings ask for, résumé bullets that carry a number, a 100-word project summary, a take-home as a portfolio piece and how to read an offer.35 min
43Capstone: take a model to productionA churn model through a data contract, a tracked run, a registry gate by slice, hybrid serving, a PSI monitor and one retrain, shipped with a datasheet, a model card, a fairness audit, a design doc and a runbook.45 min
Take a model from a notebook to a system that stays right: track and reproduce every run, search settings honestly, version data and models, promote through gates, keep training and serving in step, choose batch or online, catch drift, and retrain through a pipeline. Every number is measured by the lesson's own code.
See the course →
Never trained an AI? Teach a small one to write, right in your browser. It takes about 20 seconds and needs no account. Teach it to write