Why a model worsens with no code change: data drift and concept drift, a measure of how far inputs have moved, thresholds set from noise, and labels that arrive late.
Ordinary software keeps working until someone changes it. A model can get worse while every file stays the same, because it was fitted to a world that has since moved: customers change, prices change, fraudsters adapt. Nothing crashes and no test fails. The predictions just become less right, a little each week, until a person notices the business numbers. Monitoring is how you notice first. This lesson measures three kinds of change on a real model, including one that the usual monitor is blind to and one that looks alarming and does no harm.
What you'll be able to do
Tell data drift from concept drift, and say which of them a monitor on the inputs can and can't see
Measure how far a feature has moved from its training distribution with the population stability index
Set a drift threshold from the noise in your own window size, not from a rule of thumb alone
Monitor accuracy when labels arrive late, and say how far behind that monitor runs
Quick check
A guess before you start. A loan model was trained last year. Since then, the incomes and debts of applicants look exactly as they did in training: every chart of the inputs is unchanged. Can the model's accuracy have fallen?
No: a model fed the same kind of inputs gives the same quality of answers
Yes, but only if the server it runs on has become slower
Yes: what the inputs mean for repayment can change while the inputs stay put
No: accuracy can only fall when the code or the weights are changed
Show the answer
Yes: what the inputs mean for repayment can change while the inputs stay put
A model learns a relationship between inputs and outcomes. If the relationship shifts, say a downturn makes the same income a weaker sign of repayment, the same inputs deserve different answers. This lesson simulates exactly that: the input charts stay flat and accuracy falls from 0.79 to 0.54.
A model is trained on rows that pair inputs with outcomes. In production, two different things can change.
Data drift is a change in the inputs: who is applying, what they look like, how the values are spread. The rule that connects inputs to outcomes is the same, but the model is now being asked about cases unlike the ones it learned from.
Concept drift is a change in the rule itself. The same applicant, with the same income and debt, now has a different chance of repaying.
The model in this lesson decides loan applications from two inputs, income and debt, in thousands. It is trained once and never touched again. loans makes a week of applications and can move the world in three ways.
Setup (runs before every cell) · 26 lines
The lesson runs this once, before its first cell, and every cell, drill and example starts with what it defines. A cell that changes a setup object changes it for the cells after it, until the page reloads.
import numpy as np
from sklearn.linear_model import LogisticRegression
defloans(n, seed, income_up=0.0, debt_down=0.0, rule_change=0.0):
"""A batch of applications: income and debt in thousands,
and whether each loan was repaid."""
rng = np.random.default_rng(seed)
income = rng.normal(50 + income_up, 12, size=n)
debt = rng.normal(20 - debt_down, 6, size=n)
earn = income - 50
signal = (0.08 * earn - 0.006 * np.maximum(earn - 10, 0) ** 2
- 0.15 * (debt - 20) - rule_change * 0.16 * earn)
return (np.column_stack([income, debt]),
(signal + rng.normal(0, 1, n) > 0).astype(int))
defshares(values, edges):
counts = np.bincount(np.searchsorted(edges, values, side="right"),
minlength=len(edges) + 1)
return counts / counts.sum()
defpsi(expected, actual):
e, a = np.maximum(expected, 1e-4), np.maximum(actual, 1e-4)
returnfloat(np.sum((a - e) * np.log(a / e)))
X_train, y_train = loans(4000, 0)
model = LogisticRegression().fit(X_train, y_train)
X_train, y_train = loans(4000, 0)
model = LogisticRegression().fit(X_train, y_train)
X_test, y_test = loans(2000, 1)
print("at launch | accuracy",
round(float(model.score(X_test, y_test)), 3))
for name, change in (("incomes rise by 12 ", {"income_up": 12}),
("debts fall by 6 ", {"debt_down": 6}),
("the rule changes ", {"rule_change": 1.0})):
X, y = loans(1000, 7, **change)
print(name, "| accuracy", round(float(model.score(X, y)), 3))
Three changes, three different results.
Incomes rise and accuracy slips from 0.789 to 0.746. In this world, repayment stops improving at high incomes. The training data held few such applicants, so the model never learned that, and it approves them too readily. This is data drift that hurts.
Debts fall and accuracy goes up, to 0.808. The inputs moved just as far, but into territory where what the model learned still holds, and low-debt cases are easy ones. This is data drift that does no harm.
The rule changes, so income no longer predicts repayment the way it did, and accuracy collapses to 0.544, barely better than a coin. The inputs didn't move at all. This is concept drift.
Hold on to the second line. A change in the inputs is a reason to look, not proof of damage.
Quick check
A demand-forecasting model was trained before a new public holiday was created. On that day, shops see twice the usual sales for the same weather, weekday and prices. What is this?
Concept drift: the same inputs now lead to a different outcome
A broken pipeline: a feature is arriving with the wrong values
Nothing: one unusual day can't be told apart from noise
Data drift: the inputs have moved away from the training data
Show the answer
Concept drift: the same inputs now lead to a different outcome
The weather, weekday and prices are ones the model has seen many times. What changed is what they lead to: on this date, twice the sales. The relationship between inputs and outcome moved, which is concept drift. The remedy is a new input, a holiday flag, and retraining.
Measuring how far a feature has moved
To monitor the inputs you need one number per feature that says how different this week's values are from training. The most common is the population stability index, or PSI. It works on shares:
Cut the feature's training values into ten bins holding a tenth of the rows each. The bin edges are fixed from then on.
For a new window of data, find the share of rows that falls in each of those same bins.
Compare the two lists of shares.
PSI=i∑(ai−ei)×lneiai
In plain words: for each bin, ei is the share of training rows in it (expected) and ai is the share of this window's rows (actual). Take the difference of the shares, multiply by the natural log of their ratio, and add up the bins. A bin whose share didn't change adds nothing. A bin that gained or lost rows adds a positive amount either way.
If you remember KL divergence from the cross-entropy lesson, this is the same idea applied in both directions and added together: a distance between two distributions that is 0 when they match.
Uses the setup above, 26 lines.
import numpy as np
defpsi(expected, actual):
# a tiny floor, so an empty bin can't divide by zero
e, a = np.maximum(expected, 1e-4), np.maximum(actual, 1e-4)
returnfloat(np.sum((a - e) * np.log(a / e)))
training = np.array([0.25, 0.25, 0.25, 0.25])
print("no change: ", round(psi(training, np.array([0.25, 0.25, 0.25, 0.25])), 3))
print("small change:", round(psi(training, np.array([0.22, 0.25, 0.25, 0.28])), 3))
large = np.array([0.05, 0.15, 0.30, 0.50])
print("large change:", round(psi(training, large), 3))
print("each bin adds:", np.round((large - training) * np.log(large / training), 3))
It runs right here in your browser, with nothing to install.
Now watch it on the loan model while incomes creep up by 1.5 thousand a week:
Uses the setup above, 26 lines.
# nine edges make ten bins of training income
edges = np.quantile(X_train[:, 0], np.linspace(0.1, 0.9, 9))
base = shares(X_train[:, 0], edges)
for week inrange(1, 9):
X, y = loans(1000, 100 + week, income_up=1.5 * week)
print("week", week, "| PSI of income", round(psi(base, shares(X[:, 0], edges)), 2),
"| share approved", round(float(model.predict(X).mean()), 2),
"| accuracy", round(float(model.score(X, y)), 3))
The PSI climbs every week, from 0.02 to 0.95. The share of applications the model approves climbs with it, from 0.48 to 0.73, and accuracy drifts down from 0.789 to 0.731.
Look at which columns you could actually have in production. The first two need no labels: they are computed from the requests and the model's own answers, the same day. The third needs to know who repaid, which nobody will know for months. That is why drift monitors exist: they are the part of the picture you can see today.
Try it now
Step through three frames: the expected shares, the actual shares, and what each bin adds to the PSI.
Identical shares give 0. Moving 0.3 of the rows from one bin to the other gives 0.3 × ln(1.6) + 0.3 × ln(2.5) ≈ 0.416, and it gives the same value whichever bin gained: every bin's term is positive, so PSI measures how far the shares moved, not in which direction.
In your own words
What is the difference between data drift and concept drift, and what does a PSI of 0 mean?
Write a sentence or two, then compare it with a good answer in the lesson.
In code: PSI by week, noise, patience and late labels
The monitor is a few lines on top of the model you already have. The cells use the same loans world, set up once above them, and they show what the input monitor sees, what it misses and how late the truth arrives.
Drill
The population stability index
Write psi(expected, actual) for two lists of shares of equal length. For each bin add (a - e) * math.log(a / e). Before using a share, raise it to at least 0.0001, so an empty bin can't cause a division by zero or a log of zero. Return the total rounded to 4 places.
In the lesson, your code is checked as you write it.
What an input monitor can't see
Run the same monitor on all three changes from the start of the lesson, with a PSI for each feature:
Uses the setup above, 26 lines.
edges = [np.quantile(X_train[:, k], np.linspace(0.1, 0.9, 9)) for k in (0, 1)]
base = [shares(X_train[:, k], edges[k]) for k in (0, 1)]
for name, change in (("incomes rise by 12", {"income_up": 12}),
("debts fall by 6 ", {"debt_down": 6}),
("the rule changes ", {"rule_change": 1.0})):
X, y = loans(1000, 7, **change)
print(name, "| PSI income", round(psi(base[0], shares(X[:, 0], edges[0])), 2),
"| PSI debt", round(psi(base[1], shares(X[:, 1], edges[1])), 2),
"| accuracy", round(float(model.score(X, y)), 3))
Read it as a table of what the alarm said against what was true.
Change
The input monitor says
What happened to accuracy
Incomes rise
Income has moved a long way: 0.92
Down, to 0.746
Debts fall
Debt has moved a long way: 0.94
Up, to 0.808
The rule changes
Nothing has moved: 0.02 and 0.01
Down to 0.544
The first row is the monitor working. The second is a false alarm: the same size of PSI, and the model is fine. The third is the dangerous one. The worst failure of the three produced no signal at all, because concept drift leaves the inputs alone.
So an input monitor is a smoke detector. When it sounds, someone should look. When it's quiet, that proves only that the inputs haven't moved. The only thing that can see concept drift is the outcome: comparing what the model predicted with what really happened. And outcomes are usually late.
Quick check
From the feature pipelines lesson: to compare this week's feature values with the training values, which values should the monitor read?
The training table itself, sampled again with a new random seed
The features the service logged as it actually fed them to the model
The raw request fields, before any feature code has run on them
The features recomputed later from the warehouse by the training code
Show the answer
The features the service logged as it actually fed them to the model
The logged features are the only record of what the model really saw. Recomputing them with the training code would hide any difference between the two paths, which is one of the things the monitor is there to catch.
When is a PSI big enough to act on?
A widely repeated rule of thumb says a PSI under 0.1 means no real change, 0.1 to 0.25 a moderate one, and over 0.25 a large one. It comes from credit scoring, where windows hold many thousands of applicants. Before borrowing it, measure what PSI you get when nothing has changed: draw fresh windows from the training distribution itself, many times, at your own window size.
Uses the setup above, 26 lines.
import numpy as np
rng = np.random.default_rng(0)
train = rng.normal(50, 12, size=4000)
edges = np.quantile(train, np.linspace(0.1, 0.9, 9))
defshares(values):
counts = np.bincount(np.searchsorted(edges, values, side="right"), minlength=10)
return counts / counts.sum()
defpsi(expected, actual):
e, a = np.maximum(expected, 1e-4), np.maximum(actual, 1e-4)
returnfloat(np.sum((a - e) * np.log(a / e)))
base = shares(train)
for n in (100, 300, 1000, 3000):
values = [psi(base, shares(rng.normal(50, 12, size=n))) for trial inrange(300)]
print(n, "rows in a window | typical PSI", round(float(np.median(values)), 3),
"| 1 window in 20 is above", round(float(np.quantile(values, 0.95)), 3))
With a hundred rows in a window, the typical PSI is 0.092 when the distribution hasn't changed at all, and one window in twenty is above 0.175. A threshold of 0.1 would fire on nearly half of those windows. With three thousand rows the typical value is 0.006, and 0.1 is a strong signal.
The rule that follows: set the threshold above your own noise, from a run like this one, and prefer larger windows to smaller ones. A second defense is patience: alert only when the value has stayed above the threshold for several windows in a row, since noise seldom repeats and real drift usually persists.
Quick check
A monitor alerts when a daily value has been above the threshold for 3 days in a row. What does this print?
values = [0.05, 0.30, 0.08, 0.22, 0.26, 0.31, 0.29]
threshold, patience = 0.2, 3
run, fired = 0, []
for day, value inenumerate(values):
run = run + 1if value > threshold else0if run == patience:
fired.append(day)
print(fired)
[3]
[5]
[5, 6]
[1, 3, 4, 5, 6]
Show the answer
[5]
Day 1 is above the threshold but day 2 falls back, so the run resets. Days 3, 4 and 5 are all above it, and the third of them, day 5, fires the alert. Day 6 extends the run to 4, which is not equal to 3, so the same episode doesn't fire twice.
Drill
Alerts that wait for the signal to persist
Write alerts(values, threshold, patience). Return the list of indexes at which an alert fires. An alert fires at the moment a value has been above the threshold for patience windows in a row. It doesn't fire again while the values stay above. Once a value drops to the threshold or below, the count starts over, and a later run can fire a new alert.
Uses the setup above, 26 lines.
defalerts(values, threshold, patience):
return [i for i, v inenumerate(values) if v > threshold]
print(alerts([0.05, 0.30, 0.08, 0.22, 0.26, 0.31, 0.29], 0.2, 3))
In the lesson, your code is checked as you write it.
Worked example
Plot PSI, approvals and accuracy by week
The weekly cell above, as three lines instead of eight rows. Incomes creep up by 1.5 a week.
Uses the setup above, 26 lines.
import matplotlib.pyplot as plt
edges = np.quantile(X_train[:, 0], np.linspace(0.1, 0.9, 9))
base = shares(X_train[:, 0], edges)
weeks = list(range(1, 9))
values = {"PSI of income": [], "share approved": [], "accuracy": []}
for week in weeks:
X, y = loans(1000, 100 + week, income_up=1.5 * week)
values["PSI of income"].append(psi(base, shares(X[:, 0], edges)))
values["share approved"].append(float(model.predict(X).mean()))
values["accuracy"].append(float(model.score(X, y)))
print({k: round(v[-1], 2) for k, v in values.items()})
for name, series in values.items():
plt.plot(weeks, series, marker="o", label=name)
plt.xlabel("week")
plt.legend()
plt.show()
In the lesson you can change this and run it.
By week 8 the PSI is 0.95, the model approves 0.73 of applications and accuracy has slid to 0.73. The first line rises quickly and visibly, and the accuracy line moves less: the input monitor shouts early, but accuracy is the line that says whether it matters.
Labels arrive late
The honest measure of a model is its accuracy on what really happened. In a notebook the labels are in the next column. In production they come from the future. A loan's outcome is known months after the decision. A churn label takes thirty days by definition. So an accuracy monitor always describes the past, and the question is how far back.
Here the rule starts changing on day 20, and each loan's outcome is known 30 days after the decision.
Uses the setup above, 26 lines.
# one entry per day: (day, predictions that were right, predictions made)
log = []
for day inrange(80):
X, y = loans(300, 1000 + day, rule_change=min(1.0, max(0, day - 20) / 30))
log.append((day, int((model.predict(X) == y).sum()), len(y)))
defaccuracy(days):
right = sum(r for d, r, n in log if d in days)
total = sum(n for d, r, n in log if d in days)
returnround(right / total, 3)
delay = 30for today in (35, 45, 55, 65, 75):
newest = today - delay # the latest day whose outcomes are knownprint("day", today, "| newest known outcome: day", newest,
"| the monitor shows", accuracy(range(newest - 9, newest + 1)),
"| the truth today", accuracy(range(today - 9, today + 1)))
On day 55 the monitor reads 0.774 and looks healthy. The model is in fact at 0.505, no better than a coin, and has been getting worse for five weeks. The monitor isn't wrong. It is reporting on days 16 to 25, the newest it can know about, and it will be day 85 before it shows what is true today.
Because the inputs never moved, the input monitor is silent through all of it. This is the hardest case in monitoring, and there are only partial answers:
Find an earlier signal of the outcome. A missed first payment arrives long before a default. A customer going quiet arrives before a cancellation. It is a noisier label, but it is weeks sooner.
Label a small sample by hand, every week, where a person can judge the answer.
Watch the business numbers the model drives, which react before the labels do.
Know your blind interval. If outcomes take 30 days, say so on the dashboard: "accuracy as of 30 days ago". A stale number presented as current is how the day-55 reading fools a team.
Quick check
A churn model's labels are known 30 days after each prediction. Today is day 100. The accuracy chart's newest point is 0.91. What does that point describe?
The model's accuracy on the predictions it made today
Predictions made on day 100, checked against day 70's labels
Predictions made on or before day 70, and nothing more recent
The average accuracy over all 100 days so far
Show the answer
Predictions made on or before day 70, and nothing more recent
A prediction can be scored only once its label exists, 30 days later. On day 100 the newest predictions with labels are from day 70. Anything the model has done in the last 30 days is not in the chart yet.
Quick check
The input monitor for a fraud model has been quiet for a month. Which conclusion is justified?
The inputs look like the training data; the model may still be wrong
Monitoring of outcomes can be switched off to save cost
The model is as accurate as it was when it was launched
There has been no fraud of any new kind this month
Show the answer
The inputs look like the training data; the model may still be wrong
A quiet input monitor says the inputs haven't moved. Concept drift, such as fraudsters changing tactics inside ordinary-looking payments, leaves the inputs alone. Only outcomes can show it.
Worked example
Worked example: a weekly monitoring report
Put the pieces in one report: a PSI for each feature against a threshold set from noise, the share of approvals, and accuracy for the newest week whose outcomes are known, labeled with its age.
Uses the setup above, 26 lines.
import numpy as np
from sklearn.linear_model import LogisticRegression
defloans(n, seed, income_up=0.0, debt_down=0.0, rule_change=0.0):
rng = np.random.default_rng(seed)
income = rng.normal(50 + income_up, 12, size=n)
debt = rng.normal(20 - debt_down, 6, size=n)
earn = income - 50
signal = (0.08 * earn - 0.006 * np.maximum(earn - 10, 0) ** 2
- 0.15 * (debt - 20) - rule_change * 0.16 * earn)
return (np.column_stack([income, debt]),
(signal + rng.normal(0, 1, n) > 0).astype(int))
defshares(values, edges):
counts = np.bincount(np.searchsorted(edges, values, side="right"),
minlength=len(edges) + 1)
return counts / counts.sum()
defpsi(expected, actual):
e, a = np.maximum(expected, 1e-4), np.maximum(actual, 1e-4)
returnfloat(np.sum((a - e) * np.log(a / e)))
X_train, y_train = loans(4000, 0)
model = LogisticRegression().fit(X_train, y_train)
edges = [np.quantile(X_train[:, k], np.linspace(0.1, 0.9, 9)) for k in (0, 1)]
base = [shares(X_train[:, k], edges[k]) for k in (0, 1)]
threshold = 0.05# windows of 1,000 rows: noise stays near 0.02
weeks = [loans(1000, 300 + w, income_up=2.0 * max(0, w - 3)) for w inrange(1, 9)]
label_delay = 4# outcomes are known four weeks laterfor w, (X, y) inenumerate(weeks, start=1):
drift = [round(psi(base[k], shares(X[:, k], edges[k])), 2) for k in (0, 1)]
flags = [name for name, value inzip(("income", "debt"), drift)
if value > threshold]
known = w - label_delay
past = ("accuracy week %d: %.3f" % (known, model.score(*weeks[known - 1]))
if known >= 1else"no outcomes yet")
print("week", w, "| PSI", drift,
"| approved", round(float(model.predict(X).mean()), 2),
"|", past, "|", "LOOK AT " + ", ".join(flags) if flags else"quiet")
In the lesson you can change this and run it.
Incomes start rising in week 4. The income PSI crosses the threshold in week 5 and the approval share starts to climb the same week, both visible at once with no labels. The accuracy column can't help yet: it first speaks in week 5, about week 1, and by week 8 it has only reached week 4, the last normal-looking week. A team reading only accuracy would see nothing wrong for another month. A team reading the whole row has a named feature to investigate in week 5.
In your own words
Why can't you copy a drift threshold from a textbook, and why does the accuracy monitor lag?
Write a sentence or two, then compare it with a good answer in the lesson.
On the job: the weekly report and the blind interval
In a week, this lesson is a standing report. It lists the PSI of each important feature, the share of predictions in each class, the label-based accuracy, and a plain line that says how far back the accuracy goes: "accuracy as of 30 days ago". When the PSI alert fires, nobody retrains at once: you look at the model's own predictions, at the business numbers it drives, and at the freshest labels you can get. Only the last of those can say whether the rule changed.
In your own words
You own a model whose labels take six weeks to arrive. What do you put on its monitoring page?
Write a sentence or two, then compare it with a good answer in the lesson.
03Designing AI systemsLatency, cost, routing, fallbacks and guardrails for features built on models.40 min
04Containers and deploymentPackage an app as an image, configure it with the environment, and roll out safely.45 min
05CI/CD pipelinesTest every change automatically, and ship with gates, flags and rollbacks.40 min
06Docker ComposeSeveral containers together: service names, volumes, secrets and start order.40 min
07Kubernetes basicsDesired state, Deployments and Services, labels, probes and rolling updates.40 min
08Cloud for AI engineersAccess policies and least privilege, storage, compute choices, managed model endpoints, secrets and budgets.30 min
09Metrics, dashboards and alert rulesCounters, gauges and histograms, a rate from a counter, a percentile from buckets, a dashboard panel and an alert rule that fires on a sustained condition.40 min
11Incident lab: from the page to the postmortemOne incident run from the page to the all-clear: decide the mitigation, work out the impact in failed requests and error budget, and write a blameless postmortem, a status update and a runbook entry.40 min
13A/B testing a model changeHash assignment, per-user results from raw logs, sample ratio checks, guardrails and a decision rule.40 min
14Columnar files and a local warehouseRows against columns, the encodings that shrink a column, Parquet's row groups and the ranges that let a query skip them, and SQL over files with DuckDB.40 min
15Data quality checks and contractsSix properties of a delivery you can test, each as a count of rows that break it, a contract with the producer, and when to block, quarantine or warn.40 min
16Transformations: staging, models and testsRaw, staging, facts and marts as SQL models run in dependency order in DuckDB, tests as queries that count the rows that break a rule, an incremental load that is safe to rerun, and the same models as dbt files.45 min
17Streams and windowsEvents that never stop arriving: event time against arrival time, tumbling, sliding and session windows, a watermark for late events, and counting each event once.40 min
18Labels and annotator agreementLabels as measurements: agreement and Cohen's kappa, the pair of classes annotators confuse, votes and gold questions, and what wrong labels do to a model and a test score.45 min
19Experiment tracking and reproducible runsA run record of settings, seed, data and code behind every score, a run log in a dozen lines, and comparing settings across seeds.40 min
20Hyperparameter searchWhat a grid costs, random search on a log scale, stopping weak settings early, and why the winner's validation score flatters it.40 min
21Versioning data and modelsFiles stored under the hash of their contents, a manifest that names a whole dataset, and lineage from data and code to each model.35 min
22A model registry and promotionVersions under one name, an alias for the live one, gates a candidate must pass, group by group, and a rollback that moves a pointer.40 min
23Feature pipelines and training-serving skewWhy a model that passes offline fails live: statistics refit at serving, two code paths, and features from the future, with the fix for each.40 min
24Batch and online inferencePredict ahead of time or on request: when the inputs exist, latency near a server's limit, batching what waits, cost, and a hybrid with a freshness limit.40 min
25Monitoring a model: drift and decayWhy a model worsens with no code change: data drift and concept drift, a measure of how far inputs have moved, thresholds set from noise, and labels that arrive late.45 min
26Retraining and pipelinesWhen to retrain and on which weeks, training as steps in dependency order, a cache that reruns only what changed, and guards before and after.45 min
27Designing ML systemsOne method for an ML design, goal and metric to retraining, with the sums that size it: requests a second at peak, a year of feature storage, a label delay and a latency budget.45 min
28Datasheets: documenting a datasetThe document that travels with a dataset: a profile by column and by group, claims a program can check, splits that don't leak, a scan for personal data, and the license and limits only people can write.45 min
29Curation, active learning and synthetic dataThree levers on a training set: removing copies and broken rows, letting a model choose which rows to label, and manufacturing rows, which helps only when their source knows more than you do.45 min
30Measuring fairnessBreak a model's results down by group: selection rate, true and false positive rates and precision, why they can't all be equal, what each fix costs, and when a group is too small to judge.40 min
31Privacy engineeringWhy removing names isn't anonymity, how to hold less (minimization, retention, keyed pseudonyms), the idea behind differential privacy and its budget, what a model remembers, and a deletion that reaches every store.40 min
32A red-team labAttack a simulated assistant methodically: eight kinds of attack run as a suite, success rates beside ordinary requests blocked, fresh attacks to test the fixes, where a defense belongs, and findings kept as release tests.45 min
33What regulation asks of an engineerRisk tiers that follow a system's use, the requirements for high-risk systems as artifacts an engineer builds, evidence tied to a model version, a model card a program can check, and legal dates kept as data.35 min
34Git for teamsBranches, pull requests, rebasing and how a three-way merge finds conflicts.40 min
35Writing for engineersREADMEs, design docs, commit messages and bug reports people can act on.35 min
36Scoping and estimating workA vague request cut into tickets with acceptance criteria and a first shippable slice, estimates as ranges, an honest update for a slip, and a ticket linked to its pull request.30 min
37Roles, résumé and the searchWhat AI, ML and data engineering postings ask for, résumé bullets that carry a number, a 100-word project summary, a take-home as a portfolio piece and how to read an offer.35 min
43Capstone: take a model to productionA churn model through a data contract, a tracked run, a registry gate by slice, hybrid serving, a PSI monitor and one retrain, shipped with a datasheet, a model card, a fairness audit, a design doc and a runbook.45 min
Take a model from a notebook to a system that stays right: track and reproduce every run, search settings honestly, version data and models, promote through gates, keep training and serving in step, choose batch or online, catch drift, and retrain through a pipeline. Every number is measured by the lesson's own code.
See the course →
Never trained an AI? Teach a small one to write, right in your browser. It takes about 20 seconds and needs no account. Teach it to write