Versioning data and models gave every model a version that can never change. That raises a practical question: when the fraud service starts up at 3 a.m., which of forty immutable versions does it load? Somebody has to decide that version 12 is live, somebody has to be able to change it to 13, and when 13 misbehaves somebody has to get 12 back within a minute. A model registry is the piece that holds those decisions. This lesson builds one and, more important, the rules that decide when a new model is allowed to replace the one in production.
What you'll be able to do
Keep every version of a model under one name, with an alias that says which version is live
Write the gates a candidate must pass before it replaces the live model, including a check on each group of users
Promote and roll back by moving an alias, and keep a history of every move
Explain why a service asks the registry for a model by alias and never by file path
Quick check
A guess before you start. A new model scores 0.861 on the team's evaluation set. The model in production scores 0.843. Is that enough to replace it?
Yes, as long as the new model is also larger
Not yet: an average can hide a group that got worse
No: a model in production should never be replaced
Yes: a higher score on the same evaluation set settles it
Show the answer
Not yet: an average can hide a group that got worse
One overall number hides who gained and who lost. This lesson trains exactly this pair of models: the new one is better on average and clearly worse for one in five users. A promotion rule has to look at the groups, not only the total.
The idea: names, versions, aliases and gates
A registry is a small database, and a promotion gate is a rule over numbers. It leans on Versioning data and models for the hash of each version and on Describing data for scores by group.
A name, its versions, and a pointer
A model registry is a small database with three ideas in it.
A registered model is a name for a job, such as churn or fraud. It outlives any one set of weights.
Under the name sit numbered versions: 1, 2, 3. Each is one immutable model file with its lineage record.
An alias is a label that points at one version and can be moved. The usual one is champion: the version that is live.
It runs right here in your browser, with nothing to install.
The promotion was one assignment. No file was copied, renamed or overwritten: versions 1, 2 and 3 are all still there, and only the pointer moved.
The service never names a file. It asks the registry for "the champion of churn" when it starts, and loads whatever version that is today. That one indirection is what makes the rest of this lesson possible. A service that loads models/churn_v2_final.pkl can only be changed by editing and redeploying the service. A service that loads churn@champion is changed by moving a pointer.
MLflow's registry, the best-known open-source one, has the same three parts under the same names: registered models, versions and aliases. The registries the cloud platforms sell are built on the same ideas, under their own names.
The champion pointed at version 4 (0.86) when before was read. Then a challenger alias was added for version 5, and the champion was moved to version 3 (0.84). Moving aliases removes nothing, so all three versions remain.
Try it now
Pick a case and step through the gates. A refusal never touches the alias.
Promotion is a decision with rules
Moving the pointer is easy. Deciding whether to move it is the engineering. A promotion gate is a rule a candidate must pass before it can become champion, written down before anyone sees the candidate's numbers.
Here is a real case. The champion was trained last quarter. The candidate is a bigger model trained on a newer extract of the data. Nobody has noticed yet that a pipeline bug dropped most of the rows from region B out of that extract. Both are scored on the same fixed evaluation set: 1,000 rows that neither trained on, kept as a versioned dataset so that every candidate is measured against the same yardstick.
import numpy as np
from sklearn.ensemble import GradientBoostingClassifier
defmake(n, seed):
rng = np.random.default_rng(seed)
# one customer in five is in region B
region_b = rng.random(n) < 0.2
X = rng.normal(size=(n, 4))
y = np.where(region_b, X[:, 2] - X[:, 0] > 0,
X[:, 0] + X[:, 1] ** 2 > 1).astype(int)
y = np.where(rng.random(n) < 0.05, 1 - y, y) # a little label noisereturn np.column_stack([X, region_b.astype(float)]), y, region_b
X_old, y_old, b_old = make(1500, 1)
X_new, y_new, b_new = make(3000, 2)
# the bug: most region-B rows vanish
lost = b_new & (np.random.default_rng(3).random(3000) < 0.95)
X_new, y_new = X_new[~lost], y_new[~lost]
X_eval, y_eval, b_eval = make(1000, 9) # the fixed evaluation set
champion = GradientBoostingClassifier(n_estimators=50, max_depth=2,
random_state=0).fit(X_old, y_old)
candidate = GradientBoostingClassifier(n_estimators=80, max_depth=3,
random_state=0).fit(X_new, y_new)
for name, model in (("champion ", champion), ("candidate", candidate)):
print(name, "| all", round(float(model.score(X_eval, y_eval)), 3),
"| region A", round(float(model.score(X_eval[~b_eval], y_eval[~b_eval])), 3),
"| region B", round(float(model.score(X_eval[b_eval], y_eval[b_eval])), 3))
On the whole evaluation set the candidate wins, 0.861 to 0.843. It is better for the four customers in five who live in region A. For region B it falls from 0.711 to 0.636, not far above a coin flip. A rule that looked only at the first column would have shipped a model that fails a fifth of the customers, and the overall metric would have gone up while it happened.
A group of rows you score separately is called a slice. So a usable set of gates has at least three parts:
The record is complete. The candidate names its data version and code commit. No lineage, no promotion.
It beats the champion on the fixed evaluation set, by more than the noise you measured in the experiment tracking lesson.
No slice gets much worse. Every group you care about is scored separately, and a drop beyond a set limit blocks the promotion however good the average looks.
Teams add guardrails on top: response time, memory, cost per prediction. The principle is the same one you saw for A/B tests. The rules are fixed first, and a breached rule blocks the launch.
Quick check
A gate compares the candidate with the champion slice by slice, and blocks any slice that drops by more than 0.02. What does this print?
champion = {"all": 0.843, "region A": 0.873, "region B": 0.711}
candidate = {"all": 0.861, "region A": 0.913, "region B": 0.636}
blocked = [name for name in champion if champion[name] - candidate[name] > 0.02]
print(blocked, round(candidate["all"] - champion["all"], 3))
['region B'] 0.018
['region B'] -0.018
['all', 'region A'] 0.018
[] 0.018
Show the answer
['region B'] 0.018
The drop is the champion's score minus the candidate's. For region B that's 0.711 - 0.636 = 0.075, over the limit, so it's blocked. The other two rose, so their drops are negative. The overall gain is 0.861 - 0.843 = 0.018.
In your own words
In your own words: what is the difference between a model version and an alias?
Write a sentence or two, then compare it with a good answer in the lesson.
In code: a registry, a gate and a rollback
The registry here is a few dictionaries and a list. The ideas are the same as MLflow's, without the server.
Drill
Resolve a version number or an alias
Write resolve(registry, name, ref). ref is either a version number (an int) or an alias (a str). Return a tuple (version_number, record). Raise KeyError for a model name, an alias or a version number the registry doesn't have. An alias that points at a version which doesn't exist is a broken registry: raise KeyError for that too.
defresolve(registry, name, ref):
# an alias is looked up first; a number is used as it isreturn0, {}
registry = {"churn": {"versions": {1: {"accuracy": 0.81}, 2: {"accuracy": 0.84}},
"aliases": {"champion": 2}}}
print(resolve(registry, "churn", "champion"))
print(resolve(registry, "churn", 1))
In the lesson, your code is checked as you write it.
Quick check
Why must every candidate be scored on the same fixed, versioned evaluation set?
So a change in score comes from the model, not the rows
So the evaluation finishes faster each time
So the candidate can be trained on those rows as well
So the champion's score never has to be measured
Show the answer
So a change in score comes from the model, not the rows
Two models scored on different rows can't be compared: an easier set makes any model look better. Pinning the evaluation set, by its dataset version, leaves the model as the only thing that changed.
Drill
A promotion gate
Write gate(candidate, champion, min_gain, max_drop). Each model is a dictionary with "data", "code" and "scores", where "scores" maps a slice name to accuracy and always includes "all". Return a list of the reasons the candidate fails, in this order, or an empty list if it passes:
"no lineage" if the candidate's "data" or "code" is missing or empty.
"gain too small" if its "all" score isn't at least min_gain above the champion's.
"<slice> dropped" for each slice, in the champion's order, where the candidate is more than max_drop below the champion. A slice the candidate has no score for counts as dropped.
In the lesson, your code is checked as you write it.
Drawn as grouped bars, the same numbers show where the gate looks:
import matplotlib.pyplot as plt
names = ["all", "region A", "region B"]
champion = [0.843, 0.873, 0.711]
candidate = [0.861, 0.913, 0.636]
xs = range(3)
plt.bar([x - 0.2for x in xs], champion, width=0.4, label="champion")
plt.bar([x + 0.2for x in xs], candidate, width=0.4, label="candidate")
plt.xticks(xs, names)
plt.ylabel("accuracy")
plt.ylim(0.5, 1)
plt.legend()
plt.show()
Rolling back, and the paper trail
Sooner or later a model that passed every gate will misbehave in production, because the evaluation set can't contain everything the world does. What matters then is how fast you can get the old one back.
With a registry, a rollback is the promotion in reverse: point champion at the version it pointed at before. For that the registry has to remember, so every change of an alias is appended to a history: which alias, from which version to which, when, and why.
classRegistry:
def__init__(self):
self.versions, self.aliases, self.history = {}, {}, []
defregister(self, record):
number = len(self.versions) + 1self.versions[number] = record
return number
defset_alias(self, alias, version, when, note):
if version notinself.versions:
raiseKeyError(version)
self.history.append({"alias": alias, "from": self.aliases.get(alias),
"to": version, "when": when, "note": note})
self.aliases[alias] = version
defrollback(self, alias, when, note):
last = [h for h inself.history if h["alias"] == alias][-1]
self.set_alias(alias, last["from"], when, note)
churn = Registry()
for accuracy in (0.81, 0.84, 0.86):
churn.register({"accuracy": accuracy})
churn.set_alias("champion", 2, "2026-03-01", "first launch")
churn.set_alias("champion", 3, "2026-06-10", "passed the gates")
churn.rollback("champion", "2026-06-11", "complaints from region B")
print("champion is version", churn.aliases["champion"])
for h in churn.history:
print(h["when"], h["alias"], h["from"], "->", h["to"], "|", h["note"])
The rollback took one call, and it is itself a line in the history. Nothing is ever removed from that list. Six months later, when someone asks which model was answering customers on June 10, the history says version 3, from 2026-06-10 until the next day, and the version's record says exactly what version 3 was.
Go deeper Trying a candidate on real traffic first
Gates on an evaluation set are the first filter, not the last. Before the champion alias moves, teams often give the candidate its own alias, usually challenger, and run it in one of two ways. In shadow mode it receives a copy of real requests and its answers are logged but never shown to anyone. In a canary release it answers a small share of real traffic while its metrics are watched. Both are the ideas from the CI/CD and A/B testing lessons, applied to a model, and both rely on the registry being able to hold two live pointers at once.
Quick check
From the versioning lesson: why can a rollback be trusted to bring back exactly the model that was live before?
Rollbacks copy the new model's weights into the old version
The old model's file is the most recently modified one on disk
The registry retrains the old model from its record during the rollback
A version is stored under the hash of its contents, so it can't have changed
Show the answer
A version is stored under the hash of its contents, so it can't have changed
Versions are immutable: the file is named by its hash, and different contents would have a different name. Pointing the alias back at version 2 gives the same bytes that were version 2 last month.
Quick check
A registry's history for the champion alias reads: none → 2, then 2 → 3, then 3 → 2, then 2 → 4. Which version is live, and how many times has version 2 been champion?
Version 3 is live, and version 2 was champion twice
Version 4 is live, and version 2 was champion twice
Version 2 is live, and it was champion twice
Version 4 is live, and version 2 was champion once
Show the answer
Version 4 is live, and version 2 was champion twice
The last line moved the alias to version 4. Version 2 held the alias after the first move and again after the rollback from 3, so twice. The history keeps both periods.
Worked example
Worked example: a promotion that is refused, and one that goes through
Put the registry and the gates together. Two candidates arrive for the champion: the one trained on the damaged extract, and a later one trained the same way after the pipeline was fixed. Every score below is measured on the fixed evaluation set.
Version 3 is refused, with the reason in words anyone can act on: region B dropped. That message points straight at the data, and a manifest diff between the two dataset versions would show the missing rows. Version 4 is the same recipe trained on the repaired extract. With region B's rows back it scores 0.894 overall and 0.882 in region B, clears every gate and takes the alias, and the history gains one line. Notice that version 4 was judged against version 2, the live model, because the refused version 3 never became champion.
In your own words
Why is "the candidate scores higher than the champion" not a sufficient rule for promotion?
Write a sentence or two, then compare it with a good answer in the lesson.
On the job: MLflow's registry, shadow and canary
In production the registry is a service, and a promotion is a reviewed change with a rollback you have already tried.
Where it comes up in a week
A candidate passes its gates on Tuesday. Before the alias moves, you give it the challenger alias and run it in shadow for a day: it gets a copy of real requests, its answers are logged and never shown, and you compare them with the champion's slice by slice. If that is clean, a canary answers 5% of real traffic while the error rate and response time are watched. Only then does champion move. The rollback was tried the week before, so it is one command at 2 a.m., not a research project.
03Designing AI systemsLatency, cost, routing, fallbacks and guardrails for features built on models.40 min
04Containers and deploymentPackage an app as an image, configure it with the environment, and roll out safely.45 min
05CI/CD pipelinesTest every change automatically, and ship with gates, flags and rollbacks.40 min
06Docker ComposeSeveral containers together: service names, volumes, secrets and start order.40 min
07Kubernetes basicsDesired state, Deployments and Services, labels, probes and rolling updates.40 min
08Cloud for AI engineersAccess policies and least privilege, storage, compute choices, managed model endpoints, secrets and budgets.30 min
09Metrics, dashboards and alert rulesCounters, gauges and histograms, a rate from a counter, a percentile from buckets, a dashboard panel and an alert rule that fires on a sustained condition.40 min
11Incident lab: from the page to the postmortemOne incident run from the page to the all-clear: decide the mitigation, work out the impact in failed requests and error budget, and write a blameless postmortem, a status update and a runbook entry.40 min
13A/B testing a model changeHash assignment, per-user results from raw logs, sample ratio checks, guardrails and a decision rule.40 min
14Columnar files and a local warehouseRows against columns, the encodings that shrink a column, Parquet's row groups and the ranges that let a query skip them, and SQL over files with DuckDB.40 min
15Data quality checks and contractsSix properties of a delivery you can test, each as a count of rows that break it, a contract with the producer, and when to block, quarantine or warn.40 min
16Transformations: staging, models and testsRaw, staging, facts and marts as SQL models run in dependency order in DuckDB, tests as queries that count the rows that break a rule, an incremental load that is safe to rerun, and the same models as dbt files.45 min
17Streams and windowsEvents that never stop arriving: event time against arrival time, tumbling, sliding and session windows, a watermark for late events, and counting each event once.40 min
18Labels and annotator agreementLabels as measurements: agreement and Cohen's kappa, the pair of classes annotators confuse, votes and gold questions, and what wrong labels do to a model and a test score.45 min
19Experiment tracking and reproducible runsA run record of settings, seed, data and code behind every score, a run log in a dozen lines, and comparing settings across seeds.40 min
20Hyperparameter searchWhat a grid costs, random search on a log scale, stopping weak settings early, and why the winner's validation score flatters it.40 min
21Versioning data and modelsFiles stored under the hash of their contents, a manifest that names a whole dataset, and lineage from data and code to each model.35 min
22A model registry and promotionVersions under one name, an alias for the live one, gates a candidate must pass, group by group, and a rollback that moves a pointer.40 min
23Feature pipelines and training-serving skewWhy a model that passes offline fails live: statistics refit at serving, two code paths, and features from the future, with the fix for each.40 min
24Batch and online inferencePredict ahead of time or on request: when the inputs exist, latency near a server's limit, batching what waits, cost, and a hybrid with a freshness limit.40 min
25Monitoring a model: drift and decayWhy a model worsens with no code change: data drift and concept drift, a measure of how far inputs have moved, thresholds set from noise, and labels that arrive late.45 min
26Retraining and pipelinesWhen to retrain and on which weeks, training as steps in dependency order, a cache that reruns only what changed, and guards before and after.45 min
27Designing ML systemsOne method for an ML design, goal and metric to retraining, with the sums that size it: requests a second at peak, a year of feature storage, a label delay and a latency budget.45 min
28Datasheets: documenting a datasetThe document that travels with a dataset: a profile by column and by group, claims a program can check, splits that don't leak, a scan for personal data, and the license and limits only people can write.45 min
29Curation, active learning and synthetic dataThree levers on a training set: removing copies and broken rows, letting a model choose which rows to label, and manufacturing rows, which helps only when their source knows more than you do.45 min
30Measuring fairnessBreak a model's results down by group: selection rate, true and false positive rates and precision, why they can't all be equal, what each fix costs, and when a group is too small to judge.40 min
31Privacy engineeringWhy removing names isn't anonymity, how to hold less (minimization, retention, keyed pseudonyms), the idea behind differential privacy and its budget, what a model remembers, and a deletion that reaches every store.40 min
32A red-team labAttack a simulated assistant methodically: eight kinds of attack run as a suite, success rates beside ordinary requests blocked, fresh attacks to test the fixes, where a defense belongs, and findings kept as release tests.45 min
33What regulation asks of an engineerRisk tiers that follow a system's use, the requirements for high-risk systems as artifacts an engineer builds, evidence tied to a model version, a model card a program can check, and legal dates kept as data.35 min
34Git for teamsBranches, pull requests, rebasing and how a three-way merge finds conflicts.40 min
35Writing for engineersREADMEs, design docs, commit messages and bug reports people can act on.35 min
36Scoping and estimating workA vague request cut into tickets with acceptance criteria and a first shippable slice, estimates as ranges, an honest update for a slip, and a ticket linked to its pull request.30 min
37Roles, résumé and the searchWhat AI, ML and data engineering postings ask for, résumé bullets that carry a number, a 100-word project summary, a take-home as a portfolio piece and how to read an offer.35 min
43Capstone: take a model to productionA churn model through a data contract, a tracked run, a registry gate by slice, hybrid serving, a PSI monitor and one retrain, shipped with a datasheet, a model card, a fairness audit, a design doc and a runbook.45 min
Take a model from a notebook to a system that stays right: track and reproduce every run, search settings honestly, version data and models, promote through gates, keep training and serving in step, choose batch or online, catch drift, and retrain through a pipeline. Every number is measured by the lesson's own code.
See the course →
Never trained an AI? Teach a small one to write, right in your browser. It takes about 20 seconds and needs no account. Teach it to write