Take a machine learning model to production: the decision rules
The small functions that decide whether a model ships and stays shipped: a registry gate by slice, a hybrid serving rule, a drift alert, a retrain rule and a model-card check.
A churn model through a data contract, a tracked run, a registry gate by slice, hybrid serving, a PSI monitor and one retrain, shipped with a datasheet, a model card, a fairness audit, a design doc and a runbook.
Earlier lessons built each part of a production model on its own: a tracked run, a registry, features, serving, monitoring, retraining. This capstone puts them in one line for a churn model, and gives you one thing the parts cannot give you alone: the experience of a damaged model that every simple check passes and one careful gate stops. At the end you hold a working pipeline and the documents a reviewer asks for.
It must not be worse for any group of subscribers that matters
An average can rise while one group falls. The gate scores each slice separately.
A subscription shop wants to know which subscribers will cancel next month, so that its retention team can call them. You will take a churn model through the whole path, in order: a data contract at the edge, a tracked training run, a registry gate that scores by slice, hybrid serving, a PSI monitor, and one retraining decision. It is the path of Experiment tracking, Versioning data and models, A model registry, Batch and online inference, Monitoring a model and Retraining and pipelines, in one place, with the checks of Data quality checks and contracts at the front.
The data is made up so the truth is known. Subscribers have a tenure, a monthly spend and a count of support tickets. New subscribers (under 9 months) churn by a different rule from long-standing ones, and they are the slice that matters most. The setup has the helpers; each cell shows only what it adds.
The picture follows two versions of the model through the system. Step through both.
Pick a version and step through the system. The contract catches broken rows; only the gate catches a model that learned the wrong thing.
Every candidate is measured against the same yardstick, and a slice cannot hide behind the average
The same fixed set makes scores comparable, and scoring slices separately shows a group that fell.
The project is done when all of these hold.
Write a sentence or two, then compare it with a good answer in the lesson.
1. The contract. A batch of subscribers is checked for nulls and for values outside the range each column may take. The cell checks a clean batch and one with 30 negative spends and 10 missing tenures.
The lesson runs this once, before its first cell, and every cell, drill and example starts with what it defines. A cell that changes a setup object changes it for the cells after it, until the page reloads.
import hashlib import json import math import numpy as np from sklearn.linear_model import LogisticRegression RANGES = {"tenure": (0, 120), "spend": (0, 500), "tickets": (0, 30)} NAMES = ["tenure", "spend", "tickets", "new", "new_x_tickets"] BASE, ALL = [0, 1, 2], [0, 1, 2, 3, 4] def make_data(n, seed, drift=0.0): """Subscribers: tenure in months, monthly spend, support tickets, and churn.""" rng = np.random.default_rng(seed) tenure = rng.integers(1, 61, n).astype(float) spend = np.clip(rng.normal(60 + 40 * drift, 20, n), 0, 500) tickets = rng.poisson(1.5 + 2 * drift, n).astype(float) new = tenure < 9 z = -0.03 * tenure + 0.25 * tickets - 0.01 * spend + 0.2 z = z + new * (-4.0 + 2.6 * tickets) y = (rng.random(n) < 1 / (1 + np.exp(-z))).astype(int) return np.column_stack([tenure, spend, tickets, new, new * tickets]), y def contract_check(X): problems = [] if np.isnan(X).any(): problems.append(f"nulls: {int(np.isnan(X).any(axis=1).sum())} rows") for j, name in enumerate(NAMES[:3]): low, high = RANGES[name] bad = int(((X[:, j] < low) | (X[:, j] > high)).sum()) if bad: problems.append(f"{name} out of range: {bad} rows") return problems def data_hash(X, y): return hashlib.sha256(X.tobytes() + y.tobytes()).hexdigest()[:12] def train(X, y, cols, seed=0, C=1.0): return LogisticRegression(C=C, max_iter=2000, random_state=seed).fit(X[:, cols], y) def slices(model, cols, X, y): new = X[:, 3] == 1 return {"all": round(float(model.score(X[:, cols], y)), 3), "new": round(float(model.score(X[new][:, cols], y[new])), 3), "tenured": round(float(model.score(X[~new][:, cols], y[~new])), 3)} class Registry: def __init__(self): self.versions, self.aliases, self.history = {}, {}, [] def register(self, model, record): number = len(self.versions) + 1 self.versions[number] = {"model": model, "record": record} return number def set_alias(self, alias, version, note): self.history.append({"alias": alias, "from": self.aliases.get(alias), "to": version, "note": note}) self.aliases[alias] = version def get(self, ref): number = self.aliases[ref] if isinstance(ref, str) else ref return self.versions[number] def gate(candidate, champion, min_gain=0.005, max_drop=0.02): reasons = [] if not candidate["data"] or not candidate["code"]: reasons.append("no lineage") if candidate["scores"]["all"] - champion["scores"]["all"] < min_gain: reasons.append("gain too small") for name, score in champion["scores"].items(): if score - candidate["scores"][name] > max_drop: reasons.append(name + " dropped") return reasons def psi(expected, actual, bins=5): edges = np.quantile(expected, np.linspace(0, 1, bins + 1)) edges[0], edges[-1] = -np.inf, np.inf e = np.histogram(expected, edges)[0] / len(expected) a = np.histogram(actual, edges)[0] / len(actual) e, a = np.clip(e, 1e-4, None), np.clip(a, 1e-4, None) return float(np.sum((a - e) * np.log(a / e))) def should_retrain(psi_value, weeks_since, max_psi=0.2, max_weeks=8): return psi_value > max_psi or weeks_since >= max_weeks Xt, yt = make_data(4000, 1) # what the first champion learns from Xe, ye = make_data(3000, 9) # the fixed evaluation set X2, y2 = make_data(8000, 2) # a bigger extract, with its labels def first_registry(): reg = Registry() model = train(Xt, yt, BASE) record = {"params": {"C": 1.0}, "seed": 0, "data": data_hash(Xt, yt), "code": "a41c9e7", "scores": slices(model, BASE, Xe, ye)} reg.register(model, record) reg.set_alias("champion", 1, "first champion") return reg def registry_after_gate(): """The registry once version 3, trained on repaired labels, is champion.""" reg = first_registry() model = train(X2, y2, ALL) record = {"data": data_hash(X2, y2), "code": "c07be11", "scores": slices(model, ALL, Xe, ye)} number = reg.register(model, record) reg.set_alias("champion", number, "passed the gates") return reg
X, y = make_data(2000, 0) print(contract_check(X)) broken = X.copy() broken[:30, 1] = -5 broken[30:40, 0] = np.nan print(contract_check(broken))
The clean batch has no problems. The broken one reports 10 rows with nulls and 30 with a spend out of range, so it would be refused before training.
import numpy as np spend = np.array([60.0] * 1970 + [-5.0] * 30) print(int((spend < 0).sum()))
30
The contract counts the rows that break the range rule, and 30 rows do.
2. A tracked run and a registered version. The first champion uses the three base columns. Its record holds the settings, the seed, a hash of the data it saw, the code commit and its scores on a fixed evaluation set, by slice.
Uses the setup above, 120 lines.
reg = Registry()
m1 = train(Xt, yt, BASE)
record = {"params": {"C": 1.0}, "seed": 0, "data": data_hash(Xt, yt), "code": "a41c9e7",
"scores": slices(m1, BASE, Xe, ye)}
v1 = reg.register(m1, record)
reg.set_alias("champion", v1, "first champion")
print(v1, record["data"], record["scores"])
Version 1 is registered with data hash d5f6172ed7a2. On the evaluation set it scores 0.736 overall, 0.823 for new subscribers and 0.724 for tenured ones. Those three numbers are what every later candidate is measured against.
3. The gate. Two candidates arrive from a bigger extract of 8,000 rows. In the first, a join bug has set the label to 0 for most new subscribers. The second is trained on the repaired labels.
Uses the setup above, 120 lines.
reg = first_registry() bug = (X2[:, 3] == 1) & (np.random.default_rng(3).random(8000) < 0.7) for name, labels in (("bugged labels", np.where(bug, 0, y2)), ("repaired labels", y2)): model = train(X2, labels, ALL) cand = { "data": data_hash(X2, labels), "code": "c07be11", "scores": slices(model, ALL, Xe, ye), } number = reg.register(model, cand) reasons = gate(cand, reg.get("champion")["record"]) print("version", number, name, cand["scores"], "->", reasons or "promoted") if not reasons: reg.set_alias("champion", number, "passed the gates") print(reg.aliases, len(reg.history))
Version 2 is refused with three reasons: its gain is too small, the overall score dropped (0.71 against 0.736) and the new-subscriber slice fell from 0.823 to 0.586. Its record was complete, and its data passed the contract: the damage was in the labels. Version 3 gains overall (0.744), raises the new slice to 0.872 and holds the tenured slice at 0.726, so the alias moves. The history has two lines: the first champion and this promotion.
champ = {'all': 0.736, 'new': 0.823}
cand = {'all': 0.710, 'new': 0.586}
reasons = []
if cand['all'] - champ['all'] < 0.005:
reasons.append('gain too small')
for name, score in champ.items():
if score - cand[name] > 0.02:
reasons.append(name + ' dropped')
print(reasons)['gain too small', 'all dropped', 'new dropped']
The overall score fell, so the gain is below the minimum and the 'all' slice dropped by 0.026. The new slice dropped by 0.237.
4. Hybrid serving. Scores for subscribers are stored by a nightly job. A request reads the stored score if it is fresh, no more than 24 hours old, and runs the model live if it is stale or missing. A lookup takes 2 ms and a live score 40 ms.
Uses the setup above, 120 lines.
table = {"c1": {"score": 0.31, "at": 0}, "c2": {"score": 0.82, "at": 20}}
def source(customer, now, max_age=24):
entry = table.get(customer)
return "lookup" if entry is not None and now - entry["at"] <= max_age else "live"
sources = [source(c, 30) for c in ("c1", "c2", "c3")]
print(
sources, round(sum(2 if s == "lookup" else 40 for s in sources) / 3, 1),
"ms on average",
)At hour 30, subscriber c1's score is 30 hours old, so it is computed live. c2's is 10 hours old and is looked up. c3 has no stored score, so it is live as well. The average is 27.3 ms: two live calls and one lookup. The freshness limit is a decision for the owners of the product.
5. The monitor. A weekly check compares each input with its training values using the population stability index, PSI, from Monitoring a model: drift and decay. The cell shows four weeks of growing drift.
Uses the setup above, 120 lines.
base = Xt[:, 2] for drift in (0.0, 0.1, 0.3, 0.6): Xd, yd = make_data(3000, 20 + int(drift * 10), drift) print( drift, "tickets PSI", round(psi(base, Xd[:, 2]), 3), "| spend PSI", round(psi(Xt[:, 1], Xd[:, 1]), 3), )
With no drift both are 0.001. At drift 0.3 the spend input is at 0.36, over the 0.2 line, while tickets are still at 0.145. At 0.6 both are well over. Different inputs cross the line at different times, so the alert names the input, not just the model.
The same four weeks as a plot:
Uses the setup above, 120 lines.
import matplotlib.pyplot as plt drifts = [0.0, 0.1, 0.3, 0.6] tickets = [0.001, 0.025, 0.145, 0.449] spend = [0.001, 0.038, 0.36, 1.28] plt.plot(drifts, tickets, marker="o", label="tickets") plt.plot(drifts, spend, marker="o", label="spend") plt.axhline(0.2, linestyle=":", color="gray") plt.xlabel("how far the inputs have drifted") plt.ylabel("PSI against training") plt.legend() plt.show()
def should_retrain(psi_value, weeks_since, max_psi=0.2, max_weeks=8): return psi_value > max_psi or weeks_since >= max_weeks print(should_retrain(0.449, 3), should_retrain(0.05, 9))
True True
The first has drifted past 0.2. The second has not drifted, but the schedule floor of 8 weeks has passed.
6. One retrain. The alert fired, so the pipeline trains a candidate on fresh data from the drifted world and puts it through the same gate, scored on a fresh evaluation set from that world.
Uses the setup above, 120 lines.
reg = registry_after_gate() Xr, yr = make_data(4000, 5, drift=0.6) Xde, yde = make_data(3000, 12, drift=0.6) champion = reg.get("champion") cand = train(Xr, yr, ALL) champ_scores = slices(champion["model"], ALL, Xde, yde) record4 = { "data": data_hash(Xr, yr), "code": "c07be11", "scores": slices(cand, ALL, Xde, yde), } print("champion on the drifted world:", champ_scores) print( "candidate:", record4["scores"], "->", gate(record4, {"scores": champ_scores}) or "promoted", )
The champion scores 0.761 on the drifted world and the retrained candidate 0.759. The gate refuses the candidate: its gain is too small. The inputs drifted, but the rule that links them to churn did not, so the model in production is still right. Drift in the inputs is a reason to look, not always a reason to replace. The champion stays, and the failed candidate is recorded.
Write serve_source(table, customer, now, max_age). table maps a customer to {"score": ..., "at": hour}. Return "lookup" when the customer has a stored score that is at most max_age hours old at now, and "live" otherwise (missing, or older than max_age).
Uses the setup above, 120 lines.
def serve_source(table, customer, now, max_age): return "live" print(serve_source({"c1": {"score": 0.3, "at": 0}}, "c1", 10, 24))
Write drift_alert(psis, limit, needed). psis is the weekly PSI of one input, in order. Return the index of the week at which the PSI has been above limit for needed weeks in a row, or None if that never happens.
Uses the setup above, 120 lines.
def drift_alert(psis, limit, needed): return None print(drift_alert([0.01, 0.3, 0.4, 0.5], 0.2, 3))
Write a sentence or two, then compare it with a good answer in the lesson.
A model in production comes with its paperwork. The worked example below computes the facts for a datasheet, a model card and a fairness audit from the run itself, so none of them can disagree with the data. The people still write the rest, as in Datasheets, What regulation asks of an engineer and Measuring fairness.
Compute the evaluation set's composition, the model card's claims and the fairness audit by slice, and print them as one document. The fairness audit uses the champion, version 3.
Uses the setup above, 120 lines.
Xe, ye = make_data(3000, 9) X2, y2 = make_data(8000, 2) champion = train(X2, y2, ALL) new = Xe[:, 3] == 1 pred = champion.predict(Xe[:, ALL]) print("# Datasheet: churn evaluation set") print( "rows:", len(ye), "| churn rate:", round(float(ye.mean()), 3), "| new subscribers:", round(float(new.mean()), 3), ) print("data hash:", data_hash(Xe, ye)) print("# Model card: version 3") print("claims:", slices(champion, ALL, Xe, ye)) print("# Fairness audit: new against tenured") for name, mask in (("new", new), ("tenured", ~new)): p, t = pred[mask], ye[mask] print( name.ljust(8), int(mask.sum()), "rows | flagged", round(float(p.mean()), 3), "| caught", round(float(p[t == 1].mean()), 3), "| false alarms", round(float(p[t == 0].mean()), 3), )
The evaluation set has 3,000 rows, 29.5% of whom churn and 12.2% of whom are new subscribers. The model card's claims, 0.744 overall, 0.872 for new and 0.726 for tenured, come from the same run. The audit shows the finding to write down: the model catches 84.4% of the churn among new subscribers but only 6.3% among tenured ones. It flags only 3.3% of tenured subscribers at all. The model is good where the retention team wants it most, and nearly blind for long-standing customers. That is a limit for the card's "Limits" section, not a surprise to find in production.
About 84%
The model flags 84.4% of new subscribers who go on to churn. 6.3% is the figure for tenured subscribers.
The gate checks three slices. The retention team says low-spend subscribers matter most to them. Extend the pipeline so the scores include a low spend slice (spend under 40) and a high spend slice (40 or more), and show the gate refusing a candidate that drops either by more than 0.02.
Uses the setup above, 120 lines.
reg = Registry() Xt, yt = make_data(4000, 1) Xe, ye = make_data(3000, 9) model = train(Xt, yt, BASE) record = { "data": data_hash(Xt, yt), "code": "a41c9e7", "scores": slices(model, BASE, Xe, ye), } reg.register(model, record) reg.set_alias("champion", 1, "first champion") print(reg.aliases, record["scores"]) # your code: score a candidate by one more slice (spend under 40, and 40 or more), # add it to the record's scores, and make the gate refuse a candidate that drops it
Your team will review the churn service before it goes live. Write the one-page design doc a reviewer would read first: what it is for, the numbers it must meet, how it is built from the parts of this capstone, and what could go wrong.
## Goal and metric What it is for, with the metric and a guardrail. ## Design The parts, in order, in five sentences or fewer. ## Numbers Requests a second at peak, latency budget, cost, label delay. ## Risks Three things that could go wrong and how you would notice.
The PSI monitor has raised an alert at 2 a.m. for the spend input. Write the runbook entry for the person on call, who has not seen this service before.
## Symptom What the alert says. ## Confirm How to check it is real. ## First step The first safe action. ## Escalate Who to call, and when.
Write a sentence or two, then compare it with a good answer in the lesson.
A reviewer opens the design doc, runs the pipeline from the repository and reads the model card against the latest run. If the card's numbers match, the review is short. If you have kept the gate, the history and the runbook, the question "what happens at 2 a.m." has an answer.
Write a sentence or two, then compare it with a good answer in the lesson.
Version 2's data passed the contract, yet its model was refused. Why?
575 lessons in thirteen phases. 176 are free without an account, and a free account opens the rest. See every lesson.
The small functions that decide whether a model ships and stays shipped: a registry gate by slice, a hybrid serving rule, a drift alert, a retrain rule and a model-card check.
Take a model from a notebook to a system that stays right: track and reproduce every run, search settings honestly, version data and models, promote through gates, keep training and serving in step, choose batch or online, catch drift, and retrain through a pipeline. Every number is measured by the lesson's own code.
Never trained an AI? Teach a small one to write, right in your browser. It takes about 20 seconds and needs no account. Teach it to write