Index Ai Academy
Project tutorial · 5 steps

Take a machine learning model to production: the decision rules

The small functions that decide whether a model ships and stays shipped: a registry gate by slice, a hybrid serving rule, a drift alert, a retrain rule and a model-card check.

Free to read, with no account. The code comes from the lesson Capstone: take a model to production, and has been run.

The steps7

What you'll build

Training is the easy part of a model's life. What keeps it safe in production is a handful of decisions written as code: does this candidate beat the champion on every group, should this request use the stored score, has the data drifted for long enough to matter, and does the model card still match what was measured. This tutorial writes each as a function you can test.

What you'll learn

  • Assemble a model's path to production from the parts of the ML in production lessons
  • Gate a candidate model by slice, and explain why a data contract alone cannot catch damaged labels
  • Serve scores from a table when fresh and live otherwise, and watch inputs with PSI
  • Ship a datasheet, a model card and a fairness audit, with a design doc and a runbook

What you need

Python functions, dictionaries and lists. It uses only the standard library, so it runs anywhere with Python 3.

Step 1: A registry gate that looks at every slice

A candidate must have its data and code recorded, must beat the champion overall by min_gain, and must not drop more than max_drop on any group the champion was scored on. Every reason it fails is returned, so the person reading it sees them all at once.

def gate(candidate, champion, min_gain, max_drop):
    reasons = []
    if not candidate["data"] or not candidate["code"]:
        reasons.append("no lineage")
    if candidate["scores"]["all"] - champion["scores"]["all"] < min_gain:
        reasons.append("gain too small")
    for name, score in champion["scores"].items():
        if name not in candidate[
            "scores"
        ] or score - candidate["scores"][name] > max_drop:
            reasons.append(name + " dropped")
    return reasons

Step 2: Hybrid serving: stored score or live

A stored score is used only when it exists and is fresh enough. Otherwise the model runs live. The age limit is the freshness number the design promised.

def serve_source(table, customer, now, max_age):
    entry = table.get(customer)
    return "lookup" if entry is not None and now - entry["at"] <= max_age else "live"

Rather write it yourself?

In the lesson you write this from a starter, in your browser with nothing to install, and checks tell you what's wrong as you go. A free account opens it and all 575 lessons.

Step 3: A drift alert that needs a streak

One bad week is noise. The alert fires only when the population stability index stays above the limit for needed weeks in a row, and it says which week it fired.

def drift_alert(psis, limit, needed):
    streak = 0
    for week, value in enumerate(psis):
        streak = streak + 1 if value > limit else 0
        if streak == needed:
            return week
    return None

Step 4: A retrain rule with two triggers

Retrain when the data has drifted too far, or when the model is simply old. The second trigger keeps a model that drifts slowly from going stale unnoticed.

def should_retrain(psi_value, weeks_since, max_psi, max_weeks):
    return psi_value > max_psi or weeks_since >= max_weeks

Step 5: Check the model card against the measurements

card_gaps lists every number on the card that is missing from the latest measurements or has moved by more than the tolerance. The last line asks the retrain rule about a PSI of 0.449 three weeks after the last retrain.

def card_gaps(card, latest, tolerance):
    return sorted(
        n for n, v in latest.items() if n not in card or abs(card[n] - v) > tolerance
    )


print(should_retrain(0.449, 3, 0.2, 8))
What it prints
True

In the lesson these rules sit around a churn model: a data contract, a tracked run, the gate, hybrid serving, a PSI monitor and one retrain, shipped with a datasheet, a model card, a fairness audit, a design doc and a runbook.

Where to take it next

  • Run gate on a candidate that wins overall but loses 0.06 on one group, and read the reason.
  • Change needed in the drift alert from 2 to 3 and see which week it fires.
  • Add a rule that blocks a retrain on a day with a deploy freeze.

Mistakes to watch for

  • Believing a passing data contract means the model trained on it is right
  • Promoting a candidate because its overall score is higher, without scoring every slice
  • Retraining every time an input drifts, when the rule linking inputs to outcomes has not changed
  • Writing a model card whose claims are prose a program cannot check against the latest run
  • Leaving the runbook until the first night the alert fires
The complete program
def gate(candidate, champion, min_gain, max_drop):
    reasons = []
    if not candidate["data"] or not candidate["code"]:
        reasons.append("no lineage")
    if candidate["scores"]["all"] - champion["scores"]["all"] < min_gain:
        reasons.append("gain too small")
    for name, score in champion["scores"].items():
        if name not in candidate[
            "scores"
        ] or score - candidate["scores"][name] > max_drop:
            reasons.append(name + " dropped")
    return reasons


def serve_source(table, customer, now, max_age):
    entry = table.get(customer)
    return "lookup" if entry is not None and now - entry["at"] <= max_age else "live"


def drift_alert(psis, limit, needed):
    streak = 0
    for week, value in enumerate(psis):
        streak = streak + 1 if value > limit else 0
        if streak == needed:
            return week
    return None


def should_retrain(psi_value, weeks_since, max_psi, max_weeks):
    return psi_value > max_psi or weeks_since >= max_weeks


def card_gaps(card, latest, tolerance):
    return sorted(
        n for n, v in latest.items() if n not in card or abs(card[n] - v) > tolerance
    )


print(should_retrain(0.449, 3, 0.2, 8))

Write it yourself

In the lesson you write this from a starter, in your browser with nothing to install, and checks tell you what's wrong as you go. A free account opens it and all 575 lessons.

Keep building

More tutorials.

This is one stage of the AI engineer roadmap: the bigger picture, in order.

5 steps

Build an AI ad-copy reviewer in Python

A small Python tool that checks ad headlines against your rules, saves a JSON report, and asks a language model to rewrite the worst one.

3 steps

Train a neural network classifier in PyTorch

Standardize from the training rows only, train a small network with a validation score after every epoch, and compare it fairly with logistic regression.

6 steps

Build a GPT from scratch in Python

Write a tiny GPT in PyTorch, train it on a product catalog, and watch it write new descriptions. The whole program is about 70 lines.

9 steps

Build a REST API in Python from scratch

Authentication, validation, pagination, idempotency and clean errors, in a working API built with the standard library so you can see how each one works.

6 steps

Build an MCP server in Python from scratch

A working Model Context Protocol server with tools, resources and prompts, written without an SDK so you can see exactly what goes over the wire.

7 steps

Build a coding agent in Python

An agent that reads and edits a project, runs its tests and stops when they pass, with the safety checks that keep it inside the project.

8 steps

Ship a production AI service in Python

API keys, rate limits, validation, caching, model fallbacks, structured logs and an eval, around a service that classifies support tickets.

7 steps

Build a neural network from scratch with NumPy

A two-layer classifier for trail signs written straight from the formulas: forward pass, cross-entropy, a backward pass, training, an honest test score with its error bar, and next steps in PyTorch.

4 steps

Ship a small machine learning library in Python

The core of a typed, tested training package: a config that rejects unknown settings, mini-batches as a generator, a seeded training loop and an evaluation function.

Never trained an AI? Teach a small one to write, right in your browser. It takes about 20 seconds and needs no account. Teach it to write