Build an AI ad-copy reviewer in Python
A small Python tool that checks ad headlines against your rules, saves a JSON report, and asks a language model to rewrite the worst one.
The small functions that decide whether a model ships and stays shipped: a registry gate by slice, a hybrid serving rule, a drift alert, a retrain rule and a model-card check.
Training is the easy part of a model's life. What keeps it safe in production is a handful of decisions written as code: does this candidate beat the champion on every group, should this request use the stored score, has the data drifted for long enough to matter, and does the model card still match what was measured. This tutorial writes each as a function you can test.
Python functions, dictionaries and lists. It uses only the standard library, so it runs anywhere with Python 3.
A candidate must have its data and code recorded, must beat the champion overall by min_gain, and must not drop more than max_drop on any group the champion was scored on. Every reason it fails is returned, so the person reading it sees them all at once.
def gate(candidate, champion, min_gain, max_drop): reasons = [] if not candidate["data"] or not candidate["code"]: reasons.append("no lineage") if candidate["scores"]["all"] - champion["scores"]["all"] < min_gain: reasons.append("gain too small") for name, score in champion["scores"].items(): if name not in candidate[ "scores" ] or score - candidate["scores"][name] > max_drop: reasons.append(name + " dropped") return reasons
A stored score is used only when it exists and is fresh enough. Otherwise the model runs live. The age limit is the freshness number the design promised.
def serve_source(table, customer, now, max_age): entry = table.get(customer) return "lookup" if entry is not None and now - entry["at"] <= max_age else "live"
In the lesson you write this from a starter, in your browser with nothing to install, and checks tell you what's wrong as you go. A free account opens it and all 575 lessons.
One bad week is noise. The alert fires only when the population stability index stays above the limit for needed weeks in a row, and it says which week it fired.
def drift_alert(psis, limit, needed): streak = 0 for week, value in enumerate(psis): streak = streak + 1 if value > limit else 0 if streak == needed: return week return None
Retrain when the data has drifted too far, or when the model is simply old. The second trigger keeps a model that drifts slowly from going stale unnoticed.
def should_retrain(psi_value, weeks_since, max_psi, max_weeks): return psi_value > max_psi or weeks_since >= max_weeks
card_gaps lists every number on the card that is missing from the latest measurements or has moved by more than the tolerance. The last line asks the retrain rule about a PSI of 0.449 three weeks after the last retrain.
def card_gaps(card, latest, tolerance): return sorted( n for n, v in latest.items() if n not in card or abs(card[n] - v) > tolerance ) print(should_retrain(0.449, 3, 0.2, 8))
True
In the lesson these rules sit around a churn model: a data contract, a tracked run, the gate, hybrid serving, a PSI monitor and one retrain, shipped with a datasheet, a model card, a fairness audit, a design doc and a runbook.
gate on a candidate that wins overall but loses 0.06 on one group, and read the reason.needed in the drift alert from 2 to 3 and see which week it fires.def gate(candidate, champion, min_gain, max_drop): reasons = [] if not candidate["data"] or not candidate["code"]: reasons.append("no lineage") if candidate["scores"]["all"] - champion["scores"]["all"] < min_gain: reasons.append("gain too small") for name, score in champion["scores"].items(): if name not in candidate[ "scores" ] or score - candidate["scores"][name] > max_drop: reasons.append(name + " dropped") return reasons def serve_source(table, customer, now, max_age): entry = table.get(customer) return "lookup" if entry is not None and now - entry["at"] <= max_age else "live" def drift_alert(psis, limit, needed): streak = 0 for week, value in enumerate(psis): streak = streak + 1 if value > limit else 0 if streak == needed: return week return None def should_retrain(psi_value, weeks_since, max_psi, max_weeks): return psi_value > max_psi or weeks_since >= max_weeks def card_gaps(card, latest, tolerance): return sorted( n for n, v in latest.items() if n not in card or abs(card[n] - v) > tolerance ) print(should_retrain(0.449, 3, 0.2, 8))
In the lesson you write this from a starter, in your browser with nothing to install, and checks tell you what's wrong as you go. A free account opens it and all 575 lessons.
This is one stage of the AI engineer roadmap: the bigger picture, in order.
A small Python tool that checks ad headlines against your rules, saves a JSON report, and asks a language model to rewrite the worst one.
Build a research agent with tools, an agent loop, citations you can check and an eval, in plain Python with no framework.
Write the loss that language models train on, from scratch with NumPy, along with KL divergence, perplexity and the gradient.
A complete machine learning project with scikit-learn: baselines, a fair model comparison, one honest test, and a decision about who to contact.
Standardize from the training rows only, train a small network with a validation score after every epoch, and compare it fairly with logistic regression.
Write a tiny GPT in PyTorch, train it on a product catalog, and watch it write new descriptions. The whole program is about 70 lines.
Fine-tune a small chat model, test it on held-back products, and use release gates to catch a candidate that looks better but isn't.
A complete to-do tool with subcommands, a SQLite database, clear errors and exit codes, using only the standard library.
Authentication, validation, pagination, idempotency and clean errors, in a working API built with the standard library so you can see how each one works.
A working Model Context Protocol server with tools, resources and prompts, written without an SDK so you can see exactly what goes over the wire.
An agent that reads and edits a project, runs its tests and stops when they pass, with the safety checks that keep it inside the project.
API keys, rate limits, validation, caching, model fallbacks, structured logs and an eval, around a service that classifies support tickets.
A two-layer classifier for trail signs written straight from the formulas: forward pass, cross-entropy, a backward pass, training, an honest test score with its error bar, and next steps in PyTorch.
The checks in front of a streaming chat service: an API key, a token bucket for each user, and a daily token budget, in plain Python you can test.
The core of a typed, tested training package: a config that rejects unknown settings, mini-batches as a generator, a seeded training loop and an evaluation function.
Never trained an AI? Teach a small one to write, right in your browser. It takes about 20 seconds and needs no account. Teach it to write