One method for an ML design, goal and metric to retraining, with the sums that size it: requests a second at peak, a year of feature storage, a label delay and a latency budget.
An ML design question is one question asked of seven parts: what is it for, what data teaches it, which features it reads, what model, how it serves, what you watch, and when you retrain. Teams that skip a part find out in production: a model that needs a label that arrives a month late, a feature store ten times bigger than the budget, a server that falls over at the evening peak. This lesson gives you the method and the sums, so the numbers decide the design.
What you'll be able to do
Walk one method for an ML design: goal and metric, data and labels, features, model, serving mode, monitoring and retraining
Size a system by hand: predictions a second at peak, storage for a year of features, and a latency budget
Explain how a label delay limits how fresh a retrained model can be
Write a one-page ML design that a reviewer can check against numbers
Quick check
A guess before you start. A bank wants to flag a fraudulent card payment while the customer is waiting. Which decision comes before you choose any model?
Which cloud the service runs on
Which model has the most parameters in the family
How many data scientists to hire
What a wrong answer costs, and how fast the answer must come
Show the answer
What a wrong answer costs, and how fast the answer must come
A false alarm blocks a real customer and a miss costs money, and a decision at the till must come in milliseconds. The model follows from those two facts.
An ML system design is seven questions, asked in the same order every time.
Step
The question
The fraud scorer's answer
1. Goal and metric
What is it for, and what number says it works?
Block fraud at the till. Fraud caught per 1,000 payments, with false alarms under 2%
2. Data and labels
What examples teach it, and when does the answer arrive?
Past payments. The label is a chargeback, 21 days later
3. Features
What does it read, and how fresh must that be?
The payment itself, plus the card's last hour of activity
4. Model
What is the simplest model that could do the job?
Gradient boosting on those features
5. Serving
Batch, online or hybrid, and what is the latency budget?
Online: the inputs exist only at payment time. 100 ms in all
6. Monitoring
What do you watch once it is live?
Input drift, the fraud rate caught, label delay
7. Retraining
What starts a new model, and what gates it?
A weekly schedule or a drift alert, then the registry gate
Quick check
From Batch and online inference: the fraud check above runs while a payment is pending. Which serving mode is that?
Neither: a model cannot run that fast for a payment
Online: the inputs exist only when the payment is made
Batch: scores are computed overnight
Hybrid only: it always needs a lookup
Show the answer
Online: the inputs exist only when the payment is made
A payment does not exist until someone makes it, so its score cannot be computed ahead of time.
Try it now
Two delays shape everything
Every ML design has two clocks. Feature freshness is how old a feature may be when the model reads it: a card's activity in the last hour needs a live lookup, while a customer's birth year can sit in a table. Label delay is how long after a prediction the right answer arrives: a chargeback takes 21 days.
The label delay is the one people forget. A model retrained today can only learn from payments at least 21 days old, so "last week's fraud" is not in its training set. The picture shows the gap.
Quick check
What does this print: the average payments a second for 40,000,000 a day, and the peak at five times that?
40,000,000 / 86,400 is 462.96, which rounds to 463, and five times that is 2,315. A day has 86,400 seconds, not 24.
Quick check
What does this print: the machines needed at a peak of 1,000 requests a second, 10 ms each, kept under 50% busy?
import math
peak, ms, busy = 1000, 10, 0.5print(math.ceil(peak / (1000 / ms * busy)))
20
2
100
10
Show the answer
20
One machine at 50% does 100 × 0.5 = 50 requests a second, so 1,000 / 50 = 20 machines.
Sizing before choosing
A design that cannot be sized is a guess. Three sums do most of the work, and each is the ratio of two quantities you can look up.
Requests a second. Divide the day's predictions by 86,400. Then multiply by a peak factor: the busiest minute of a day is often several times the average, and a service has to survive it. Storage. Features cost bytes: features per prediction, times bytes each, times predictions a day, times the days you keep them. A year of 200 four-byte features for 40 million payments a day is 11.68 terabytes, which is a different conversation from the 32 gigabytes of one day. A latency budget. The caller has a total, say 100 ms, and every hop spends some of it: the network, the feature lookup, the model. What the others leave is what the model gets.
None of these needs a fast model or a clever one. They need arithmetic, written down where a reviewer can argue with it.
In your own words
In your own words: why can a model retrained today not learn from last week's fraud, and what does that mean for how often retraining helps?
Write a sentence or two, then compare it with a good answer in the lesson.
In code: a fraud scorer's budget
The sizes above become code. A server takes 8.5 ms to score one payment, one at a time. Its limit is 1,000 / 8.5 ≈ 117.6 requests a second. The cell below runs one server for 4,000 random arrivals at four loads, using the simulate and p95 helpers from the setup, and prints the average and the p95 latency.
Setup (runs before every cell) · 19 lines
The lesson runs this once, before its first cell, and every cell, drill and example starts with what it defines. A cell that changes a setup object changes it for the cells after it, until the page reloads.
import math
import random
defp95(values):
ordered = sorted(values)
return ordered[math.ceil(0.95 * len(ordered)) - 1]
defsimulate(rate, service_ms, n=4000, seed=0):
"""One server, first come first served. rate is requests a second."""
rng = random.Random(seed)
t, free_at, out = 0.0, 0.0, []
for _ inrange(n):
t += rng.expovariate(rate) * 1000
start = max(t, free_at)
free_at = start + service_ms
out.append(free_at - t)
return out
for rate in (40, 80, 100, 115):
latency = simulate(rate, 8.5)
print(
rate, "per second | mean", round(sum(latency) / len(latency), 1), "| p95",
round(p95(latency), 1),
)
At 40 a second the p95 is 19.6 ms. At 100 it is 71.7 ms, and at 115 it is 293.9 ms. The server never saw more than its limit of 117.6, and the tail still grew fifteen times. That is the shape to design around: you size for the p95 and the peak, not the average.
Quick check
Which of these is a monitoring number for a fraud model that does not need labels?
The share of payments over a set amount, compared with training
Fraud caught per 1,000 payments, confirmed
The F1 score on this month's payments
The chargeback rate of last week's payments, once known
Show the answer
The share of payments over a set amount, compared with training
Labels arrive 21 days late, so right now you can only watch inputs and scores. Outcomes follow three weeks later.
The same simulation as a plot shows the cliff beside the limit:
Uses the setup above, 19 lines.
import matplotlib.pyplot as plt
rates = [40, 80, 100, 115]
tail = [p95(simulate(rate, 8.5)) for rate in rates]
plt.plot(rates, tail, marker="o")
plt.axvline(117.6, linestyle=":", color="gray")
plt.xlabel("requests per second")
plt.ylabel("p95 latency (ms)")
plt.show()
print([round(v, 1) for v in tail])
The dotted line is the server's limit. The curve bends up well before it.
Quick check
In the plot, at which load does the p95 first pass 100 ms?
Between 100 and 115 requests a second
Below 40
Never: it stays under 100 ms
Between 40 and 80
Show the answer
Between 100 and 115 requests a second
At 100 it is 71.7 ms and at 115 it is 293.9 ms, so it crosses 100 ms in between.
Worked example
Worked example: a design you can check
Put the budget together for the fraud scorer, with the slower server (8.5 ms a payment). Every number in the design comes out of one short run.
Uses the setup above, 19 lines.
per_day, peak_factor, ms = 40_000_000, 5, 8.5
average = per_day / 86400
peak = average * peak_factor
limit = 1000 / ms
machines = max(2, math.ceil(peak / (limit * 0.7)))
dollars_a_day = machines * 0.5 * 24print("peak", round(peak), "a second | one machine's limit", round(limit, 1))
print("machines at 70% busy:", machines, "| dollars a day:", dollars_a_day)
print("cost per million:", round(dollars_a_day / per_day * 1_000_000, 2))
print("days with labels in a 56-day window:", 56 - 21)
In the lesson you can change this and run it.
A peak of 2,315 payments a second against a limit of 117.6 per machine needs 29 machines to stay under 70% busy, and 29 machines all day cost $348, or $8.70 per million payments. A faster model is worth money: at 8 ms the same peak needs 27 machines. And a 56-day training window has only 35 days with labels. Each of those four numbers is a line a reviewer can challenge, which is what makes it a design and not a story.
Drill
Size a service
Write size_system(per_day, peak_factor, ms_per_request, max_utilization). Return a dictionary with "avg_rps" (per_day over 86,400, rounded to a whole number), "peak_rps" (the average times peak_factor, rounded to a whole number) and "machines": the number of machines, at least 2, so that at the peak each machine is busy no more than max_utilization of the time. One machine at 100% finishes 1000 / ms_per_request requests a second.
In the lesson, your code is checked as you write it.
Drill
Storage for a year of features
Write storage_tb(features, bytes_each, per_day, days): the terabytes needed to keep features values of bytes_each bytes for each of per_day predictions a day, for days days. A terabyte is 10^12 bytes. Round to 2 places.
In the lesson, your code is checked as you write it.
In your own words
A colleague sizes a service for the average load. What happens at the evening peak, and what number should they have used?
Write a sentence or two, then compare it with a good answer in the lesson.
On the job: a design review and a one-page answer
In a design review, someone draws the seven boxes and the room asks for the numbers: requests a second at peak, bytes of features for a year, the delay on the label, the latency budget. The reviewer's job is to find the step where nobody has a number. Your job, in an interview or a document, is to have one for each step.
The Interviews page has two ML sessions that ask exactly this: Fraud scoring and A recommender. Practice them against the method.
Written brief · Design answer
A one-page design for a recommender
A shop with 2 million daily visitors wants a "customers like you also bought" list on each product page, shown within 100 ms. Write the one-page design you would bring to the review: the method, with the numbers.
It must
Names one metric and one guardrail
Gives at least two sized numbers (requests a second, storage or cost)
Says how stale a stored list may be
Go further
Says what happens for a visitor with no history
The template
## Goal and metric
What the list is for, and one number that says it works.
## Data and labels
Where the examples come from, and how late a label arrives.
## Features and serving
Which features, how fresh, and batch, online or hybrid, with the numbers that decide it.
## Monitoring and retraining
What you watch, and what triggers a retrain.
In the lesson, you write over it, 140 to 320 words, and the tutor can mark it.
03Designing AI systemsLatency, cost, routing, fallbacks and guardrails for features built on models.40 min
04Containers and deploymentPackage an app as an image, configure it with the environment, and roll out safely.45 min
05CI/CD pipelinesTest every change automatically, and ship with gates, flags and rollbacks.40 min
06Docker ComposeSeveral containers together: service names, volumes, secrets and start order.40 min
07Kubernetes basicsDesired state, Deployments and Services, labels, probes and rolling updates.40 min
08Cloud for AI engineersAccess policies and least privilege, storage, compute choices, managed model endpoints, secrets and budgets.30 min
09Metrics, dashboards and alert rulesCounters, gauges and histograms, a rate from a counter, a percentile from buckets, a dashboard panel and an alert rule that fires on a sustained condition.40 min
11Incident lab: from the page to the postmortemOne incident run from the page to the all-clear: decide the mitigation, work out the impact in failed requests and error budget, and write a blameless postmortem, a status update and a runbook entry.40 min
13A/B testing a model changeHash assignment, per-user results from raw logs, sample ratio checks, guardrails and a decision rule.40 min
14Columnar files and a local warehouseRows against columns, the encodings that shrink a column, Parquet's row groups and the ranges that let a query skip them, and SQL over files with DuckDB.40 min
15Data quality checks and contractsSix properties of a delivery you can test, each as a count of rows that break it, a contract with the producer, and when to block, quarantine or warn.40 min
16Transformations: staging, models and testsRaw, staging, facts and marts as SQL models run in dependency order in DuckDB, tests as queries that count the rows that break a rule, an incremental load that is safe to rerun, and the same models as dbt files.45 min
17Streams and windowsEvents that never stop arriving: event time against arrival time, tumbling, sliding and session windows, a watermark for late events, and counting each event once.40 min
18Labels and annotator agreementLabels as measurements: agreement and Cohen's kappa, the pair of classes annotators confuse, votes and gold questions, and what wrong labels do to a model and a test score.45 min
19Experiment tracking and reproducible runsA run record of settings, seed, data and code behind every score, a run log in a dozen lines, and comparing settings across seeds.40 min
20Hyperparameter searchWhat a grid costs, random search on a log scale, stopping weak settings early, and why the winner's validation score flatters it.40 min
21Versioning data and modelsFiles stored under the hash of their contents, a manifest that names a whole dataset, and lineage from data and code to each model.35 min
22A model registry and promotionVersions under one name, an alias for the live one, gates a candidate must pass, group by group, and a rollback that moves a pointer.40 min
23Feature pipelines and training-serving skewWhy a model that passes offline fails live: statistics refit at serving, two code paths, and features from the future, with the fix for each.40 min
24Batch and online inferencePredict ahead of time or on request: when the inputs exist, latency near a server's limit, batching what waits, cost, and a hybrid with a freshness limit.40 min
25Monitoring a model: drift and decayWhy a model worsens with no code change: data drift and concept drift, a measure of how far inputs have moved, thresholds set from noise, and labels that arrive late.45 min
26Retraining and pipelinesWhen to retrain and on which weeks, training as steps in dependency order, a cache that reruns only what changed, and guards before and after.45 min
27Designing ML systemsOne method for an ML design, goal and metric to retraining, with the sums that size it: requests a second at peak, a year of feature storage, a label delay and a latency budget.45 min
28Datasheets: documenting a datasetThe document that travels with a dataset: a profile by column and by group, claims a program can check, splits that don't leak, a scan for personal data, and the license and limits only people can write.45 min
29Curation, active learning and synthetic dataThree levers on a training set: removing copies and broken rows, letting a model choose which rows to label, and manufacturing rows, which helps only when their source knows more than you do.45 min
30Measuring fairnessBreak a model's results down by group: selection rate, true and false positive rates and precision, why they can't all be equal, what each fix costs, and when a group is too small to judge.40 min
31Privacy engineeringWhy removing names isn't anonymity, how to hold less (minimization, retention, keyed pseudonyms), the idea behind differential privacy and its budget, what a model remembers, and a deletion that reaches every store.40 min
32A red-team labAttack a simulated assistant methodically: eight kinds of attack run as a suite, success rates beside ordinary requests blocked, fresh attacks to test the fixes, where a defense belongs, and findings kept as release tests.45 min
33What regulation asks of an engineerRisk tiers that follow a system's use, the requirements for high-risk systems as artifacts an engineer builds, evidence tied to a model version, a model card a program can check, and legal dates kept as data.35 min
34Git for teamsBranches, pull requests, rebasing and how a three-way merge finds conflicts.40 min
35Writing for engineersREADMEs, design docs, commit messages and bug reports people can act on.35 min
36Scoping and estimating workA vague request cut into tickets with acceptance criteria and a first shippable slice, estimates as ranges, an honest update for a slip, and a ticket linked to its pull request.30 min
37Roles, résumé and the searchWhat AI, ML and data engineering postings ask for, résumé bullets that carry a number, a 100-word project summary, a take-home as a portfolio piece and how to read an offer.35 min
43Capstone: take a model to productionA churn model through a data contract, a tracked run, a registry gate by slice, hybrid serving, a PSI monitor and one retrain, shipped with a datasheet, a model card, a fairness audit, a design doc and a runbook.45 min
Take a model from a notebook to a system that stays right: track and reproduce every run, search settings honestly, version data and models, promote through gates, keep training and serving in step, choose batch or online, catch drift, and retrain through a pipeline. Every number is measured by the lesson's own code.
See the course →
Never trained an AI? Teach a small one to write, right in your browser. It takes about 20 seconds and needs no account. Teach it to write