Index Ai Academy
Phase 11 · System design

Designing ML systems

One method for an ML design, goal and metric to retraining, with the sums that size it: requests a second at peak, a year of feature storage, a label delay and a latency budget.

About 45 minutes · A free account opens the whole lesson.
Already have an account? Sign in to open this lesson

What it covers7

Why it matters

An ML design question is one question asked of seven parts: what is it for, what data teaches it, which features it reads, what model, how it serves, what you watch, and when you retrain. Teams that skip a part find out in production: a model that needs a label that arrives a month late, a feature store ten times bigger than the budget, a server that falls over at the evening peak. This lesson gives you the method and the sums, so the numbers decide the design.

What you'll be able to do

  • Walk one method for an ML design: goal and metric, data and labels, features, model, serving mode, monitoring and retraining
  • Size a system by hand: predictions a second at peak, storage for a year of features, and a latency budget
  • Explain how a label delay limits how fresh a retrained model can be
  • Write a one-page ML design that a reviewer can check against numbers
Quick check
A guess before you start. A bank wants to flag a fraudulent card payment while the customer is waiting. Which decision comes before you choose any model?
  1. Which cloud the service runs on
  2. Which model has the most parameters in the family
  3. How many data scientists to hire
  4. What a wrong answer costs, and how fast the answer must come
Show the answer

What a wrong answer costs, and how fast the answer must come

A false alarm blocks a real customer and a miss costs money, and a decision at the till must come in milliseconds. The model follows from those two facts.

The idea: seven questions, asked in order

You met the parts of an AI feature in Designing AI systems and the two serving modes in Batch and online inference. The sums here are the ratios of Ratio and proportion: per second, per day, per year.

An ML system design is seven questions, asked in the same order every time.

StepThe questionThe fraud scorer's answer
1. Goal and metricWhat is it for, and what number says it works?Block fraud at the till. Fraud caught per 1,000 payments, with false alarms under 2%
2. Data and labelsWhat examples teach it, and when does the answer arrive?Past payments. The label is a chargeback, 21 days later
3. FeaturesWhat does it read, and how fresh must that be?The payment itself, plus the card's last hour of activity
4. ModelWhat is the simplest model that could do the job?Gradient boosting on those features
5. ServingBatch, online or hybrid, and what is the latency budget?Online: the inputs exist only at payment time. 100 ms in all
6. MonitoringWhat do you watch once it is live?Input drift, the fraud rate caught, label delay
7. RetrainingWhat starts a new model, and what gates it?A weekly schedule or a drift alert, then the registry gate
Quick check
From Batch and online inference: the fraud check above runs while a payment is pending. Which serving mode is that?
  1. Neither: a model cannot run that fast for a payment
  2. Online: the inputs exist only when the payment is made
  3. Batch: scores are computed overnight
  4. Hybrid only: it always needs a lookup
Show the answer

Online: the inputs exist only when the payment is made

A payment does not exist until someone makes it, so its score cannot be computed ahead of time.

Try it now

Two delays shape everything

Every ML design has two clocks. Feature freshness is how old a feature may be when the model reads it: a card's activity in the last hour needs a live lookup, while a customer's birth year can sit in a table. Label delay is how long after a prediction the right answer arrives: a chargeback takes 21 days.

The label delay is the one people forget. A model retrained today can only learn from payments at least 21 days old, so "last week's fraud" is not in its training set. The picture shows the gap.

Quick check
What does this print: the average payments a second for 40,000,000 a day, and the peak at five times that?
per_day = 40_000_000
average = per_day / 86400
print(round(average), round(average * 5))
  1. 40000 200000
  2. 463 2315
  3. 463 463
  4. 1666667 8333333
Show the answer

463 2315

40,000,000 / 86,400 is 462.96, which rounds to 463, and five times that is 2,315. A day has 86,400 seconds, not 24.

Quick check
What does this print: the machines needed at a peak of 1,000 requests a second, 10 ms each, kept under 50% busy?
import math
peak, ms, busy = 1000, 10, 0.5
print(math.ceil(peak / (1000 / ms * busy)))
  1. 20
  2. 2
  3. 100
  4. 10
Show the answer

20

One machine at 50% does 100 × 0.5 = 50 requests a second, so 1,000 / 50 = 20 machines.

Sizing before choosing

A design that cannot be sized is a guess. Three sums do most of the work, and each is the ratio of two quantities you can look up.

Requests a second. Divide the day's predictions by 86,400. Then multiply by a peak factor: the busiest minute of a day is often several times the average, and a service has to survive it. Storage. Features cost bytes: features per prediction, times bytes each, times predictions a day, times the days you keep them. A year of 200 four-byte features for 40 million payments a day is 11.68 terabytes, which is a different conversation from the 32 gigabytes of one day. A latency budget. The caller has a total, say 100 ms, and every hop spends some of it: the network, the feature lookup, the model. What the others leave is what the model gets.

None of these needs a fast model or a clever one. They need arithmetic, written down where a reviewer can argue with it.

In your own words
In your own words: why can a model retrained today not learn from last week's fraud, and what does that mean for how often retraining helps?

Write a sentence or two, then compare it with a good answer in the lesson.

In code: a fraud scorer's budget

The sizes above become code. A server takes 8.5 ms to score one payment, one at a time. Its limit is 1,000 / 8.5 ≈ 117.6 requests a second. The cell below runs one server for 4,000 random arrivals at four loads, using the simulate and p95 helpers from the setup, and prints the average and the p95 latency.

Setup (runs before every cell) · 19 lines

The lesson runs this once, before its first cell, and every cell, drill and example starts with what it defines. A cell that changes a setup object changes it for the cells after it, until the page reloads.

import math
import random


def p95(values):
    ordered = sorted(values)
    return ordered[math.ceil(0.95 * len(ordered)) - 1]


def simulate(rate, service_ms, n=4000, seed=0):
    """One server, first come first served. rate is requests a second."""
    rng = random.Random(seed)
    t, free_at, out = 0.0, 0.0, []
    for _ in range(n):
        t += rng.expovariate(rate) * 1000
        start = max(t, free_at)
        free_at = start + service_ms
        out.append(free_at - t)
    return out
for rate in (40, 80, 100, 115):
    latency = simulate(rate, 8.5)
    print(
        rate, "per second | mean", round(sum(latency) / len(latency), 1), "| p95",
        round(p95(latency), 1),
    )

At 40 a second the p95 is 19.6 ms. At 100 it is 71.7 ms, and at 115 it is 293.9 ms. The server never saw more than its limit of 117.6, and the tail still grew fifteen times. That is the shape to design around: you size for the p95 and the peak, not the average.

Quick check
Which of these is a monitoring number for a fraud model that does not need labels?
  1. The share of payments over a set amount, compared with training
  2. Fraud caught per 1,000 payments, confirmed
  3. The F1 score on this month's payments
  4. The chargeback rate of last week's payments, once known
Show the answer

The share of payments over a set amount, compared with training

Labels arrive 21 days late, so right now you can only watch inputs and scores. Outcomes follow three weeks later.

The same simulation as a plot shows the cliff beside the limit:

Uses the setup above, 19 lines.

import matplotlib.pyplot as plt

rates = [40, 80, 100, 115]
tail = [p95(simulate(rate, 8.5)) for rate in rates]
plt.plot(rates, tail, marker="o")
plt.axvline(117.6, linestyle=":", color="gray")
plt.xlabel("requests per second")
plt.ylabel("p95 latency (ms)")
plt.show()
print([round(v, 1) for v in tail])

The dotted line is the server's limit. The curve bends up well before it.

Quick check
In the plot, at which load does the p95 first pass 100 ms?
  1. Between 100 and 115 requests a second
  2. Below 40
  3. Never: it stays under 100 ms
  4. Between 40 and 80
Show the answer

Between 100 and 115 requests a second

At 100 it is 71.7 ms and at 115 it is 293.9 ms, so it crosses 100 ms in between.

Worked example

Worked example: a design you can check

Put the budget together for the fraud scorer, with the slower server (8.5 ms a payment). Every number in the design comes out of one short run.

Uses the setup above, 19 lines.

per_day, peak_factor, ms = 40_000_000, 5, 8.5
average = per_day / 86400
peak = average * peak_factor
limit = 1000 / ms
machines = max(2, math.ceil(peak / (limit * 0.7)))
dollars_a_day = machines * 0.5 * 24
print("peak", round(peak), "a second | one machine's limit", round(limit, 1))
print("machines at 70% busy:", machines, "| dollars a day:", dollars_a_day)
print("cost per million:", round(dollars_a_day / per_day * 1_000_000, 2))
print("days with labels in a 56-day window:", 56 - 21)
In the lesson you can change this and run it.

A peak of 2,315 payments a second against a limit of 117.6 per machine needs 29 machines to stay under 70% busy, and 29 machines all day cost $348, or $8.70 per million payments. A faster model is worth money: at 8 ms the same peak needs 27 machines. And a 56-day training window has only 35 days with labels. Each of those four numbers is a line a reviewer can challenge, which is what makes it a design and not a story.

Drill

Size a service

Write size_system(per_day, peak_factor, ms_per_request, max_utilization). Return a dictionary with "avg_rps" (per_day over 86,400, rounded to a whole number), "peak_rps" (the average times peak_factor, rounded to a whole number) and "machines": the number of machines, at least 2, so that at the peak each machine is busy no more than max_utilization of the time. One machine at 100% finishes 1000 / ms_per_request requests a second.

Uses the setup above, 19 lines.

import math


def size_system(per_day, peak_factor, ms_per_request, max_utilization):
    return {"avg_rps": 0, "peak_rps": 0, "machines": 0}


print(size_system(40_000_000, 5, 8, 0.7))
In the lesson, your code is checked as you write it.
Drill

Storage for a year of features

Write storage_tb(features, bytes_each, per_day, days): the terabytes needed to keep features values of bytes_each bytes for each of per_day predictions a day, for days days. A terabyte is 10^12 bytes. Round to 2 places.

Uses the setup above, 19 lines.

def storage_tb(features, bytes_each, per_day, days):
    return 0.0


print(storage_tb(200, 4, 40_000_000, 365))
In the lesson, your code is checked as you write it.
In your own words
A colleague sizes a service for the average load. What happens at the evening peak, and what number should they have used?

Write a sentence or two, then compare it with a good answer in the lesson.

On the job: a design review and a one-page answer

In a design review, someone draws the seven boxes and the room asks for the numbers: requests a second at peak, bytes of features for a year, the delay on the label, the latency budget. The reviewer's job is to find the step where nobody has a number. Your job, in an interview or a document, is to have one for each step.

The Interviews page has two ML sessions that ask exactly this: Fraud scoring and A recommender. Practice them against the method.

Written brief · Design answer

A one-page design for a recommender

A shop with 2 million daily visitors wants a "customers like you also bought" list on each product page, shown within 100 ms. Write the one-page design you would bring to the review: the method, with the numbers.

It must

  • Names one metric and one guardrail
  • Gives at least two sized numbers (requests a second, storage or cost)
  • Says how stale a stored list may be

Go further

  • Says what happens for a visitor with no history

The template

## Goal and metric
What the list is for, and one number that says it works.

## Data and labels
Where the examples come from, and how late a label arrives.

## Features and serving
Which features, how fresh, and batch, online or hybrid, with the numbers that decide it.

## Monitoring and retraining
What you watch, and what triggers a retrain.
In the lesson, you write over it, 140 to 320 words, and the tutor can mark it.

Goes deeper in

In your own words
Pick one of the seven steps and say what number you would ask for in a design review, and why.

Write a sentence or two, then compare it with a good answer in the lesson.

Practice what you just read

  • 3 code examples you can edit and run
  • a practice exercise that checks your code
  • a 5-question quiz
  • 9 review cards for spaced repetition

Try a quiz question

Why is a model sized for the average load likely to fail in production?

  1. Traffic peaks, and latency climbs well before a server's limit
  2. Averages are always wrong by a fixed amount in practice
  3. Servers are slower on average than at peak
  4. A model's accuracy falls with load

Answer it at the end of the lesson to see if you're right, and why.

Before this lesson

Keep learning

Where this lesson leads.

575 lessons in thirteen phases. 176 are free without an account, and a free account opens the rest. See every lesson.

Phase 11 · about 34 hours of study

Professional practice

Design, ship and run real systems, work well in a team, and get hired 43 lessons 1 free
  1. System design fundamentals FreeRequirements, rough numbers, the standard parts and where systems break.40 min
  2. Caches, queues and background jobsSpeed reads with caches, and move slow work to queues that retry safely.40 min
  3. Designing AI systemsLatency, cost, routing, fallbacks and guardrails for features built on models.40 min
  4. Containers and deploymentPackage an app as an image, configure it with the environment, and roll out safely.45 min
  5. CI/CD pipelinesTest every change automatically, and ship with gates, flags and rollbacks.40 min
  6. Docker ComposeSeveral containers together: service names, volumes, secrets and start order.40 min
  7. Kubernetes basicsDesired state, Deployments and Services, labels, probes and rolling updates.40 min
  8. Cloud for AI engineersAccess policies and least privilege, storage, compute choices, managed model endpoints, secrets and budgets.30 min
  9. Metrics, dashboards and alert rulesCounters, gauges and histograms, a rate from a counter, a percentile from buckets, a dashboard panel and an alert rule that fires on a sustained condition.40 min
  10. Reliability and incidentsSLOs, error budgets, alerts, on-call and blameless postmortems.40 min
  11. Incident lab: from the page to the postmortemOne incident run from the page to the all-clear: decide the mitigation, work out the impact in failed requests and error budget, and write a blameless postmortem, a status update and a runbook entry.40 min
  12. Measuring an AI feature from its logsTyped views over event logs, metric specs, cost from a price table, and rollups you can rerun.40 min
  13. A/B testing a model changeHash assignment, per-user results from raw logs, sample ratio checks, guardrails and a decision rule.40 min
  14. Columnar files and a local warehouseRows against columns, the encodings that shrink a column, Parquet's row groups and the ranges that let a query skip them, and SQL over files with DuckDB.40 min
  15. Data quality checks and contractsSix properties of a delivery you can test, each as a count of rows that break it, a contract with the producer, and when to block, quarantine or warn.40 min
  16. Transformations: staging, models and testsRaw, staging, facts and marts as SQL models run in dependency order in DuckDB, tests as queries that count the rows that break a rule, an incremental load that is safe to rerun, and the same models as dbt files.45 min
  17. Streams and windowsEvents that never stop arriving: event time against arrival time, tumbling, sliding and session windows, a watermark for late events, and counting each event once.40 min
  18. Labels and annotator agreementLabels as measurements: agreement and Cohen's kappa, the pair of classes annotators confuse, votes and gold questions, and what wrong labels do to a model and a test score.45 min
  19. Experiment tracking and reproducible runsA run record of settings, seed, data and code behind every score, a run log in a dozen lines, and comparing settings across seeds.40 min
  20. Hyperparameter searchWhat a grid costs, random search on a log scale, stopping weak settings early, and why the winner's validation score flatters it.40 min
  21. Versioning data and modelsFiles stored under the hash of their contents, a manifest that names a whole dataset, and lineage from data and code to each model.35 min
  22. A model registry and promotionVersions under one name, an alias for the live one, gates a candidate must pass, group by group, and a rollback that moves a pointer.40 min
  23. Feature pipelines and training-serving skewWhy a model that passes offline fails live: statistics refit at serving, two code paths, and features from the future, with the fix for each.40 min
  24. Batch and online inferencePredict ahead of time or on request: when the inputs exist, latency near a server's limit, batching what waits, cost, and a hybrid with a freshness limit.40 min
  25. Monitoring a model: drift and decayWhy a model worsens with no code change: data drift and concept drift, a measure of how far inputs have moved, thresholds set from noise, and labels that arrive late.45 min
  26. Retraining and pipelinesWhen to retrain and on which weeks, training as steps in dependency order, a cache that reruns only what changed, and guards before and after.45 min
  27. Designing ML systemsOne method for an ML design, goal and metric to retraining, with the sums that size it: requests a second at peak, a year of feature storage, a label delay and a latency budget.45 min
  28. Datasheets: documenting a datasetThe document that travels with a dataset: a profile by column and by group, claims a program can check, splits that don't leak, a scan for personal data, and the license and limits only people can write.45 min
  29. Curation, active learning and synthetic dataThree levers on a training set: removing copies and broken rows, letting a model choose which rows to label, and manufacturing rows, which helps only when their source knows more than you do.45 min
  30. Measuring fairnessBreak a model's results down by group: selection rate, true and false positive rates and precision, why they can't all be equal, what each fix costs, and when a group is too small to judge.40 min
  31. Privacy engineeringWhy removing names isn't anonymity, how to hold less (minimization, retention, keyed pseudonyms), the idea behind differential privacy and its budget, what a model remembers, and a deletion that reaches every store.40 min
  32. A red-team labAttack a simulated assistant methodically: eight kinds of attack run as a suite, success rates beside ordinary requests blocked, fresh attacks to test the fixes, where a defense belongs, and findings kept as release tests.45 min
  33. What regulation asks of an engineerRisk tiers that follow a system's use, the requirements for high-risk systems as artifacts an engineer builds, evidence tied to a model version, a model card a program can check, and legal dates kept as data.35 min
  34. Git for teamsBranches, pull requests, rebasing and how a three-way merge finds conflicts.40 min
  35. Writing for engineersREADMEs, design docs, commit messages and bug reports people can act on.35 min
  36. Scoping and estimating workA vague request cut into tickets with acceptance criteria and a first shippable slice, estimates as ranges, an honest update for a slip, and a ticket linked to its pull request.30 min
  37. Roles, résumé and the searchWhat AI, ML and data engineering postings ask for, résumé bullets that carry a number, a 100-word project summary, a take-home as a portfolio piece and how to read an offer.35 min
  38. Interviews and your portfolioShow your work, and solve interview problems the way professionals do.40 min
  39. Code reviewWhat to look for, how to give and take feedback, and automating the routine checks.35 min
  40. Reading unfamiliar codeFind your way around a big codebase fast, with tools that map it for you.35 min
  41. Working with AI coding toolsGive agents good context, keep tasks small, and check every change they make.35 min
  42. Capstone: ship a production AI serviceAn AI-backed service with validation, caching, rate limits, fallbacks, logging and evals.45 min
  43. Capstone: take a model to productionA churn model through a data contract, a tracked run, a registry gate by slice, hybrid serving, a PSI monitor and one retrain, shipped with a datasheet, a model card, a fairness audit, a design doc and a runbook.45 min
Part of a free course 11 lessons

Free MLOps course: running machine learning in production

Take a model from a notebook to a system that stays right: track and reproduce every run, search settings honestly, version data and models, promote through gates, keep training and serving in step, choose batch or online, catch drift, and retrain through a pipeline. Every number is measured by the lesson's own code.

Never trained an AI? Teach a small one to write, right in your browser. It takes about 20 seconds and needs no account. Teach it to write