Index Ai Academy
Phase 11 · ML in production

Batch and online inference

Predict ahead of time or on request: when the inputs exist, latency near a server's limit, batching what waits, cost, and a hybrid with a freshness limit.

About 40 minutes · A free account opens the whole lesson.
Already have an account? Sign in to open this lesson

What it covers6

Why it matters

A model in a notebook predicts when you press Run. A model in a product has to predict at some particular moment, for someone, at some cost. Do you score every customer overnight and store the answers, or compute each answer when it is asked for? The first is cheap and can be stale. The second is fresh and must answer in milliseconds, at 3 a.m., under load. Most of the cost and most of the outages of a deployed model follow from this choice, and it is usually made by accident. This lesson makes it on purpose.

What you'll be able to do

  • Tell batch inference from online inference, and choose between them by when the inputs exist and how fresh the answer must be
  • Read a server's latency at the median and at the 95th percentile, and see what happens when requests arrive faster than it can serve
  • Explain how grouping waiting requests raises a server's capacity, and what a fixed wait costs
  • Work out the cost per million predictions for each mode, and design a hybrid with a freshness limit
Quick check
A guess before you start. A model server handles one request at a time and takes 8.5 milliseconds for each. Requests begin arriving at 300 per second. What happens to the waiting time?
  1. It grows without limit: the server can finish only about 118 per second
  2. It falls, since a busy server wastes less time between requests
  3. It settles at about 8.5 milliseconds, the time one request takes
  4. It settles at about 25 milliseconds, three requests' worth of work
Show the answer

It grows without limit: the server can finish only about 118 per second

One request per 8.5 ms is about 118 per second. At 300 per second, 182 more arrive every second than can leave, so the queue, and every new request's wait, keeps growing for as long as the load lasts. The lesson's simulation shows waits of several seconds within ten seconds.

The idea: two moments to predict, and a queue

Latency is read at the 95th percentile, the way Describing data reads a spread, and the first check recalls the same idea from Metrics, logs and traces.

Two moments to predict

Batch inference runs the model ahead of time, on a schedule, over many rows at once. The predictions are written to a table, and the product reads them from there. The nightly job that scores every customer's churn risk is batch inference.

Online inference runs the model when a request arrives, and the caller waits for the answer. The fraud check that runs while a card payment is pending is online inference.

Two questions decide between them.

  1. When do the inputs exist? A churn score depends on the customer's history, which is already in the warehouse tonight. A fraud score depends on the payment, which doesn't exist until someone makes it. If the inputs only arrive with the request, you have no choice: it's online.
  2. How fresh must the answer be? If a score computed last night is still a good answer this afternoon, batch will do. If it must reflect what happened a minute ago, it won't.

Batch is the cheaper and simpler mode by a wide margin, so the working rule is: use batch unless one of those two questions rules it out.

The speed difference between scoring rows together and one at a time is large, even for a small model:

import time
import numpy as np
from sklearn.linear_model import LogisticRegression

rng = np.random.default_rng(0)
X = rng.normal(size=(2000, 20))
y = (X[:, 0] + X[:, 1] > 0).astype(int)
model = LogisticRegression().fit(X, y)

start = time.perf_counter()
together = model.predict(X)                                   # one call, 2,000 rows
batch_seconds = time.perf_counter() - start

start = time.perf_counter()
# 2,000 calls, one row each
one_by_one = [int(model.predict(X[i:i + 1])[0]) for i in range(2000)]
single_seconds = time.perf_counter() - start

print("same predictions:", together.tolist() == one_by_one)
print("one call was at least 10 times faster:", single_seconds > 10 * batch_seconds)
It runs right here in your browser, with nothing to install.

The arithmetic is the same either way. What differs is the overhead: every call pays a fixed price to check its input and set up the computation, before any row is scored. One call pays it once. Two thousand calls pay it two thousand times. Keep that fixed price in mind. It is the reason for everything in the third part of this lesson.

Try it now

Ask when the inputs exist and how old an answer may be. If neither forces your hand, choose batch.

Quick check
A team wants a model to estimate delivery time, shown to the customer on the checkout page. The estimate depends on what is in the basket and how busy the couriers are right now. Which mode fits?
  1. Neither: a model can't be used for a number shown to customers
  2. Batch: delivery estimates can be computed for everyone overnight
  3. Batch: an overnight table is cheaper, and cost comes first
  4. Online: the basket and the current load exist only at request time
Show the answer

Online: the basket and the current load exist only at request time

Both inputs, the basket and the couriers' load at this moment, come into existence with the request. Nothing computed last night could have known them, so the prediction has to be made online.

Latency, and the edge of the cliff

An online model is judged on two numbers. Latency is how long one request waits for its answer. Throughput is how many requests the server finishes per second. You met both in Metrics, logs and traces and Serving language models, along with the habit of reading latency at the 95th percentile, p95, because an average hides the slow requests that users remember.

A server has a limit. If one request takes 8.5 ms, the server can finish about 118 a second and no more. The next cell simulates one server under three loads. Requests arrive at random moments, as real ones do, and each waits its turn.

import math
import numpy as np

def percentile(values, p):
    ordered = sorted(values)
    return ordered[math.ceil(p / 100 * len(ordered)) - 1]

def serve(arrivals, max_size, max_wait, overhead=8.0, per_row=0.5):
    """One model server. Times in milliseconds. Returns each request's latency."""
    latencies, free_at, i = [], 0.0, 0
    while i < len(arrivals):
        # the server is free and a request is waiting
        start = max(free_at, arrivals[i])
        # optionally hold the batch open a little
        close = max(start, arrivals[i] + max_wait)
        j = i
        while j < len(arrivals) and j - i < max_size and arrivals[j] <= close:
            j += 1
        begin = max(start, arrivals[j - 1]) if j - i == max_size else close
        # one call: fixed overhead, plus a little per row
        done = begin + overhead + per_row * (j - i)
        latencies += [done - a for a in arrivals[i:j]]
        free_at, i = done, j
    return latencies

for rate in (50, 100, 300):
    rng = np.random.default_rng(0)
    arrivals = np.cumsum(rng.exponential(1000 / rate, size=3000)).tolist()
    latency = serve(arrivals, max_size=1, max_wait=0)     # one request per call
    print(rate, "requests/s | busy", round(rate * 8.5 / 1000, 2),
          "| p50", round(percentile(latency, 50), 1),
          "ms | p95", round(percentile(latency, 95), 1), "ms")

At 50 requests a second the median request waits 8.5 ms: it arrives, finds the server free, and is served. The p95 is already 21.8 ms, because requests sometimes arrive in a cluster and queue behind each other.

At 100 a second, still under the limit of 118, the p95 has climbed to 79.1 ms. The busy column says why: the server is working 85% of the time, so a cluster takes a long while to clear.

At 300 a second the server would have to be busy 2.55 times over. The median wait is more than seven seconds and still rising. This is not a slow server. It is a server past its limit, where the queue grows for as long as the load lasts.

Notice the shape: fine, fine, a little worse, then a cliff. Latency gives little warning before throughput runs out, which is why teams watch how close to the limit a server runs, and keep spare capacity.

Quick check
From the observability lesson: what does this print?
import math

def percentile(values, p):
    ordered = sorted(values)
    return ordered[math.ceil(p / 100 * len(ordered)) - 1]

latency_ms = [9, 8, 9, 10, 8, 9, 120, 9, 8, 10]
print(percentile(latency_ms, 50), percentile(latency_ms, 95),
      sum(latency_ms) / len(latency_ms))
  1. 20.0 120 9
  2. 9 120 20.0
  3. 9 10 20.0
  4. 9 120 9.0
Show the answer

9 120 20.0

Sorted, the fifth of ten values is 9, the median. The 95th percentile is the tenth value, 120: the one slow request. The mean, 20.0, describes none of the requests: nine were faster and one was far slower.

Quick check
A server takes 20 ms per request and handles one at a time. Requests arrive at a steady 40 per second. What should the team expect?
  1. It is nearly idle, since 40 per second is a light load
  2. It falls behind at once, since 40 is more than 20
  3. Its latency will be exactly 20 ms for every request
  4. It keeps up, but with little room: its limit is 50 per second
Show the answer

It keeps up, but with little room: its limit is 50 per second

At 20 ms each, the server finishes 1,000 ÷ 20 = 50 requests a second. Forty a second is 80% of that. It keeps up on average, but any burst queues, so the p95 will sit well above 20 ms.

In your own words
In your own words: what two questions decide between batch and online inference?

Write a sentence or two, then compare it with a good answer in the lesson.

In code: groups of requests, a PyTorch batch and the cost of each mode

Every cell simulates a server in a few lines of Python, so the cliff in the latency curve is something you can move.

Drill

How close to the limit?

Write load_report(ms_per_request, requests_per_second). Return a dictionary with "limit", the most requests the server can finish per second (1,000 divided by the milliseconds each takes), "utilization", the arrival rate as a share of that limit, rounded to 2 places, and "verdict": "overloaded" if utilization is 1 or more, "at risk" if it is 0.7 or more, and "ok" otherwise.

def load_report(ms_per_request, requests_per_second):
    return {"limit": 0, "utilization": 0, "verdict": "ok"}


print(load_report(8.5, 100))
In the lesson, your code is checked as you write it.

Serving requests in groups

The first part of this lesson showed that one call for many rows is far cheaper than many calls for one row, because the fixed overhead is paid once. A server can use that. When several requests are waiting, it scores them together in one call. This is micro-batching: the serving lesson's static batching, applied to any model.

In the simulation a call costs 8 ms of overhead plus 0.5 ms a row. Alone, a request takes 8.5 ms. In a batch of 32, the call takes 24 ms and each request's share is 0.75 ms: the same server can now finish over a thousand requests a second.

Here are three policies under the same three loads. The first is one request per call. The second takes whatever is waiting when the server becomes free, up to 32. The third also holds each batch open for 10 ms, hoping to fill it.

import math
import numpy as np

def percentile(values, p):
    ordered = sorted(values)
    return ordered[math.ceil(p / 100 * len(ordered)) - 1]

def serve(arrivals, max_size, max_wait, overhead=8.0, per_row=0.5):
    latencies, free_at, i = [], 0.0, 0
    while i < len(arrivals):
        start = max(free_at, arrivals[i])
        close = max(start, arrivals[i] + max_wait)
        j = i
        while j < len(arrivals) and j - i < max_size and arrivals[j] <= close:
            j += 1
        begin = max(start, arrivals[j - 1]) if j - i == max_size else close
        done = begin + overhead + per_row * (j - i)
        latencies += [done - a for a in arrivals[i:j]]
        free_at, i = done, j
    return latencies

for rate in (50, 100, 300):
    rng = np.random.default_rng(0)
    arrivals = np.cumsum(rng.exponential(1000 / rate, size=3000)).tolist()
    for name, size, wait in (("one at a time    ", 1, 0), ("batch what waits ", 32, 0),
                             ("batch, wait 10 ms", 32, 10)):
        latency = serve(arrivals, size, wait)
        print(rate, "requests/s |", name, "| p50", round(percentile(latency, 50), 1),
              "| p95", round(percentile(latency, 95), 1))

Read the middle policy down the three loads: a p95 of 16.2, 16.9 and 19.3 ms. At 300 requests a second, where the unbatched server was seconds behind, the batching one barely notices. When requests queue, it clears the whole queue in one call, so the busier it gets, the more efficient each call becomes.

Now the third policy. At 50 a second its median is 18.5 ms against 8.5: every request waited 10 ms for company that mostly didn't arrive. A fixed wait buys nothing at low load and costs latency on every request. It only pays when calls are so expensive that fuller batches matter more than the wait.

So the default is the middle policy: batch whatever is already waiting, and don't wait for more.

Quick check
A call to a model costs 6 ms of overhead plus 1 ms per row. What does this print: the most requests per second the server can finish with batches of 1, and with batches of 20?
overhead, per_row = 6, 1
print([round(size * 1000 / (overhead + per_row * size)) for size in (1, 20)])
  1. [143, 769]
  2. [167, 769]
  3. [1000, 1000]
  4. [143, 2857]
Show the answer

[143, 769]

A call of one row takes 7 ms: 1,000 ÷ 7 ≈ 143 a second. A call of 20 rows takes 26 ms and finishes 20 requests: 20 × 1,000 ÷ 26 ≈ 769 a second. The overhead is shared by twenty requests, so capacity rises more than five times.

Drill

Group the waiting requests

Write batches(arrivals, call_ms, max_size). arrivals is a sorted list of the times, in ms, at which requests arrive at one server. The server follows the policy "batch what waits": whenever it is free and at least one request has arrived, it takes every request that has arrived by then, up to max_size, and is busy for call_ms. If it is free and nothing is waiting, it starts as soon as the next request arrives. Return the list of batches, each a list of arrival times.

def batches(arrivals, call_ms, max_size):
    return [[a] for a in arrivals]


print(batches([0, 1, 2, 3, 30], call_ms=10, max_size=32))
In the lesson, your code is checked as you write it.

The same table as lines, on a log axis so the cliff and the flat batching curve share a plot:

import matplotlib.pyplot as plt

loads = [50, 100, 300]
one = [21.8, 79.1, 14549.2]
waits = [16.2, 16.9, 19.3]
hold = [19.5, 19.5, 21.0]
for name, ys in (
    ("one at a time", one), ("batch what waits", waits), ("batch, wait 10 ms", hold)
):
    plt.plot(loads, ys, marker="o", label=name)
plt.yscale("log")
plt.xlabel("requests per second")
plt.ylabel("p95 latency (ms, log)")
plt.legend()
plt.show()
In your own words
Why does batching whatever is waiting help most exactly when a server is busiest?

Write a sentence or two, then compare it with a good answer in the lesson.

The model call in the cells above is a stand-in. With a real PyTorch model the same win comes from stacking rows into one tensor, with the model in eval() mode and gradients off. This academy's torch is a teaching subset (same code, CPU only):

import torch
from torch import nn

torch.manual_seed(0)
model = nn.Linear(4, 1)
model.eval()
rows = torch.randn(64, 4)
with torch.no_grad():
    stacked = model(rows)
    looped = torch.cat([model(r.reshape(1, 4)) for r in rows])
print(stacked.shape, bool(((stacked - looped).abs() < 1e-5).all()))

The stacked call and the loop give the same answers; only the number of calls differs.

What each mode costs

An online service has to be running whenever a request might come, and it has to be big enough for the busiest minute of the day. For the rest of the day most of that capacity sits idle, and idle machines cost the same as busy ones. A batch job is the opposite: a machine is started, works at full speed until the rows are done, and is turned off.

def cost_per_million(price_per_hour, machines, hours, predictions):
    return price_per_hour * machines * hours / predictions * 1_000_000

predictions = 5_000_000                     # one score for each customer, each day
price = 0.50                                # dollars per machine-hour

batch_hours = predictions / 2000 / 3600     # one machine scoring 2,000 rows a second
batch = cost_per_million(price, 1, batch_hours, predictions)
online = cost_per_million(price, 2, 24, predictions)
print("batch job:", round(batch_hours * 60), "minutes on 1 machine |", round(batch, 2),
      "dollars per million predictions")
print("online:    24 hours on 2 machines   |", round(online, 2),
      "dollars per million predictions")
print("online costs", round(online / batch), "times as much, to serve an average of",
      round(predictions / 86400), "requests a second")

A million predictions cost 7 cents in batch and 4 dollars 80 online: 69 times as much. The batch machine works for 42 minutes and stops. The online pair, two so that one can fail, runs all day to serve an average of 58 requests a second, a small fraction of what it could do.

That is the price of freshness, and it's often worth paying. But it shouldn't be paid for predictions that didn't need it.

The hybrid, and its freshness limit

Many products use both. A batch job scores everyone it can ahead of time. The service looks the answer up, which takes a millisecond or two, and only runs the model live when there is no stored answer or the stored one is too old. How old is too old is a number you choose, the freshness limit.

def score(customer, now):                   # stands in for a model that takes 40 ms
    return round((len(customer) * 7 + now) % 100 / 100, 2)

# the nightly job, at hour 0
table = {name: {"score": score(name, 0), "at": 0} for name in ("ana", "ben", "cy")}

def serve(customer, now, max_age):
    entry = table.get(customer)
    if entry is not None and now - entry["at"] <= max_age:
        return entry["score"], "lookup", 2      # milliseconds
    table[customer] = {"score": score(customer, now), "at": now}
    return table[customer]["score"], "live", 40

for customer, hour in (("ana", 3), ("dee", 3), ("dee", 5), ("ben", 30), ("ben", 31)):
    print("hour", hour, customer, serve(customer, hour, max_age=24))

Ana is found in the table: 2 ms. Dee signed up today and has no stored score, so hers is computed live, 40 ms, and kept, so that her next request is a lookup. Ben's stored score is 30 hours old, past the limit of 24, so it is computed again.

The stored score is a point-in-time feature in the sense of Feature pipelines: it says what was true when the job ran. Whether that is good enough depends on the product, which is why the freshness limit is a decision for the people who own the product and not a default to accept.

Quick check
A service looks up precomputed scores and falls back to the live model when a score is missing or older than the freshness limit. What does this print?
table = {"ana": {"score": 0.2, "at": 0}, "ben": {"score": 0.7, "at": 20}}
now, max_age = 30, 24

def source(customer):
    entry = table.get(customer)
    fresh = entry is not None and now - entry["at"] <= max_age
    return "lookup" if fresh else "live"

print([source(customer) for customer in ("ana", "ben", "cy")])
  1. ['lookup', 'lookup', 'live']
  2. ['live', 'lookup', 'live']
  3. ['lookup', 'live', 'live']
  4. ['live', 'live', 'live']
Show the answer

['live', 'lookup', 'live']

Ana's score is 30 hours old, past the limit of 24, so it is recomputed. Ben's is 10 hours old and is served from the table. Cy has no entry at all, so the model runs live.

Worked example

Worked example: choosing a mode from the numbers

A shop wants a "customers like you also bought" list on every product page. Five million page views a day, and the list must appear within 100 ms. Should it be batch, online or a hybrid? Put numbers on each option.

import math

views_per_day = 5_000_000
# share of views by customers scored in last night's job
known_share = 0.92
lookup_ms, live_ms = 2, 40
price = 0.50                            # dollars per machine-hour

def daily_cost(live_views):
    # the busiest minute is 5 times the average
    peak_per_second = live_views / 86400 * 5
    # 800 requests/s per machine, never fewer than 2
    machines = max(2, math.ceil(peak_per_second / 800))
    return machines * price * 24, machines

# the nightly job: 5 million rows at 2,000 a second
batch_job = round(price * 5_000_000 / 2000 / 3600, 2)
for name, live_views in (("online only", views_per_day),
                         ("hybrid", views_per_day * (1 - known_share))):
    cost, machines = daily_cost(live_views)
    lookups = views_per_day - live_views
    mean_ms = (lookups * lookup_ms + live_views * live_ms) / views_per_day
    extra = batch_job if lookups else 0
    print(name, "|", machines, "machines |", round(cost + extra, 2),
          "dollars a day | mean latency", round(mean_ms, 1), "ms")
print("batch only | misses", round((1 - known_share) * 100),
      "percent of views: new customers have no list")
In the lesson you can change this and run it.

Batch alone is cheapest but leaves eight views in a hundred with nothing to show, which the product can't accept. Online alone serves everyone, with every view paying the model's 40 ms. The hybrid looks up the 92% it scored last night and runs the model only for the rest: the mean latency drops from 40 ms to about 5, and both options still need two machines, since two is the minimum for a service that mustn't go down with one failure. At this size the hybrid's gain is speed, not money. It starts saving money when the traffic grows large enough that the online-only service needs more machines than the hybrid does.

In your own words
Why does a server's latency get much worse before it reaches its throughput limit?

Write a sentence or two, then compare it with a good answer in the lesson.

On the job: a serving endpoint, a load test and capacity

Capacity planning is these sums done before a launch: how many requests a second at peak, and how many machines that needs.

Where it comes up in a week

Before a launch, the product team says peak traffic will be 300 requests a second. You measure the model's call time, work out the limit per machine, keep utilization under about 70% at peak, add one spare, and run a load test at 1x, 2x and 3x the expected peak while watching p95. If the answer is "it falls off a cliff at 2x", the fix may be micro-batching, a cache for repeat customers, or more machines, and the tradeoff is the cost-per-million sum from this lesson.

Goes deeper in

In your own words
A colleague proposes serving every prediction online "to keep things simple". What would you want to know before agreeing?

Write a sentence or two, then compare it with a good answer in the lesson.

Practice what you just read

  • 8 code examples you can edit and run
  • a practice exercise that checks your code
  • a 5-question quiz
  • 12 review cards for spaced repetition

Try a quiz question

Which fact forces a prediction to be made online and not in a batch job?

  1. Its inputs don't exist until the request arrives
  2. The model behind it is very large and slow
  3. It is needed for millions of customers each day
  4. Its result will be shown on a web page

Answer it at the end of the lesson to see if you're right, and why.

Before this lesson

Keep learning

Where this lesson leads.

575 lessons in thirteen phases. 176 are free without an account, and a free account opens the rest. See every lesson.

Phase 11 · about 34 hours of study

Professional practice

Design, ship and run real systems, work well in a team, and get hired 43 lessons 1 free
  1. System design fundamentals FreeRequirements, rough numbers, the standard parts and where systems break.40 min
  2. Caches, queues and background jobsSpeed reads with caches, and move slow work to queues that retry safely.40 min
  3. Designing AI systemsLatency, cost, routing, fallbacks and guardrails for features built on models.40 min
  4. Containers and deploymentPackage an app as an image, configure it with the environment, and roll out safely.45 min
  5. CI/CD pipelinesTest every change automatically, and ship with gates, flags and rollbacks.40 min
  6. Docker ComposeSeveral containers together: service names, volumes, secrets and start order.40 min
  7. Kubernetes basicsDesired state, Deployments and Services, labels, probes and rolling updates.40 min
  8. Cloud for AI engineersAccess policies and least privilege, storage, compute choices, managed model endpoints, secrets and budgets.30 min
  9. Metrics, dashboards and alert rulesCounters, gauges and histograms, a rate from a counter, a percentile from buckets, a dashboard panel and an alert rule that fires on a sustained condition.40 min
  10. Reliability and incidentsSLOs, error budgets, alerts, on-call and blameless postmortems.40 min
  11. Incident lab: from the page to the postmortemOne incident run from the page to the all-clear: decide the mitigation, work out the impact in failed requests and error budget, and write a blameless postmortem, a status update and a runbook entry.40 min
  12. Measuring an AI feature from its logsTyped views over event logs, metric specs, cost from a price table, and rollups you can rerun.40 min
  13. A/B testing a model changeHash assignment, per-user results from raw logs, sample ratio checks, guardrails and a decision rule.40 min
  14. Columnar files and a local warehouseRows against columns, the encodings that shrink a column, Parquet's row groups and the ranges that let a query skip them, and SQL over files with DuckDB.40 min
  15. Data quality checks and contractsSix properties of a delivery you can test, each as a count of rows that break it, a contract with the producer, and when to block, quarantine or warn.40 min
  16. Transformations: staging, models and testsRaw, staging, facts and marts as SQL models run in dependency order in DuckDB, tests as queries that count the rows that break a rule, an incremental load that is safe to rerun, and the same models as dbt files.45 min
  17. Streams and windowsEvents that never stop arriving: event time against arrival time, tumbling, sliding and session windows, a watermark for late events, and counting each event once.40 min
  18. Labels and annotator agreementLabels as measurements: agreement and Cohen's kappa, the pair of classes annotators confuse, votes and gold questions, and what wrong labels do to a model and a test score.45 min
  19. Experiment tracking and reproducible runsA run record of settings, seed, data and code behind every score, a run log in a dozen lines, and comparing settings across seeds.40 min
  20. Hyperparameter searchWhat a grid costs, random search on a log scale, stopping weak settings early, and why the winner's validation score flatters it.40 min
  21. Versioning data and modelsFiles stored under the hash of their contents, a manifest that names a whole dataset, and lineage from data and code to each model.35 min
  22. A model registry and promotionVersions under one name, an alias for the live one, gates a candidate must pass, group by group, and a rollback that moves a pointer.40 min
  23. Feature pipelines and training-serving skewWhy a model that passes offline fails live: statistics refit at serving, two code paths, and features from the future, with the fix for each.40 min
  24. Batch and online inferencePredict ahead of time or on request: when the inputs exist, latency near a server's limit, batching what waits, cost, and a hybrid with a freshness limit.40 min
  25. Monitoring a model: drift and decayWhy a model worsens with no code change: data drift and concept drift, a measure of how far inputs have moved, thresholds set from noise, and labels that arrive late.45 min
  26. Retraining and pipelinesWhen to retrain and on which weeks, training as steps in dependency order, a cache that reruns only what changed, and guards before and after.45 min
  27. Designing ML systemsOne method for an ML design, goal and metric to retraining, with the sums that size it: requests a second at peak, a year of feature storage, a label delay and a latency budget.45 min
  28. Datasheets: documenting a datasetThe document that travels with a dataset: a profile by column and by group, claims a program can check, splits that don't leak, a scan for personal data, and the license and limits only people can write.45 min
  29. Curation, active learning and synthetic dataThree levers on a training set: removing copies and broken rows, letting a model choose which rows to label, and manufacturing rows, which helps only when their source knows more than you do.45 min
  30. Measuring fairnessBreak a model's results down by group: selection rate, true and false positive rates and precision, why they can't all be equal, what each fix costs, and when a group is too small to judge.40 min
  31. Privacy engineeringWhy removing names isn't anonymity, how to hold less (minimization, retention, keyed pseudonyms), the idea behind differential privacy and its budget, what a model remembers, and a deletion that reaches every store.40 min
  32. A red-team labAttack a simulated assistant methodically: eight kinds of attack run as a suite, success rates beside ordinary requests blocked, fresh attacks to test the fixes, where a defense belongs, and findings kept as release tests.45 min
  33. What regulation asks of an engineerRisk tiers that follow a system's use, the requirements for high-risk systems as artifacts an engineer builds, evidence tied to a model version, a model card a program can check, and legal dates kept as data.35 min
  34. Git for teamsBranches, pull requests, rebasing and how a three-way merge finds conflicts.40 min
  35. Writing for engineersREADMEs, design docs, commit messages and bug reports people can act on.35 min
  36. Scoping and estimating workA vague request cut into tickets with acceptance criteria and a first shippable slice, estimates as ranges, an honest update for a slip, and a ticket linked to its pull request.30 min
  37. Roles, résumé and the searchWhat AI, ML and data engineering postings ask for, résumé bullets that carry a number, a 100-word project summary, a take-home as a portfolio piece and how to read an offer.35 min
  38. Interviews and your portfolioShow your work, and solve interview problems the way professionals do.40 min
  39. Code reviewWhat to look for, how to give and take feedback, and automating the routine checks.35 min
  40. Reading unfamiliar codeFind your way around a big codebase fast, with tools that map it for you.35 min
  41. Working with AI coding toolsGive agents good context, keep tasks small, and check every change they make.35 min
  42. Capstone: ship a production AI serviceAn AI-backed service with validation, caching, rate limits, fallbacks, logging and evals.45 min
  43. Capstone: take a model to productionA churn model through a data contract, a tracked run, a registry gate by slice, hybrid serving, a PSI monitor and one retrain, shipped with a datasheet, a model card, a fairness audit, a design doc and a runbook.45 min
Part of a free course 11 lessons

Free MLOps course: running machine learning in production

Take a model from a notebook to a system that stays right: track and reproduce every run, search settings honestly, version data and models, promote through gates, keep training and serving in step, choose batch or online, catch drift, and retrain through a pipeline. Every number is measured by the lesson's own code.

Never trained an AI? Teach a small one to write, right in your browser. It takes about 20 seconds and needs no account. Teach it to write