Predict ahead of time or on request: when the inputs exist, latency near a server's limit, batching what waits, cost, and a hybrid with a freshness limit.
A model in a notebook predicts when you press Run. A model in a product has to predict at some particular moment, for someone, at some cost. Do you score every customer overnight and store the answers, or compute each answer when it is asked for? The first is cheap and can be stale. The second is fresh and must answer in milliseconds, at 3 a.m., under load. Most of the cost and most of the outages of a deployed model follow from this choice, and it is usually made by accident. This lesson makes it on purpose.
What you'll be able to do
Tell batch inference from online inference, and choose between them by when the inputs exist and how fresh the answer must be
Read a server's latency at the median and at the 95th percentile, and see what happens when requests arrive faster than it can serve
Explain how grouping waiting requests raises a server's capacity, and what a fixed wait costs
Work out the cost per million predictions for each mode, and design a hybrid with a freshness limit
Quick check
A guess before you start. A model server handles one request at a time and takes 8.5 milliseconds for each. Requests begin arriving at 300 per second. What happens to the waiting time?
It grows without limit: the server can finish only about 118 per second
It falls, since a busy server wastes less time between requests
It settles at about 8.5 milliseconds, the time one request takes
It settles at about 25 milliseconds, three requests' worth of work
Show the answer
It grows without limit: the server can finish only about 118 per second
One request per 8.5 ms is about 118 per second. At 300 per second, 182 more arrive every second than can leave, so the queue, and every new request's wait, keeps growing for as long as the load lasts. The lesson's simulation shows waits of several seconds within ten seconds.
Batch inference runs the model ahead of time, on a schedule, over many rows at once. The predictions are written to a table, and the product reads them from there. The nightly job that scores every customer's churn risk is batch inference.
Online inference runs the model when a request arrives, and the caller waits for the answer. The fraud check that runs while a card payment is pending is online inference.
Two questions decide between them.
When do the inputs exist? A churn score depends on the customer's history, which is already in the warehouse tonight. A fraud score depends on the payment, which doesn't exist until someone makes it. If the inputs only arrive with the request, you have no choice: it's online.
How fresh must the answer be? If a score computed last night is still a good answer this afternoon, batch will do. If it must reflect what happened a minute ago, it won't.
Batch is the cheaper and simpler mode by a wide margin, so the working rule is: use batch unless one of those two questions rules it out.
The speed difference between scoring rows together and one at a time is large, even for a small model:
import time
import numpy as np
from sklearn.linear_model import LogisticRegression
rng = np.random.default_rng(0)
X = rng.normal(size=(2000, 20))
y = (X[:, 0] + X[:, 1] > 0).astype(int)
model = LogisticRegression().fit(X, y)
start = time.perf_counter()
together = model.predict(X) # one call, 2,000 rows
batch_seconds = time.perf_counter() - start
start = time.perf_counter()
# 2,000 calls, one row each
one_by_one = [int(model.predict(X[i:i + 1])[0]) for i inrange(2000)]
single_seconds = time.perf_counter() - start
print("same predictions:", together.tolist() == one_by_one)
print("one call was at least 10 times faster:", single_seconds > 10 * batch_seconds)
It runs right here in your browser, with nothing to install.
The arithmetic is the same either way. What differs is the overhead: every call pays a fixed price to check its input and set up the computation, before any row is scored. One call pays it once. Two thousand calls pay it two thousand times. Keep that fixed price in mind. It is the reason for everything in the third part of this lesson.
Try it now
Ask when the inputs exist and how old an answer may be. If neither forces your hand, choose batch.
Quick check
A team wants a model to estimate delivery time, shown to the customer on the checkout page. The estimate depends on what is in the basket and how busy the couriers are right now. Which mode fits?
Neither: a model can't be used for a number shown to customers
Batch: delivery estimates can be computed for everyone overnight
Batch: an overnight table is cheaper, and cost comes first
Online: the basket and the current load exist only at request time
Show the answer
Online: the basket and the current load exist only at request time
Both inputs, the basket and the couriers' load at this moment, come into existence with the request. Nothing computed last night could have known them, so the prediction has to be made online.
Latency, and the edge of the cliff
An online model is judged on two numbers. Latency is how long one request waits for its answer. Throughput is how many requests the server finishes per second. You met both in Metrics, logs and traces and Serving language models, along with the habit of reading latency at the 95th percentile, p95, because an average hides the slow requests that users remember.
A server has a limit. If one request takes 8.5 ms, the server can finish about 118 a second and no more. The next cell simulates one server under three loads. Requests arrive at random moments, as real ones do, and each waits its turn.
import math
import numpy as np
defpercentile(values, p):
ordered = sorted(values)
return ordered[math.ceil(p / 100 * len(ordered)) - 1]
defserve(arrivals, max_size, max_wait, overhead=8.0, per_row=0.5):
"""One model server. Times in milliseconds. Returns each request's latency."""
latencies, free_at, i = [], 0.0, 0while i < len(arrivals):
# the server is free and a request is waiting
start = max(free_at, arrivals[i])
# optionally hold the batch open a little
close = max(start, arrivals[i] + max_wait)
j = i
while j < len(arrivals) and j - i < max_size and arrivals[j] <= close:
j += 1
begin = max(start, arrivals[j - 1]) if j - i == max_size else close
# one call: fixed overhead, plus a little per row
done = begin + overhead + per_row * (j - i)
latencies += [done - a for a in arrivals[i:j]]
free_at, i = done, j
return latencies
for rate in (50, 100, 300):
rng = np.random.default_rng(0)
arrivals = np.cumsum(rng.exponential(1000 / rate, size=3000)).tolist()
latency = serve(arrivals, max_size=1, max_wait=0) # one request per callprint(rate, "requests/s | busy", round(rate * 8.5 / 1000, 2),
"| p50", round(percentile(latency, 50), 1),
"ms | p95", round(percentile(latency, 95), 1), "ms")
At 50 requests a second the median request waits 8.5 ms: it arrives, finds the server free, and is served. The p95 is already 21.8 ms, because requests sometimes arrive in a cluster and queue behind each other.
At 100 a second, still under the limit of 118, the p95 has climbed to 79.1 ms. The busy column says why: the server is working 85% of the time, so a cluster takes a long while to clear.
At 300 a second the server would have to be busy 2.55 times over. The median wait is more than seven seconds and still rising. This is not a slow server. It is a server past its limit, where the queue grows for as long as the load lasts.
Notice the shape: fine, fine, a little worse, then a cliff. Latency gives little warning before throughput runs out, which is why teams watch how close to the limit a server runs, and keep spare capacity.
Quick check
From the observability lesson: what does this print?
Sorted, the fifth of ten values is 9, the median. The 95th percentile is the tenth value, 120: the one slow request. The mean, 20.0, describes none of the requests: nine were faster and one was far slower.
Quick check
A server takes 20 ms per request and handles one at a time. Requests arrive at a steady 40 per second. What should the team expect?
It is nearly idle, since 40 per second is a light load
It falls behind at once, since 40 is more than 20
Its latency will be exactly 20 ms for every request
It keeps up, but with little room: its limit is 50 per second
Show the answer
It keeps up, but with little room: its limit is 50 per second
At 20 ms each, the server finishes 1,000 ÷ 20 = 50 requests a second. Forty a second is 80% of that. It keeps up on average, but any burst queues, so the p95 will sit well above 20 ms.
In your own words
In your own words: what two questions decide between batch and online inference?
Write a sentence or two, then compare it with a good answer in the lesson.
In code: groups of requests, a PyTorch batch and the cost of each mode
Every cell simulates a server in a few lines of Python, so the cliff in the latency curve is something you can move.
Drill
How close to the limit?
Write load_report(ms_per_request, requests_per_second). Return a dictionary with "limit", the most requests the server can finish per second (1,000 divided by the milliseconds each takes), "utilization", the arrival rate as a share of that limit, rounded to 2 places, and "verdict": "overloaded" if utilization is 1 or more, "at risk" if it is 0.7 or more, and "ok" otherwise.
In the lesson, your code is checked as you write it.
Serving requests in groups
The first part of this lesson showed that one call for many rows is far cheaper than many calls for one row, because the fixed overhead is paid once. A server can use that. When several requests are waiting, it scores them together in one call. This is micro-batching: the serving lesson's static batching, applied to any model.
In the simulation a call costs 8 ms of overhead plus 0.5 ms a row. Alone, a request takes 8.5 ms. In a batch of 32, the call takes 24 ms and each request's share is 0.75 ms: the same server can now finish over a thousand requests a second.
Here are three policies under the same three loads. The first is one request per call. The second takes whatever is waiting when the server becomes free, up to 32. The third also holds each batch open for 10 ms, hoping to fill it.
import math
import numpy as np
defpercentile(values, p):
ordered = sorted(values)
return ordered[math.ceil(p / 100 * len(ordered)) - 1]
defserve(arrivals, max_size, max_wait, overhead=8.0, per_row=0.5):
latencies, free_at, i = [], 0.0, 0while i < len(arrivals):
start = max(free_at, arrivals[i])
close = max(start, arrivals[i] + max_wait)
j = i
while j < len(arrivals) and j - i < max_size and arrivals[j] <= close:
j += 1
begin = max(start, arrivals[j - 1]) if j - i == max_size else close
done = begin + overhead + per_row * (j - i)
latencies += [done - a for a in arrivals[i:j]]
free_at, i = done, j
return latencies
for rate in (50, 100, 300):
rng = np.random.default_rng(0)
arrivals = np.cumsum(rng.exponential(1000 / rate, size=3000)).tolist()
for name, size, wait in (("one at a time ", 1, 0), ("batch what waits ", 32, 0),
("batch, wait 10 ms", 32, 10)):
latency = serve(arrivals, size, wait)
print(rate, "requests/s |", name, "| p50", round(percentile(latency, 50), 1),
"| p95", round(percentile(latency, 95), 1))
Read the middle policy down the three loads: a p95 of 16.2, 16.9 and 19.3 ms. At 300 requests a second, where the unbatched server was seconds behind, the batching one barely notices. When requests queue, it clears the whole queue in one call, so the busier it gets, the more efficient each call becomes.
Now the third policy. At 50 a second its median is 18.5 ms against 8.5: every request waited 10 ms for company that mostly didn't arrive. A fixed wait buys nothing at low load and costs latency on every request. It only pays when calls are so expensive that fuller batches matter more than the wait.
So the default is the middle policy: batch whatever is already waiting, and don't wait for more.
Quick check
A call to a model costs 6 ms of overhead plus 1 ms per row. What does this print: the most requests per second the server can finish with batches of 1, and with batches of 20?
overhead, per_row = 6, 1print([round(size * 1000 / (overhead + per_row * size)) for size in (1, 20)])
[143, 769]
[167, 769]
[1000, 1000]
[143, 2857]
Show the answer
[143, 769]
A call of one row takes 7 ms: 1,000 ÷ 7 ≈ 143 a second. A call of 20 rows takes 26 ms and finishes 20 requests: 20 × 1,000 ÷ 26 ≈ 769 a second. The overhead is shared by twenty requests, so capacity rises more than five times.
Drill
Group the waiting requests
Write batches(arrivals, call_ms, max_size). arrivals is a sorted list of the times, in ms, at which requests arrive at one server. The server follows the policy "batch what waits": whenever it is free and at least one request has arrived, it takes every request that has arrived by then, up to max_size, and is busy for call_ms. If it is free and nothing is waiting, it starts as soon as the next request arrives. Return the list of batches, each a list of arrival times.
defbatches(arrivals, call_ms, max_size):
return [[a] for a in arrivals]
print(batches([0, 1, 2, 3, 30], call_ms=10, max_size=32))
In the lesson, your code is checked as you write it.
The same table as lines, on a log axis so the cliff and the flat batching curve share a plot:
import matplotlib.pyplot as plt
loads = [50, 100, 300]
one = [21.8, 79.1, 14549.2]
waits = [16.2, 16.9, 19.3]
hold = [19.5, 19.5, 21.0]
for name, ys in (
("one at a time", one), ("batch what waits", waits), ("batch, wait 10 ms", hold)
):
plt.plot(loads, ys, marker="o", label=name)
plt.yscale("log")
plt.xlabel("requests per second")
plt.ylabel("p95 latency (ms, log)")
plt.legend()
plt.show()
In your own words
Why does batching whatever is waiting help most exactly when a server is busiest?
Write a sentence or two, then compare it with a good answer in the lesson.
The model call in the cells above is a stand-in. With a real PyTorch model the same win comes from stacking rows into one tensor, with the model in eval() mode and gradients off. This academy's torch is a teaching subset (same code, CPU only):
import torch
from torch import nn
torch.manual_seed(0)
model = nn.Linear(4, 1)
model.eval()
rows = torch.randn(64, 4)
with torch.no_grad():
stacked = model(rows)
looped = torch.cat([model(r.reshape(1, 4)) for r in rows])
print(stacked.shape, bool(((stacked - looped).abs() < 1e-5).all()))
The stacked call and the loop give the same answers; only the number of calls differs.
What each mode costs
An online service has to be running whenever a request might come, and it has to be big enough for the busiest minute of the day. For the rest of the day most of that capacity sits idle, and idle machines cost the same as busy ones. A batch job is the opposite: a machine is started, works at full speed until the rows are done, and is turned off.
defcost_per_million(price_per_hour, machines, hours, predictions):
return price_per_hour * machines * hours / predictions * 1_000_000
predictions = 5_000_000# one score for each customer, each day
price = 0.50# dollars per machine-hour
batch_hours = predictions / 2000 / 3600# one machine scoring 2,000 rows a second
batch = cost_per_million(price, 1, batch_hours, predictions)
online = cost_per_million(price, 2, 24, predictions)
print("batch job:", round(batch_hours * 60), "minutes on 1 machine |", round(batch, 2),
"dollars per million predictions")
print("online: 24 hours on 2 machines |", round(online, 2),
"dollars per million predictions")
print("online costs", round(online / batch), "times as much, to serve an average of",
round(predictions / 86400), "requests a second")
A million predictions cost 7 cents in batch and 4 dollars 80 online: 69 times as much. The batch machine works for 42 minutes and stops. The online pair, two so that one can fail, runs all day to serve an average of 58 requests a second, a small fraction of what it could do.
That is the price of freshness, and it's often worth paying. But it shouldn't be paid for predictions that didn't need it.
The hybrid, and its freshness limit
Many products use both. A batch job scores everyone it can ahead of time. The service looks the answer up, which takes a millisecond or two, and only runs the model live when there is no stored answer or the stored one is too old. How old is too old is a number you choose, the freshness limit.
defscore(customer, now): # stands in for a model that takes 40 msreturnround((len(customer) * 7 + now) % 100 / 100, 2)
# the nightly job, at hour 0
table = {name: {"score": score(name, 0), "at": 0} for name in ("ana", "ben", "cy")}
defserve(customer, now, max_age):
entry = table.get(customer)
if entry isnotNoneand now - entry["at"] <= max_age:
return entry["score"], "lookup", 2# milliseconds
table[customer] = {"score": score(customer, now), "at": now}
return table[customer]["score"], "live", 40for customer, hour in (("ana", 3), ("dee", 3), ("dee", 5), ("ben", 30), ("ben", 31)):
print("hour", hour, customer, serve(customer, hour, max_age=24))
Ana is found in the table: 2 ms. Dee signed up today and has no stored score, so hers is computed live, 40 ms, and kept, so that her next request is a lookup. Ben's stored score is 30 hours old, past the limit of 24, so it is computed again.
The stored score is a point-in-time feature in the sense of Feature pipelines: it says what was true when the job ran. Whether that is good enough depends on the product, which is why the freshness limit is a decision for the people who own the product and not a default to accept.
Quick check
A service looks up precomputed scores and falls back to the live model when a score is missing or older than the freshness limit. What does this print?
Ana's score is 30 hours old, past the limit of 24, so it is recomputed. Ben's is 10 hours old and is served from the table. Cy has no entry at all, so the model runs live.
Worked example
Worked example: choosing a mode from the numbers
A shop wants a "customers like you also bought" list on every product page. Five million page views a day, and the list must appear within 100 ms. Should it be batch, online or a hybrid? Put numbers on each option.
import math
views_per_day = 5_000_000# share of views by customers scored in last night's job
known_share = 0.92
lookup_ms, live_ms = 2, 40
price = 0.50# dollars per machine-hourdefdaily_cost(live_views):
# the busiest minute is 5 times the average
peak_per_second = live_views / 86400 * 5# 800 requests/s per machine, never fewer than 2
machines = max(2, math.ceil(peak_per_second / 800))
return machines * price * 24, machines
# the nightly job: 5 million rows at 2,000 a second
batch_job = round(price * 5_000_000 / 2000 / 3600, 2)
for name, live_views in (("online only", views_per_day),
("hybrid", views_per_day * (1 - known_share))):
cost, machines = daily_cost(live_views)
lookups = views_per_day - live_views
mean_ms = (lookups * lookup_ms + live_views * live_ms) / views_per_day
extra = batch_job if lookups else0print(name, "|", machines, "machines |", round(cost + extra, 2),
"dollars a day | mean latency", round(mean_ms, 1), "ms")
print("batch only | misses", round((1 - known_share) * 100),
"percent of views: new customers have no list")
In the lesson you can change this and run it.
Batch alone is cheapest but leaves eight views in a hundred with nothing to show, which the product can't accept. Online alone serves everyone, with every view paying the model's 40 ms. The hybrid looks up the 92% it scored last night and runs the model only for the rest: the mean latency drops from 40 ms to about 5, and both options still need two machines, since two is the minimum for a service that mustn't go down with one failure. At this size the hybrid's gain is speed, not money. It starts saving money when the traffic grows large enough that the online-only service needs more machines than the hybrid does.
In your own words
Why does a server's latency get much worse before it reaches its throughput limit?
Write a sentence or two, then compare it with a good answer in the lesson.
On the job: a serving endpoint, a load test and capacity
Capacity planning is these sums done before a launch: how many requests a second at peak, and how many machines that needs.
Where it comes up in a week
Before a launch, the product team says peak traffic will be 300 requests a second. You measure the model's call time, work out the limit per machine, keep utilization under about 70% at peak, add one spare, and run a load test at 1x, 2x and 3x the expected peak while watching p95. If the answer is "it falls off a cliff at 2x", the fix may be micro-batching, a cache for repeat customers, or more machines, and the tradeoff is the cost-per-million sum from this lesson.
03Designing AI systemsLatency, cost, routing, fallbacks and guardrails for features built on models.40 min
04Containers and deploymentPackage an app as an image, configure it with the environment, and roll out safely.45 min
05CI/CD pipelinesTest every change automatically, and ship with gates, flags and rollbacks.40 min
06Docker ComposeSeveral containers together: service names, volumes, secrets and start order.40 min
07Kubernetes basicsDesired state, Deployments and Services, labels, probes and rolling updates.40 min
08Cloud for AI engineersAccess policies and least privilege, storage, compute choices, managed model endpoints, secrets and budgets.30 min
09Metrics, dashboards and alert rulesCounters, gauges and histograms, a rate from a counter, a percentile from buckets, a dashboard panel and an alert rule that fires on a sustained condition.40 min
11Incident lab: from the page to the postmortemOne incident run from the page to the all-clear: decide the mitigation, work out the impact in failed requests and error budget, and write a blameless postmortem, a status update and a runbook entry.40 min
13A/B testing a model changeHash assignment, per-user results from raw logs, sample ratio checks, guardrails and a decision rule.40 min
14Columnar files and a local warehouseRows against columns, the encodings that shrink a column, Parquet's row groups and the ranges that let a query skip them, and SQL over files with DuckDB.40 min
15Data quality checks and contractsSix properties of a delivery you can test, each as a count of rows that break it, a contract with the producer, and when to block, quarantine or warn.40 min
16Transformations: staging, models and testsRaw, staging, facts and marts as SQL models run in dependency order in DuckDB, tests as queries that count the rows that break a rule, an incremental load that is safe to rerun, and the same models as dbt files.45 min
17Streams and windowsEvents that never stop arriving: event time against arrival time, tumbling, sliding and session windows, a watermark for late events, and counting each event once.40 min
18Labels and annotator agreementLabels as measurements: agreement and Cohen's kappa, the pair of classes annotators confuse, votes and gold questions, and what wrong labels do to a model and a test score.45 min
19Experiment tracking and reproducible runsA run record of settings, seed, data and code behind every score, a run log in a dozen lines, and comparing settings across seeds.40 min
20Hyperparameter searchWhat a grid costs, random search on a log scale, stopping weak settings early, and why the winner's validation score flatters it.40 min
21Versioning data and modelsFiles stored under the hash of their contents, a manifest that names a whole dataset, and lineage from data and code to each model.35 min
22A model registry and promotionVersions under one name, an alias for the live one, gates a candidate must pass, group by group, and a rollback that moves a pointer.40 min
23Feature pipelines and training-serving skewWhy a model that passes offline fails live: statistics refit at serving, two code paths, and features from the future, with the fix for each.40 min
24Batch and online inferencePredict ahead of time or on request: when the inputs exist, latency near a server's limit, batching what waits, cost, and a hybrid with a freshness limit.40 min
25Monitoring a model: drift and decayWhy a model worsens with no code change: data drift and concept drift, a measure of how far inputs have moved, thresholds set from noise, and labels that arrive late.45 min
26Retraining and pipelinesWhen to retrain and on which weeks, training as steps in dependency order, a cache that reruns only what changed, and guards before and after.45 min
27Designing ML systemsOne method for an ML design, goal and metric to retraining, with the sums that size it: requests a second at peak, a year of feature storage, a label delay and a latency budget.45 min
28Datasheets: documenting a datasetThe document that travels with a dataset: a profile by column and by group, claims a program can check, splits that don't leak, a scan for personal data, and the license and limits only people can write.45 min
29Curation, active learning and synthetic dataThree levers on a training set: removing copies and broken rows, letting a model choose which rows to label, and manufacturing rows, which helps only when their source knows more than you do.45 min
30Measuring fairnessBreak a model's results down by group: selection rate, true and false positive rates and precision, why they can't all be equal, what each fix costs, and when a group is too small to judge.40 min
31Privacy engineeringWhy removing names isn't anonymity, how to hold less (minimization, retention, keyed pseudonyms), the idea behind differential privacy and its budget, what a model remembers, and a deletion that reaches every store.40 min
32A red-team labAttack a simulated assistant methodically: eight kinds of attack run as a suite, success rates beside ordinary requests blocked, fresh attacks to test the fixes, where a defense belongs, and findings kept as release tests.45 min
33What regulation asks of an engineerRisk tiers that follow a system's use, the requirements for high-risk systems as artifacts an engineer builds, evidence tied to a model version, a model card a program can check, and legal dates kept as data.35 min
34Git for teamsBranches, pull requests, rebasing and how a three-way merge finds conflicts.40 min
35Writing for engineersREADMEs, design docs, commit messages and bug reports people can act on.35 min
36Scoping and estimating workA vague request cut into tickets with acceptance criteria and a first shippable slice, estimates as ranges, an honest update for a slip, and a ticket linked to its pull request.30 min
37Roles, résumé and the searchWhat AI, ML and data engineering postings ask for, résumé bullets that carry a number, a 100-word project summary, a take-home as a portfolio piece and how to read an offer.35 min
43Capstone: take a model to productionA churn model through a data contract, a tracked run, a registry gate by slice, hybrid serving, a PSI monitor and one retrain, shipped with a datasheet, a model card, a fairness audit, a design doc and a runbook.45 min
Take a model from a notebook to a system that stays right: track and reproduce every run, search settings honestly, version data and models, promote through gates, keep training and serving in step, choose batch or online, catch drift, and retrain through a pipeline. Every number is measured by the lesson's own code.
See the course →
Never trained an AI? Teach a small one to write, right in your browser. It takes about 20 seconds and needs no account. Teach it to write