Free Python developer course
Go from your first line of code to working as a Python developer: projects, testing, data, databases, web apps and APIs, all in your browser.
Load once with lifespan, a typed predict route, a batch route, a cap on work in flight, 429 with Retry-After, a readiness route and a 202 job.
APIs with FastAPI named model serving as FastAPI's main use. This lesson does it. The shape is simple: load the model once, answer many requests, and protect it, because a model is the most expensive thing in the service and a route that lets any caller queue unlimited work is an outage waiting for a bad afternoon. The API side is what you build here: types, routes, limits and statuses. One request through a serving stack: latency and cost owns what happens on the GPU, and Batch and online inference owns running it in production.
Every request pays the load time, so replies are slow
Loading is the slow part. A route that loads for each request pays it on every call. Load once at startup, and let every request reuse the object.
This rests on APIs with FastAPI, where a lifespan function loaded a model once, and Rate limits, retries and idempotency, where a token bucket limited a client. A served model has a shape you can draw. At startup the service loads the model and, until it is done, says it is not ready. Then each request arrives, waits in line if the model is busy, runs, and returns. If the model can do several requests in one pass, a batch shares the fixed cost of a pass: a batch of 8 may cost 120 ms where eight single calls cost 400 ms, so waiting a moment to fill a batch trades a little latency for much more throughput.
Two numbers describe a service. Latency is how long one request takes, from arrival to reply, including its wait. Throughput is how many requests finish each second. They pull against each other, and a single average hides the problem: the slow ones are what users remember, so teams read p50 (half of requests are faster) and p95 (95 percent are faster). The picture queues six requests, each needing 50 ms, on one worker that handles one at a time.
All six arrive together. The last one waits for five others before its own 50 ms starts.
tokens, capacity, rate = 3.0, 3.0, 1.0 for t in [0.0, 0.0, 0.0, 0.4]: tokens = min(capacity, tokens + rate * (0.4 if t == 0.4 else 0)) if tokens >= 1: tokens -= 1 print("allowed", end=" ") else: print("429", end=" ")
allowed allowed allowed 429
The first three spend the three tokens. By the fourth, 0.4 s has dripped back, which is 0.4 of a token, and a request needs a whole one, so it is refused with 429.
Write a sentence or two, then compare it with a good answer in the lesson.
The practice customers table is the data: 600 customers, and a churned column to predict. The service fits a small logistic regression once, in lifespan, and every request reuses it. The route takes a pydantic model in and gives one out, so a bad request gets a 422 before the model runs. serve runs the app's startup once and then answers calls, the way a server would.
The model is fit at startup from the practice customers. The route validates the body, and returns a probability.
The lesson runs this once, before its first cell, and every cell, drill and example starts with what it defines. A cell that changes a setup object changes it for the cells after it, until the page reloads.
import asyncio import time import numpy as np import requests BASE = "https://api.gradient.test" FEATURES = [ "tenure_months", "orders_last_year", "avg_order_value", "days_since_last_order", ] def percentile(values, q): """Nearest-rank percentile: the smallest value with q percent at or below it.""" ordered = sorted(values) rank = max(1, -(-len(ordered) * q // 100)) return ordered[int(rank) - 1]
from contextlib import asynccontextmanager from devserver import serve from fastapi import FastAPI from pydantic import BaseModel, Field from sklearn.linear_model import LogisticRegression from sklearn.preprocessing import StandardScaler from sklearn.pipeline import make_pipeline STATE = {"model": None, "loads": 0} @asynccontextmanager async def lifespan(app): rows = requests.get(f"{BASE}/v1/customers", params={"limit": 600}).json()["data"] X = np.array([[r[f] for f in FEATURES] for r in rows], dtype=float) y = np.array([r["churned"] for r in rows]) STATE["model"] = make_pipeline(StandardScaler(), LogisticRegression()).fit(X, y) STATE["loads"] += 1 yield STATE["model"] = None class CustomerIn(BaseModel): tenure_months: float = Field(ge=0) orders_last_year: float = Field(ge=0) avg_order_value: float = Field(ge=0) days_since_last_order: float = Field(ge=0) class Prediction(BaseModel): churn_probability: float api = FastAPI(lifespan=lifespan) @api.post("/predict") def predict(customer: CustomerIn) -> Prediction: x = np.array([[getattr(customer, f) for f in FEATURES]]) return Prediction(churn_probability=float(STATE["model"].predict_proba(x)[0, 1])) base = serve(api) good = {"tenure_months": 6, "orders_last_year": 2, "avg_order_value": 40, "days_since_last_order": 200} r = requests.post(f"{base}/predict", json=good) print(r.status_code, 0 <= r.json()["churn_probability"] <= 1) r = requests.post(f"{base}/predict", json={**good, "tenure_months": -1}) print(r.status_code) for _ in range(3): requests.post(f"{base}/predict", json=good) print("model loads:", STATE["loads"])
A valid customer got a probability between 0 and 1. A negative tenure never reached the model: the body gate answered 422. After four predictions the model had been loaded once, because lifespan ran at startup, not per request.
customer: CustomerIn where tenure_months: float = Field(ge=0). A request sends -1. What happens?422 before the model runs
The body is checked against the model first. A negative tenure breaks ge=0, so the response is 422 with the field's location, and the function, and so the model, never runs.
import numpy as np X = np.arange(6).reshape(3, 2) print(X.shape, X[0:1].shape)
(3, 2) (1, 2)
X[0:1] is a slice, so it keeps the row axis: shape (1, 2), a batch of one. That is why scoring one customer at a time still passes a 2-D array to predict_proba.
A batch route takes a list and scores it in one call to the model, which is where the fixed cost is shared. The cell scores 500 customers one at a time and then as one matrix, and compares the two. The model is fit in the setup of this cell's own run, since the cost to compare is the call, not the fit.
Uses the setup above, 16 lines.
from sklearn.linear_model import LogisticRegression rng = np.random.default_rng(0) X = rng.normal(size=(500, 4)) y = (X[:, 0] + rng.normal(size=500) > 0).astype(int) model = LogisticRegression().fit(X, y) start = time.perf_counter() one_by_one = [model.predict_proba(X[i:i + 1])[0, 1] for i in range(len(X))] t_single = time.perf_counter() - start start = time.perf_counter() batched = model.predict_proba(X)[:, 1] t_batch = time.perf_counter() - start print("same answers:", np.allclose(one_by_one, batched)) print("batch is faster:", t_batch < t_single)
The answers are the same. The batch is faster because the per-call overhead is paid once. Queued requests wait for a batch to fill, which is why real servers set a short batch window, such as 10 ms, and a maximum batch size.
A few slow requests drag the tail: one in twenty takes 900 ms or more
An average hides the slow ones. A p95 of 900 ms means 5 percent of requests take at least that long, and those are what users complain about.
Write pct(values, q): the nearest-rank percentile. Sort the values, find the rank ceil(len * q / 100) (at least 1), and return the value at that rank (counting from 1).
Uses the setup above, 16 lines.
def pct(values, q): return sum(values) / len(values) print(pct([40, 42, 45, 50, 52, 55, 60, 75, 120, 400], 95))
A simulation shows why the average lies. One hundred requests arrive at random over ten seconds, each needing 80 ms from one worker. The plot compares the latency of each request under a single queue and under a worker that scores a batch every 100 ms. The percentile helper is in this lesson's setup.
Uses the setup above, 16 lines.
import random import matplotlib.pyplot as plt rng = random.Random(1) arrivals = sorted(rng.uniform(0, 10) for _ in range(100)) free, queued = 0.0, [] for t in arrivals: start = max(t, free) free = start + 0.08 queued.append(free - t) batched, free = [], 0.0 for window in sorted({int(t * 10) for t in arrivals}): members = [t for t in arrivals if int(t * 10) == window] start = max((window + 1) / 10, free) free = start + 0.04 + 0.01 * len(members) batched += [free - t for t in members] fig, ax = plt.subplots(figsize=(6, 3)) ax.hist(queued, bins=20, alpha=0.6, label="one at a time, 80 ms each") ax.hist(batched, bins=20, alpha=0.6, label="batch every 100 ms") ax.set_xlabel("latency (seconds)") ax.set_ylabel("requests") ax.legend() for name, values in [("one at a time", queued), ("batched", batched)]: p50, p95 = percentile(values, 50), percentile(values, 95) print(name, "p50", round(p50, 2), "p95", round(p95, 2))
At this load the single worker is nearly saturated, so a queue builds, and p95 sits far above p50. The batched worker does the same work in less time and its p95 stays close to its p50. Change 0.08 to 0.1 and rerun: a worker that cannot keep up lets the queue, and every latency, grow without bound.
A cap protects the model. A request that arrives when the cap is full gets 429 with a Retry-After, so a polite client waits and tries again, and the service never queues unbounded work. The cell sends six requests together to a route that needs 50 ms each and allows three at once. It also adds a readiness route: until the model is loaded it answers 503, so a load balancer sends it nothing.
Uses the setup above, 16 lines.
import asyncio import httpx from fastapi import FastAPI, Response api = FastAPI() MAX_INFLIGHT = 3 state = {"in_flight": 0, "ready": False} @api.get("/readyz") def readyz(response: Response): if not state["ready"]: response.status_code = 503 return {"ready": state["ready"]} @api.post("/predict") async def predict(response: Response): if state["in_flight"] >= MAX_INFLIGHT: response.status_code = 429 response.headers["Retry-After"] = "1" return {"error": "busy"} state["in_flight"] += 1 try: await asyncio.sleep(0.05) return {"ok": True} finally: state["in_flight"] -= 1 async def main(): transport = httpx.ASGITransport(app=api) async with httpx.AsyncClient(transport=transport, base_url="http://t") as c: print("before load:", (await c.get("/readyz")).status_code) state["ready"] = True print("after load: ", (await c.get("/readyz")).status_code) replies = await asyncio.gather(*(c.post("/predict") for _ in range(6))) print([r.status_code for r in replies], replies[-1].headers.get("retry-after")) asyncio.run(main())
Three requests were admitted and three were refused with Retry-After: 1. The readiness route told the truth in both states. A liveness route would answer 200 the whole time, since the process is running, and that is why the two are separate.
Write admit(state, key, tokens): state has "in_flight" (a number), "max" and "budget" (a dict of tokens left per key). Return 429 if in_flight >= max. Otherwise, return 429 if the key has fewer than tokens left, and do not change anything. Otherwise subtract tokens from the key's budget, add 1 to in_flight, and return 200.
Uses the setup above, 16 lines.
def admit(state, key, tokens): return 200 state = {"in_flight": 0, "max": 2, "budget": {"ana": 1000}} print(admit(state, "ana", 600), state)
import math n, q = 10, 95 print(math.ceil(n * q / 100), 10 * 95 // 100)
10 9
Nearest rank rounds up: 9.5 becomes rank 10. Plain integer division rounds down to 9, which would give the 9th value and understate the tail.
Write a sentence or two, then compare it with a good answer in the lesson.
A text model is served differently from a classifier in three ways, and each shows up as a limit. A reply takes seconds, so the route takes a timeout and a max_tokens cap, since an unbounded reply holds the model and the bill. A key gets a daily token budget, charged before the work, so one user cannot spend everything. And work too slow to wait for, such as scoring a whole file, is accepted with 202 Accepted and a job id, and the client polls a status resource. The cell runs all three on the tiny chat model from the Phase 6 lessons.
A budget of tokens is charged before the work: the prompt's tokens plus the most the reply may use. A reply is capped by max_tokens. A big request is accepted as a job and read from a status route.
Uses the setup above, 16 lines.
import json import torch from devserver import serve from fastapi import FastAPI, Header, HTTPException, Response from pydantic import BaseModel, Field from tinygpt import load_pretrained model, tok = load_pretrained("tiny-gpt-chat") STOP = tok.token_id("<end>") BUDGET = {"key-ana": 20} JOBS = {} class GenerateIn(BaseModel): prompt: str = Field(min_length=1, max_length=200) max_tokens: int = Field(default=8, ge=1, le=16) def write(prompt, max_tokens): idx = torch.tensor([tok.encode(prompt)]) n_prompt = idx.shape[1] idx = model.generate(idx, max_tokens, temperature=0, stop_id=STOP) return n_prompt, tok.decode([int(i) for i in idx[0, n_prompt:] if int(i) != STOP]) api = FastAPI() @api.post("/generate") def generate(body: GenerateIn, response: Response, x_api_key: str = Header(default="")): left = BUDGET.get(x_api_key) if left is None: raise HTTPException(401, "unknown key") cost = len(tok.encode(body.prompt)) + body.max_tokens if cost > left: raise HTTPException( 429, "daily token budget used", headers={"Retry-After": "3600"} ) BUDGET[x_api_key] = left - cost _, text = write(body.prompt, body.max_tokens) return {"text": text, "charged": cost, "left": BUDGET[x_api_key]} @api.post("/jobs", status_code=202) def start(body: GenerateIn, response: Response): job_id = f"job_{len(JOBS) + 1}" JOBS[job_id] = {"status": "queued", "input": body.model_dump()} response.headers["Location"] = f"/jobs/{job_id}" return {"id": job_id, "status": "queued"} @api.get("/jobs/{job_id}") def job(job_id: str): j = JOBS.get(job_id) if j is None: raise HTTPException(404, "no such job") if j["status"] == "queued": j["status"] = "running" else: j["status"] = "done" return {"id": job_id, "status": j["status"]} base = serve(api) h = {"x-api-key": "key-ana"} p = "<user> Describe the River tent . <assistant>" r = requests.post(f"{base}/generate", json={"prompt": p, "max_tokens": 8}, headers=h) print(r.status_code, sorted(r.json())) r = requests.post(f"{base}/generate", json={"prompt": p, "max_tokens": 16}, headers=h) print(r.status_code, r.headers.get("retry-after")) print(requests.post(f"{base}/generate", json={"prompt": p}).status_code) r = requests.post(f"{base}/jobs", json={"prompt": p}) print(r.status_code, r.headers["location"]) print([requests.get(f"{base}/jobs/job_1").json()["status"] for _ in range(2)])
The first call was charged its prompt plus 8 and answered. The second asked for more than the key had left, so it got 429 with a Retry-After before the model ran. A call with no key got 401. The job got 202 and a Location, and the status route moved it from queued to running to done.
A model service gets paged for four things. Slow starts: the model loads on the first request, so a deploy answers slowly or times out; the fix is lifespan and a readiness route. A queue with no cap: a burst piles up work and every latency grows; the fix is a cap and a 429. A spend spike: one key runs a loop overnight; the fix is a token budget charged before the call. A blocked event loop: a sync model call inside async def, which APIs with FastAPI warned about; the fix is a plain def route or a thread. This lesson stays on the API side. One request through a serving stack: latency and cost explains why batches help on a GPU, and Batch and online inference covers choosing between batch and online serving and watching a model once it is live. Goes deeper in: Capstone: a streaming chat service puts these limits in front of a stream.
Write a sentence or two, then compare it with a good answer in the lesson.
Where should a service load its model?
575 lessons in thirteen phases. 176 are free without an account, and a free account opens the rest. See every lesson.
Go from your first line of code to working as a Python developer: projects, testing, data, databases, web apps and APIs, all in your browser.
Never trained an AI? Teach a small one to write, right in your browser. It takes about 20 seconds and needs no account. Teach it to write