Index Ai Academy
Phase 8 · FastAPI services

Serving a model behind FastAPI

Load once with lifespan, a typed predict route, a batch route, a cap on work in flight, 429 with Retry-After, a readiness route and a 202 job.

About 45 minutes · A free account opens the whole lesson.
Already have an account? Sign in to open this lesson

What it covers7

Why it matters

APIs with FastAPI named model serving as FastAPI's main use. This lesson does it. The shape is simple: load the model once, answer many requests, and protect it, because a model is the most expensive thing in the service and a route that lets any caller queue unlimited work is an outage waiting for a bad afternoon. The API side is what you build here: types, routes, limits and statuses. One request through a serving stack: latency and cost owns what happens on the GPU, and Batch and online inference owns running it in production.

What you'll be able to do

  • Load a model once at startup with lifespan, and answer a typed predict route with pydantic in and out
  • Work out latency and throughput by hand for one worker, a queue and a batch, and read p50 and p95
  • Cap the work in flight, answer 429 with Retry-After, and keep a readiness route honest
  • Put a token budget and a timeout in front of a generate route, and answer a long job with 202
Quick check
A route loads the model's weights from disk inside the function that answers each request. What is the likely result?
  1. Every request pays the load time, so replies are slow
  2. Requests run in parallel, because each has its own copy
  3. The model is more accurate, since it is fresh each time
  4. Nothing: loading is nearly free
Show the answer

Every request pays the load time, so replies are slow

Loading is the slow part. A route that loads for each request pays it on every call. Load once at startup, and let every request reuse the object.

The idea: load once, answer many, protect the model

This rests on APIs with FastAPI, where a lifespan function loaded a model once, and Rate limits, retries and idempotency, where a token bucket limited a client. A served model has a shape you can draw. At startup the service loads the model and, until it is done, says it is not ready. Then each request arrives, waits in line if the model is busy, runs, and returns. If the model can do several requests in one pass, a batch shares the fixed cost of a pass: a batch of 8 may cost 120 ms where eight single calls cost 400 ms, so waiting a moment to fill a batch trades a little latency for much more throughput.

Two numbers describe a service. Latency is how long one request takes, from arrival to reply, including its wait. Throughput is how many requests finish each second. They pull against each other, and a single average hides the problem: the slow ones are what users remember, so teams read p50 (half of requests are faster) and p95 (95 percent are faster). The picture queues six requests, each needing 50 ms, on one worker that handles one at a time.

Try it now

All six arrive together. The last one waits for five others before its own 50 ms starts.

Quick check
Recall: a token bucket holds 3 tokens and refills 1 a second. Three requests arrive at once. A fourth arrives 0.4 s later. What does this print?
tokens, capacity, rate = 3.0, 3.0, 1.0
for t in [0.0, 0.0, 0.0, 0.4]:
    tokens = min(capacity, tokens + rate * (0.4 if t == 0.4 else 0))
    if tokens >= 1:
        tokens -= 1
        print("allowed", end=" ")
    else:
        print("429", end=" ")
  1. 429 429 429 429
  2. allowed allowed 429 429
  3. allowed allowed allowed allowed
  4. allowed allowed allowed 429
Show the answer

allowed allowed allowed 429

The first three spend the three tokens. By the fourth, 0.4 s has dripped back, which is 0.4 of a token, and a request needs a whole one, so it is refused with 429.

In your own words
In your own words: why does a batch raise throughput, and why read p95 as well as the average?

Write a sentence or two, then compare it with a good answer in the lesson.

In Python: a model, a route, a cap and a readiness check

The practice customers table is the data: 600 customers, and a churned column to predict. The service fits a small logistic regression once, in lifespan, and every request reuses it. The route takes a pydantic model in and gives one out, so a bad request gets a 422 before the model runs. serve runs the app's startup once and then answers calls, the way a server would.

Worked example

A churn model behind POST /predict

The model is fit at startup from the practice customers. The route validates the body, and returns a probability.

Setup (runs before every cell) · 16 lines

The lesson runs this once, before its first cell, and every cell, drill and example starts with what it defines. A cell that changes a setup object changes it for the cells after it, until the page reloads.

import asyncio
import time
import numpy as np
import requests

BASE = "https://api.gradient.test"
FEATURES = [
    "tenure_months", "orders_last_year", "avg_order_value", "days_since_last_order",
]


def percentile(values, q):
    """Nearest-rank percentile: the smallest value with q percent at or below it."""
    ordered = sorted(values)
    rank = max(1, -(-len(ordered) * q // 100))
    return ordered[int(rank) - 1]
from contextlib import asynccontextmanager
from devserver import serve
from fastapi import FastAPI
from pydantic import BaseModel, Field
from sklearn.linear_model import LogisticRegression
from sklearn.preprocessing import StandardScaler
from sklearn.pipeline import make_pipeline

STATE = {"model": None, "loads": 0}

@asynccontextmanager
async def lifespan(app):
    rows = requests.get(f"{BASE}/v1/customers", params={"limit": 600}).json()["data"]
    X = np.array([[r[f] for f in FEATURES] for r in rows], dtype=float)
    y = np.array([r["churned"] for r in rows])
    STATE["model"] = make_pipeline(StandardScaler(), LogisticRegression()).fit(X, y)
    STATE["loads"] += 1
    yield
    STATE["model"] = None

class CustomerIn(BaseModel):
    tenure_months: float = Field(ge=0)
    orders_last_year: float = Field(ge=0)
    avg_order_value: float = Field(ge=0)
    days_since_last_order: float = Field(ge=0)

class Prediction(BaseModel):
    churn_probability: float

api = FastAPI(lifespan=lifespan)

@api.post("/predict")
def predict(customer: CustomerIn) -> Prediction:
    x = np.array([[getattr(customer, f) for f in FEATURES]])
    return Prediction(churn_probability=float(STATE["model"].predict_proba(x)[0, 1]))

base = serve(api)
good = {"tenure_months": 6, "orders_last_year": 2, "avg_order_value": 40,
        "days_since_last_order": 200}
r = requests.post(f"{base}/predict", json=good)
print(r.status_code, 0 <= r.json()["churn_probability"] <= 1)
r = requests.post(f"{base}/predict", json={**good, "tenure_months": -1})
print(r.status_code)
for _ in range(3):
    requests.post(f"{base}/predict", json=good)
print("model loads:", STATE["loads"])
In the lesson you can change this and run it.

A valid customer got a probability between 0 and 1. A negative tenure never reached the model: the body gate answered 422. After four predictions the model had been loaded once, because lifespan ran at startup, not per request.

Quick check
A predict route takes customer: CustomerIn where tenure_months: float = Field(ge=0). A request sends -1. What happens?
  1. 200 with the field set to 0
  2. 422 before the model runs
  3. The model runs and returns a probability
  4. 500, because the model fails
Show the answer

422 before the model runs

The body is checked against the model first. A negative tenure breaks ge=0, so the response is 422 with the field's location, and the function, and so the model, never runs.

Quick check
What does this print?
import numpy as np
X = np.arange(6).reshape(3, 2)
print(X.shape, X[0:1].shape)
  1. (3, 2) (1, 2)
  2. (3, 2) (3, 2)
  3. (3, 2) (2,)
  4. (2, 3) (1, 3)
Show the answer

(3, 2) (1, 2)

X[0:1] is a slice, so it keeps the row axis: shape (1, 2), a batch of one. That is why scoring one customer at a time still passes a 2-D array to predict_proba.

Batching, and the cost it saves

A batch route takes a list and scores it in one call to the model, which is where the fixed cost is shared. The cell scores 500 customers one at a time and then as one matrix, and compares the two. The model is fit in the setup of this cell's own run, since the cost to compare is the call, not the fit.

Uses the setup above, 16 lines.

from sklearn.linear_model import LogisticRegression

rng = np.random.default_rng(0)
X = rng.normal(size=(500, 4))
y = (X[:, 0] + rng.normal(size=500) > 0).astype(int)
model = LogisticRegression().fit(X, y)

start = time.perf_counter()
one_by_one = [model.predict_proba(X[i:i + 1])[0, 1] for i in range(len(X))]
t_single = time.perf_counter() - start

start = time.perf_counter()
batched = model.predict_proba(X)[:, 1]
t_batch = time.perf_counter() - start

print("same answers:", np.allclose(one_by_one, batched))
print("batch is faster:", t_batch < t_single)

The answers are the same. The batch is faster because the per-call overhead is paid once. Queued requests wait for a batch to fill, which is why real servers set a short batch window, such as 10 ms, and a maximum batch size.

Quick check
A service averages 100 ms but its p95 is 900 ms. What does that tell you?
  1. The service is faster than the average says, since p95 is high
  2. Every request takes about 100 ms
  3. The p95 is a typing mistake
  4. A few slow requests drag the tail: one in twenty takes 900 ms or more
Show the answer

A few slow requests drag the tail: one in twenty takes 900 ms or more

An average hides the slow ones. A p95 of 900 ms means 5 percent of requests take at least that long, and those are what users complain about.

Drill

p50 and p95 by nearest rank

Write pct(values, q): the nearest-rank percentile. Sort the values, find the rank ceil(len * q / 100) (at least 1), and return the value at that rank (counting from 1).

Uses the setup above, 16 lines.

def pct(values, q):
    return sum(values) / len(values)


print(pct([40, 42, 45, 50, 52, 55, 60, 75, 120, 400], 95))
In the lesson, your code is checked as you write it.

What the queue does to p95

A simulation shows why the average lies. One hundred requests arrive at random over ten seconds, each needing 80 ms from one worker. The plot compares the latency of each request under a single queue and under a worker that scores a batch every 100 ms. The percentile helper is in this lesson's setup.

Uses the setup above, 16 lines.

import random
import matplotlib.pyplot as plt

rng = random.Random(1)
arrivals = sorted(rng.uniform(0, 10) for _ in range(100))

free, queued = 0.0, []
for t in arrivals:
    start = max(t, free)
    free = start + 0.08
    queued.append(free - t)

batched, free = [], 0.0
for window in sorted({int(t * 10) for t in arrivals}):
    members = [t for t in arrivals if int(t * 10) == window]
    start = max((window + 1) / 10, free)
    free = start + 0.04 + 0.01 * len(members)
    batched += [free - t for t in members]

fig, ax = plt.subplots(figsize=(6, 3))
ax.hist(queued, bins=20, alpha=0.6, label="one at a time, 80 ms each")
ax.hist(batched, bins=20, alpha=0.6, label="batch every 100 ms")
ax.set_xlabel("latency (seconds)")
ax.set_ylabel("requests")
ax.legend()
for name, values in [("one at a time", queued), ("batched", batched)]:
    p50, p95 = percentile(values, 50), percentile(values, 95)
    print(name, "p50", round(p50, 2), "p95", round(p95, 2))

At this load the single worker is nearly saturated, so a queue builds, and p95 sits far above p50. The batched worker does the same work in less time and its p95 stays close to its p50. Change 0.08 to 0.1 and rerun: a worker that cannot keep up lets the queue, and every latency, grow without bound.

A cap protects the model. A request that arrives when the cap is full gets 429 with a Retry-After, so a polite client waits and tries again, and the service never queues unbounded work. The cell sends six requests together to a route that needs 50 ms each and allows three at once. It also adds a readiness route: until the model is loaded it answers 503, so a load balancer sends it nothing.

Uses the setup above, 16 lines.

import asyncio
import httpx
from fastapi import FastAPI, Response

api = FastAPI()
MAX_INFLIGHT = 3
state = {"in_flight": 0, "ready": False}

@api.get("/readyz")
def readyz(response: Response):
    if not state["ready"]:
        response.status_code = 503
    return {"ready": state["ready"]}

@api.post("/predict")
async def predict(response: Response):
    if state["in_flight"] >= MAX_INFLIGHT:
        response.status_code = 429
        response.headers["Retry-After"] = "1"
        return {"error": "busy"}
    state["in_flight"] += 1
    try:
        await asyncio.sleep(0.05)
        return {"ok": True}
    finally:
        state["in_flight"] -= 1

async def main():
    transport = httpx.ASGITransport(app=api)
    async with httpx.AsyncClient(transport=transport, base_url="http://t") as c:
        print("before load:", (await c.get("/readyz")).status_code)
        state["ready"] = True
        print("after load: ", (await c.get("/readyz")).status_code)
        replies = await asyncio.gather(*(c.post("/predict") for _ in range(6)))
        print([r.status_code for r in replies], replies[-1].headers.get("retry-after"))

asyncio.run(main())

Three requests were admitted and three were refused with Retry-After: 1. The readiness route told the truth in both states. A liveness route would answer 200 the whole time, since the process is running, and that is why the two are separate.

Drill

A cap and a budget for one key

Write admit(state, key, tokens): state has "in_flight" (a number), "max" and "budget" (a dict of tokens left per key). Return 429 if in_flight >= max. Otherwise, return 429 if the key has fewer than tokens left, and do not change anything. Otherwise subtract tokens from the key's budget, add 1 to in_flight, and return 200.

Uses the setup above, 16 lines.

def admit(state, key, tokens):
    return 200


state = {"in_flight": 0, "max": 2, "budget": {"ana": 1000}}
print(admit(state, "ana", 600), state)
In the lesson, your code is checked as you write it.
Quick check
What does this print?
import math
n, q = 10, 95
print(math.ceil(n * q / 100), 10 * 95 // 100)
  1. 10 10
  2. 9 9
  3. 9 10
  4. 10 9
Show the answer

10 9

Nearest rank rounds up: 9.5 becomes rank 10. Plain integer division rounds down to 9, which would give the 9th value and understate the tail.

In your own words
In your own words: what do a cap, a 429 with Retry-After and a readiness route each protect?

Write a sentence or two, then compare it with a good answer in the lesson.

In AI and ML systems: a generate route, a budget and a job

A text model is served differently from a classifier in three ways, and each shows up as a limit. A reply takes seconds, so the route takes a timeout and a max_tokens cap, since an unbounded reply holds the model and the bill. A key gets a daily token budget, charged before the work, so one user cannot spend everything. And work too slow to wait for, such as scoring a whole file, is accepted with 202 Accepted and a job id, and the client polls a status resource. The cell runs all three on the tiny chat model from the Phase 6 lessons.

Worked example

POST /generate with a cap, a budget and a job

A budget of tokens is charged before the work: the prompt's tokens plus the most the reply may use. A reply is capped by max_tokens. A big request is accepted as a job and read from a status route.

Uses the setup above, 16 lines.

import json
import torch
from devserver import serve
from fastapi import FastAPI, Header, HTTPException, Response
from pydantic import BaseModel, Field
from tinygpt import load_pretrained

model, tok = load_pretrained("tiny-gpt-chat")
STOP = tok.token_id("<end>")
BUDGET = {"key-ana": 20}
JOBS = {}

class GenerateIn(BaseModel):
    prompt: str = Field(min_length=1, max_length=200)
    max_tokens: int = Field(default=8, ge=1, le=16)

def write(prompt, max_tokens):
    idx = torch.tensor([tok.encode(prompt)])
    n_prompt = idx.shape[1]
    idx = model.generate(idx, max_tokens, temperature=0, stop_id=STOP)
    return n_prompt, tok.decode([int(i) for i in idx[0, n_prompt:] if int(i) != STOP])

api = FastAPI()

@api.post("/generate")
def generate(body: GenerateIn, response: Response, x_api_key: str = Header(default="")):
    left = BUDGET.get(x_api_key)
    if left is None:
        raise HTTPException(401, "unknown key")
    cost = len(tok.encode(body.prompt)) + body.max_tokens
    if cost > left:
        raise HTTPException(
            429, "daily token budget used", headers={"Retry-After": "3600"}
        )
    BUDGET[x_api_key] = left - cost
    _, text = write(body.prompt, body.max_tokens)
    return {"text": text, "charged": cost, "left": BUDGET[x_api_key]}

@api.post("/jobs", status_code=202)
def start(body: GenerateIn, response: Response):
    job_id = f"job_{len(JOBS) + 1}"
    JOBS[job_id] = {"status": "queued", "input": body.model_dump()}
    response.headers["Location"] = f"/jobs/{job_id}"
    return {"id": job_id, "status": "queued"}

@api.get("/jobs/{job_id}")
def job(job_id: str):
    j = JOBS.get(job_id)
    if j is None:
        raise HTTPException(404, "no such job")
    if j["status"] == "queued":
        j["status"] = "running"
    else:
        j["status"] = "done"
    return {"id": job_id, "status": j["status"]}

base = serve(api)
h = {"x-api-key": "key-ana"}
p = "<user> Describe the River tent . <assistant>"
r = requests.post(f"{base}/generate", json={"prompt": p, "max_tokens": 8}, headers=h)
print(r.status_code, sorted(r.json()))
r = requests.post(f"{base}/generate", json={"prompt": p, "max_tokens": 16}, headers=h)
print(r.status_code, r.headers.get("retry-after"))
print(requests.post(f"{base}/generate", json={"prompt": p}).status_code)
r = requests.post(f"{base}/jobs", json={"prompt": p})
print(r.status_code, r.headers["location"])
print([requests.get(f"{base}/jobs/job_1").json()["status"] for _ in range(2)])
In the lesson you can change this and run it.

The first call was charged its prompt plus 8 and answered. The second asked for more than the key had left, so it got 429 with a Retry-After before the model ran. A call with no key got 401. The job got 202 and a Location, and the status route moved it from queued to running to done.

Where this shows up in an engineer's week

A model service gets paged for four things. Slow starts: the model loads on the first request, so a deploy answers slowly or times out; the fix is lifespan and a readiness route. A queue with no cap: a burst piles up work and every latency grows; the fix is a cap and a 429. A spend spike: one key runs a loop overnight; the fix is a token budget charged before the call. A blocked event loop: a sync model call inside async def, which APIs with FastAPI warned about; the fix is a plain def route or a thread. This lesson stays on the API side. One request through a serving stack: latency and cost explains why batches help on a GPU, and Batch and online inference covers choosing between batch and online serving and watching a model once it is live. Goes deeper in: Capstone: a streaming chat service puts these limits in front of a stream.

In your own words
In your own words: why charge a token budget before the model call, and what does a 202 give the client?

Write a sentence or two, then compare it with a good answer in the lesson.

Practice what you just read

  • 5 code examples you can edit and run
  • a practice exercise that checks your code
  • a 5-question quiz
  • 9 review cards for spaced repetition

Try a quiz question

Where should a service load its model?

  1. Once at startup, in lifespan
  2. Inside each route, on every request
  3. On the first request only, with no check
  4. In the client

Answer it at the end of the lesson to see if you're right, and why.

Before this lesson

Keep learning

Where this lesson leads.

575 lessons in thirteen phases. 176 are free without an account, and a free account opens the rest. See every lesson.

Phase 8 · about 23 hours of study

APIs

Design, build, secure and scale web APIs from first principles 27 lessons 1 free
  1. HTTP from the wire up FreeWhat really travels between client and server: bytes, headers and bodies.40 min
  2. API stylesREST, RPC, GraphQL, gRPC and streaming: what each is for.35 min
  3. Build an API server from scratchRouting, handlers and responses, with no framework.45 min
  4. A web framework, from the insideDecorators, middleware and error handling: how Flask and FastAPI work.40 min
  5. Designing a REST APIResources, methods, status codes, errors and pagination done well.45 min
  6. Validating requests with schemasCheck every request against a JSON Schema and say exactly what's wrong.45 min
  7. A database-backed APICRUD endpoints on SQLite, with safe concurrent updates.45 min
  8. Authentication and authorizationAPI keys, sessions, tokens and OAuth: who's calling, and what may they do?45 min
  9. Rate limits, retries and idempotencyKeep an API up under load and make retries safe.45 min
  10. Async HTTP clientsMany requests at once with asyncio and httpx: limits, timeouts and retries.45 min
  11. Webhooks and eventsPush events to other systems: signed, retried and deduplicated.45 min
  12. Documenting APIs with OpenAPIA machine-readable contract for docs, SDKs and AI tools.40 min
  13. Running APIs in productionVersioning, CORS, caching, request ids and health checks.45 min
  14. Project: a reviews APIAn authenticated, validated, paginated API with tests.45 min
  15. Web pages with FlaskRoutes, Jinja templates, a shared layout, links and error pages, tested with the test client.45 min
  16. Forms and redirectsGET and POST, validation messages, post/redirect/get, flash messages and CSRF tokens.45 min
  17. Accounts and loginsPassword hashing with bcrypt, sessions, pages that need a login, and safe logout.45 min
  18. Project: a trail log web appA multi-page app with a database, accounts, forms and tests.45 min
  19. JavaScript for Python programmersVariables, functions, arrays and objects mapped from Python, and the traps: truthy empty arrays, for...in and string sort.35 min
  20. Async JavaScript: promises, await and fetchThe event loop, async and await, fetch with res.ok checks, Promise.all, and timeouts with AbortController.35 min
  21. Modules, Node and reading TypeScriptimport and export, package.json, classes, and types as notes the compiler checks, read through a small tool server.30 min
  22. A chat front end in the browserThe DOM and events, model text kept out of innerHTML, a streamed reply parsed from server-sent events, and a Stop button.40 min
  23. APIs with FastAPITyped routes, pydantic models, automatic validation and docs, tested with httpx.45 min
  24. Streaming from the serverServer-sent events written by a server: an ASGI app that sends a body in pieces, FastAPI's StreamingResponse, heartbeats, and what a closed connection looks like.40 min
  25. Project: a FastAPI service with accountsA session per request, bcrypt sign-ups, signed tokens, ownership checks and tests on a fresh database.45 min
  26. Serving a model behind FastAPILoad once with lifespan, a typed predict route, a batch route, a cap on work in flight, 429 with Retry-After, a readiness route and a 202 job.45 min
  27. Capstone: a streaming chat serviceA FastAPI service with key auth, a per-user bucket, a token budget, request ids and an event-stream reply, tested on fresh state.45 min
Part of a free course 127 lessons

Free Python developer course

Go from your first line of code to working as a Python developer: projects, testing, data, databases, web apps and APIs, all in your browser.

Never trained an AI? Teach a small one to write, right in your browser. It takes about 20 seconds and needs no account. Teach it to write