Index Ai Academy
Phase 8 · FastAPI services

Capstone: a streaming chat service

A FastAPI service with key auth, a per-user bucket, a token budget, request ids and an event-stream reply, tested on fresh state.

About 45 minutes · A free account opens the whole lesson.
Already have an account? Sign in to open this lesson

What it covers7

Why it matters

The phase began with bytes and ends with the backend an AI feature sits behind: a service that checks who is calling, limits how much they may spend, calls a model and streams the answer back, with a trail you can follow when something goes wrong. Every piece is one you have built: the parser of HTTP from the wire up, the checks of Project: a reviews API, the limits of Rate limits, retries and idempotency, the stream of Streaming from the server and the budget of Serving a model behind FastAPI. The work here is putting them in order, so that each protects the ones after it, and proving it with tests.

What you'll be able to do

  • Design a streaming chat service: routes, the order of checks, limits and the stream format
  • Build it in FastAPI with key auth, a per-user token bucket, a daily token budget, request ids and an event-stream reply
  • Test it on fresh state, with a clock you control, through one entry point
  • Review a chat route for security faults and plan its deployment
Quick check
A chat service has a per-user token bucket and a daily token budget. A user over the daily budget sends a request with a valid key. Which status should it get?
  1. 200 with an empty reply
  2. 500, since the budget is a server problem
  3. 429 with a Retry-After that points at the reset
  4. 401, since the key no longer works
Show the answer

429 with a Retry-After that points at the reset

The key is fine, so it is not 401. The user has used what they may spend, which is a rate-limit style answer: 429, with a Retry-After that says when the budget resets.

The brief: a streaming chat service

This rests on Streaming from the server, which wrote a stream, and Serving a model behind FastAPI, which put limits in front of a model. The brief: "Our app needs a backend for a chat box. A signed-in user sends a prompt and sees the reply appear word by word. Each user may send a few messages a second and spend a fixed number of tokens a day, because every reply costs money. When something goes wrong, support must be able to find the request."

Turn it into a plan before any code, the way you would sketch a floor plan before building: a table of what each route needs and what it answers. One route does the work, POST /chat, with a JSON body {prompt, max_tokens} and the header X-API-Key. It answers with an event stream: data: {"delta": "..."} for each word and data: [DONE] at the end. Every reply carries X-Request-Id.

Quick check
Recall: why is the daily budget charged before the model is called, not after the reply?
  1. The cost is known up front, and a call already running cannot be un-spent
  2. It makes the reply faster
  3. A budget can only be charged once a day
  4. The model cannot count the tokens it used after it replies
Show the answer

The cost is known up front, and a call already running cannot be un-spent

Charging first, with the most the reply may use, means a key can never overspend by a call already in flight.

The checks run in a fixed order, and each protects the ones after it: the key first, since an unknown caller must learn nothing; then the per-user bucket, which refuses a burst before any work is done; then the body; then the daily budget, charged before the model is called, since the cost is known as the prompt plus max_tokens; and only then the stream. Pick each case and watch where the request leaves. Two details are worth fixing in the plan now. The reply is capped by max_tokens, with a limit in the body schema, so no request can ask for an unbounded stream. And the per-user state, the buckets and the used tokens, is created lazily and held in one place, so a test can start with none of it and a deployment can move it elsewhere.

Try it now

The earlier the check, the less a failing caller learns, and the less work is done for a request that will fail.

In your own words
In your own words: why do the checks run in a fixed order, and what does each earlier check protect?

Write a sentence or two, then compare it with a good answer in the lesson.

Build it: the service, and tests that start clean

The service is one function, make_app, that builds a fresh app and its state. A test calls it and gets an empty bucket table and an empty budget, so no test depends on another, and the clock is an argument, so a test can move time without waiting. The checks are FastAPI dependencies in the order of the plan, each raising an HTTPException, and a middleware adds the request id. The reply is the stream from Streaming from the server, over the tiny chat model loaded in this lesson's setup.

Worked example

The service, called through its one entry point

make_app builds the app and its state. The cell sends eight requests with a clock it controls, and prints the status of each, with the header that matters.

Setup (runs before every cell) · 42 lines

The lesson runs this once, before its first cell, and every cell, drill and example starts with what it defines. A cell that changes a setup object changes it for the cells after it, until the page reloads.

import asyncio
import json
import math
import time
import uuid
import httpx
import torch
from fastapi import Depends, FastAPI, Header, HTTPException, Request
from fastapi.responses import StreamingResponse
from pydantic import BaseModel, Field
from tinygpt import load_pretrained

model, tok = load_pretrained("tiny-gpt-chat")
STOP = tok.token_id("<end>")


def token_stream(prompt, max_new):
    """One token at a time: the words a small chat model writes, greedily."""
    idx = torch.tensor([tok.encode(prompt)])
    for _ in range(max_new):
        idx = model.generate(idx, 1, temperature=0)
        nxt = int(idx[0, -1])
        if nxt == STOP:
            return
        yield tok.vocab[nxt]


class TokenBucket:
    def __init__(self, capacity, rate, now):
        self.capacity, self.rate = capacity, rate
        self.tokens, self.updated = capacity, now

    def allow(self, now):
        self.tokens = min(self.capacity, self.tokens + (now - self.updated) * self.rate)
        self.updated = now
        if self.tokens >= 1:
            self.tokens -= 1
            return True
        return False

    def retry_after(self):
        return max(0.0, (1 - self.tokens) / self.rate)
KEYS = {"key-ana": "ana", "key-raj": "raj"}

class ChatIn(BaseModel):
    prompt: str = Field(min_length=1, max_length=200)
    max_tokens: int = Field(default=6, ge=1, le=12)

def make_app(clock, daily_budget=40, burst=2, rate=1.0):
    app = FastAPI()
    state = {"buckets": {}, "used": {}, "log": []}

    @app.middleware("http")
    async def request_id(request: Request, call_next):
        rid = request.headers.get("x-request-id") or uuid.uuid4().hex[:8]
        response = await call_next(request)
        response.headers["X-Request-Id"] = rid
        state["log"].append((rid, response.status_code))
        return response

    def known_user(x_api_key: str = Header(default="")):
        user = KEYS.get(x_api_key)
        if user is None:
            raise HTTPException(401, "unknown key")
        return user

    def limited_user(user: str = Depends(known_user)):
        fresh = TokenBucket(burst, rate, clock())
        bucket = state["buckets"].setdefault(user, fresh)
        if not bucket.allow(clock()):
            wait = str(max(1, math.ceil(bucket.retry_after())))
            raise HTTPException(429, "slow down", headers={"Retry-After": wait})
        return user

    @app.post("/chat")
    async def chat(body: ChatIn, user: str = Depends(limited_user)):
        cost = len(tok.encode(body.prompt)) + body.max_tokens
        if state["used"].get(user, 0) + cost > daily_budget:
            raise HTTPException(
                429, "daily budget used", headers={"Retry-After": "3600"}
            )
        state["used"][user] = state["used"].get(user, 0) + cost

        async def events():
            for word in token_stream(body.prompt, body.max_tokens):
                yield f"data: {json.dumps({'delta': word})}\n\n"
                await asyncio.sleep(0)
            yield "data: [DONE]\n\n"

        return StreamingResponse(events(), media_type="text/event-stream")

    return app, state

now = [0.0]
app, state = make_app(lambda: now[0])
PROMPT = "<user> Describe the River tent . <assistant>"

async def post(client, key, **extra):
    headers = {"x-api-key": key} if key else {}
    r = await client.post("/chat", json={"prompt": PROMPT, **extra}, headers=headers)
    return r

async def main():
    transport = httpx.ASGITransport(app=app)
    async with httpx.AsyncClient(transport=transport, base_url="http://chat") as c:
        r = await post(c, "key-evil")
        print("unknown key:", r.status_code)
        r = await post(c, "key-ana")
        events = [line for line in r.text.split("\n\n") if line]
        has_id = bool(r.headers["x-request-id"])
        print("first:", r.status_code, len(events) > 1, events[-1], has_id)
        print("second:", (await post(c, "key-ana")).status_code)
        r = await post(c, "key-ana")
        print("burst:", r.status_code, r.headers.get("retry-after"))
        now[0] = 5.0
        print("after 5 s:", (await post(c, "key-ana")).status_code)
        r = await post(c, "key-ana")
        print("over the day:", r.status_code, r.headers.get("retry-after"))
        print("raj is separate:", (await post(c, "key-raj")).status_code)
        print("too long:", (await post(c, "key-raj", max_tokens=500)).status_code)

asyncio.run(main())
ids = {rid for rid, _ in state["log"]}
print("log:", len(state["log"]), "requests, one id each:", len(ids))
In the lesson you can change this and run it.

An unknown key got 401 and nothing more. The first reply streamed, and ended with the DONE event. The second used the burst, and the third was refused with Retry-After 1; moving the clock by 5 s refilled the bucket. By then Ana's daily tokens were spent, so the next call was 429 with Retry-After 3600, while Raj's budget was separate. max_tokens of 500 stopped at the body gate. Every request, refused or served, got an id and one log line.

Quick check
In make_app, why is the clock a function the caller passes in, rather than time.monotonic() inside the service?
  1. So the clock can be shared by all users
  2. time.monotonic is not allowed in FastAPI routes, middleware or tests
  3. A test can move time with no waiting, so bucket tests are instant and repeatable
  4. It makes the service faster
Show the answer

A test can move time with no waiting, so bucket tests are instant and repeatable

A service that reads the real clock makes tests slow and flaky. Passing the clock in lets a test set the time to 0, then to 5, and check exactly what the bucket does.

Quick check
What does this print?
used, limit, cost = 26, 40, 13
print(used + cost > limit, used + cost, limit - used)
  1. True 39 14
  2. True 26 14
  3. False 39 14
  4. False 39 13
Show the answer

False 39 14

26 + 13 is 39, which is not over 40, so the request fits. The budget has 14 tokens left before it, and 1 after. The check is a plain comparison, written before the call.

Quick check
What does this print?
def first_failure(request, checks):
    for check in checks:
        result = check(request)
        if result is not None:
            return result
    return 200

checks = [lambda r: None, lambda r: 429 if r > 5 else None, lambda r: 401]
print(first_failure(3, checks), first_failure(9, checks))
  1. 429 401
  2. 200 429
  3. 401 401
  4. 401 429
Show the answer

401 429

For 3, the first two checks pass and the third returns 401. For 9, the second check fires first and returns 429, so the third never runs. Order decides which error a client sees.

Drill

A Retry-After in whole seconds

A bucket tells you wait in seconds as a float. A Retry-After header holds whole seconds, at least 1, rounded up. Write retry_after(wait) returning that integer as a string.

Uses the setup above, 42 lines.

def retry_after(wait):
    return str(round(wait))


print(retry_after(0.2), retry_after(1.01), retry_after(3))
In the lesson, your code is checked as you write it.
Drill

Charge a budget only when the call fits

Write charge(used, user, cost, limit): used maps each user to tokens spent. If the user's spend plus cost is over limit, return False and change nothing. Otherwise add cost to their spend and return True.

Uses the setup above, 42 lines.

def charge(used, user, cost, limit):
    used[user] = used.get(user, 0) + cost
    return True


used = {}
print(charge(used, 'ana', 25, 40), charge(used, 'ana', 25, 40), used)
In the lesson, your code is checked as you write it.
In your own words
In your own words: how do fresh state and an injected clock make this service testable?

Write a sentence or two, then compare it with a good answer in the lesson.

Ship it: review it, break it, deploy it

A service that passes its tests is not safe until someone has read it as an attacker would. The diff below is a first draft of the chat route, written quickly. Mark the lines you would stop at before it merged.

The route also has two weaknesses a reviewer might add: it never checks whether the client has left, and a token that contains a blank line could forge an event in the stream. Both are worth a comment. The front end that reads this stream is the page from A chat front end in the browser: point its fetch at /chat, send the key in a header from your own server (a page must never hold the key), read the events with the parser you wrote, and show the text with textContent. The same checks guard it however it is called.

Deployment comes last, and is a short checklist, since the service is stateless apart from two dictionaries that a real deployment would move to Redis: read the keys and limits from the environment, never the code; run it under Uvicorn with a health route and a readiness route; put TLS and CORS at the edge, as in Running APIs in production; and send the request id into the logs and the traces.

Quick check
What does this print?
import uuid
def request_id(headers):
    return headers.get("x-request-id") or uuid.uuid4().hex[:8]
print(request_id({"x-request-id": "edge-7f3a"}), len(request_id({})))
  1. edge-7f3a 0
  2. edge-7f3a 8
  3. edge-7f3a 32
  4. None 8
Show the answer

edge-7f3a 8

An id that arrives from upstream is kept, so the trail continues across services. With none, a new one is made, and hex[:8] is 8 characters.

Quick check
The service keeps each user's bucket and used tokens in dictionaries in memory. You deploy four copies behind a load balancer. What goes wrong?
  1. The model loads only once for all copies
  2. The load balancer rejects the requests
  3. Nothing: the dictionaries are shared between the copies automatically
  4. Each copy has its own counts, so a user can get four times the limits
Show the answer

Each copy has its own counts, so a user can get four times the limits

A dictionary lives in one process. Four copies each count separately, so a user sent to all four gets four times the allowance. Counts that must be shared live in a shared store, such as Redis.

Portfolio brief

Deploy the chat service

The prototype works on your laptop, and the app team wants to try it on a staging server. You are asked to make it ready to run there, with its limits, its logs and a way to tell when it is healthy.

The hard part is the state. Two dictionaries held the buckets and the budgets, and a second copy of the service would count separately. Decide what moves out of memory, and what stays.

It must

  • The API keys, the daily budget and the bucket settings come from environment variables, with no default key in the code
  • A GET /healthz route answers 200 whenever the process is running, and a GET /readyz route answers 503 until the model is loaded
  • Every response, refused or served, carries X-Request-Id, and one log line holds the id, the user name, the path and the status, but never the key or the prompt
  • The per-user bucket and the used-token counts are kept in a store that two copies of the service could share, behind a small interface you can replace
  • A test builds the app with a fresh store and a fake clock, and checks 401, 429 with Retry-After, 422 and a served stream

Go further

  • Stop generating when the client disconnects
  • A Dockerfile that runs it with Uvicorn
  • A 202 job route for a long prompt, with a status resource

Uses the setup above, 42 lines.

import os

KEYS = dict(
    pair.split("=") for pair in os.environ.get("API_KEYS", "").split(",") if pair
)
DAILY_BUDGET = int(os.environ.get("DAILY_BUDGET", "40"))

class MemoryStore:
    """Counts kept in this process. Replace with one two copies could share."""

    def __init__(self):
        self.used = {}

    def charge(self, user, tokens, limit):
        if self.used.get(user, 0) + tokens > limit:
            return False
        self.used[user] = self.used.get(user, 0) + tokens
        return True

# your code: the app, /healthz, /readyz, the request id and the tests
print(len(KEYS), DAILY_BUDGET, MemoryStore().charge("ana", 13, DAILY_BUDGET))
In the lesson, you build it here and the tutor can review it.

Where this fits, and what comes next

A streaming chat backend is the shape of most AI products: a gate that decides who may spend, a limit on how much, a model, and a stream. Each piece is a lesson you can reread: Authentication and authorization for the key, Rate limits, retries and idempotency for the bucket, Validating requests with schemas for the body, Streaming from the server for the stream and Running APIs in production for the edge. Goes deeper in: Transports II: Streamable HTTP sends MCP over the same kind of stream, Streaming, interrupts and sessions keeps a conversation's state, and System design fundamentals draws the whole service with its load balancer, queue and cache.

In your own words
In your own words: why must the counts leave process memory before you run two copies, and what stays the same?

Write a sentence or two, then compare it with a good answer in the lesson.

Practice what you just read

  • 1 code example you can edit and run
  • a practice exercise that checks your code
  • a 5-question quiz
  • 9 review cards for spaced repetition

Try a quiz question

Why does the key check come before every other check in the chat service?

  1. An unknown caller must learn nothing else about the service
  2. It is the fastest check
  3. Keys decide the stream format
  4. The bucket needs the key's hash to look up the user

Answer it at the end of the lesson to see if you're right, and why.

Before this lesson

Keep learning

Where this lesson leads.

575 lessons in thirteen phases. 176 are free without an account, and a free account opens the rest. See every lesson.

Phase 8 · about 23 hours of study

APIs

Design, build, secure and scale web APIs from first principles 27 lessons 1 free
  1. HTTP from the wire up FreeWhat really travels between client and server: bytes, headers and bodies.40 min
  2. API stylesREST, RPC, GraphQL, gRPC and streaming: what each is for.35 min
  3. Build an API server from scratchRouting, handlers and responses, with no framework.45 min
  4. A web framework, from the insideDecorators, middleware and error handling: how Flask and FastAPI work.40 min
  5. Designing a REST APIResources, methods, status codes, errors and pagination done well.45 min
  6. Validating requests with schemasCheck every request against a JSON Schema and say exactly what's wrong.45 min
  7. A database-backed APICRUD endpoints on SQLite, with safe concurrent updates.45 min
  8. Authentication and authorizationAPI keys, sessions, tokens and OAuth: who's calling, and what may they do?45 min
  9. Rate limits, retries and idempotencyKeep an API up under load and make retries safe.45 min
  10. Async HTTP clientsMany requests at once with asyncio and httpx: limits, timeouts and retries.45 min
  11. Webhooks and eventsPush events to other systems: signed, retried and deduplicated.45 min
  12. Documenting APIs with OpenAPIA machine-readable contract for docs, SDKs and AI tools.40 min
  13. Running APIs in productionVersioning, CORS, caching, request ids and health checks.45 min
  14. Project: a reviews APIAn authenticated, validated, paginated API with tests.45 min
  15. Web pages with FlaskRoutes, Jinja templates, a shared layout, links and error pages, tested with the test client.45 min
  16. Forms and redirectsGET and POST, validation messages, post/redirect/get, flash messages and CSRF tokens.45 min
  17. Accounts and loginsPassword hashing with bcrypt, sessions, pages that need a login, and safe logout.45 min
  18. Project: a trail log web appA multi-page app with a database, accounts, forms and tests.45 min
  19. JavaScript for Python programmersVariables, functions, arrays and objects mapped from Python, and the traps: truthy empty arrays, for...in and string sort.35 min
  20. Async JavaScript: promises, await and fetchThe event loop, async and await, fetch with res.ok checks, Promise.all, and timeouts with AbortController.35 min
  21. Modules, Node and reading TypeScriptimport and export, package.json, classes, and types as notes the compiler checks, read through a small tool server.30 min
  22. A chat front end in the browserThe DOM and events, model text kept out of innerHTML, a streamed reply parsed from server-sent events, and a Stop button.40 min
  23. APIs with FastAPITyped routes, pydantic models, automatic validation and docs, tested with httpx.45 min
  24. Streaming from the serverServer-sent events written by a server: an ASGI app that sends a body in pieces, FastAPI's StreamingResponse, heartbeats, and what a closed connection looks like.40 min
  25. Project: a FastAPI service with accountsA session per request, bcrypt sign-ups, signed tokens, ownership checks and tests on a fresh database.45 min
  26. Serving a model behind FastAPILoad once with lifespan, a typed predict route, a batch route, a cap on work in flight, 429 with Retry-After, a readiness route and a 202 job.45 min
  27. Capstone: a streaming chat serviceA FastAPI service with key auth, a per-user bucket, a token budget, request ids and an event-stream reply, tested on fresh state.45 min
Part of a free course 127 lessons

Free Python developer course

Go from your first line of code to working as a Python developer: projects, testing, data, databases, web apps and APIs, all in your browser.

Never trained an AI? Teach a small one to write, right in your browser. It takes about 20 seconds and needs no account. Teach it to write