Build a chat API with keys, rate limits and a token budget
The checks in front of a streaming chat service: an API key, a token bucket for each user, and a daily token budget, in plain Python you can test.
A FastAPI service with key auth, a per-user bucket, a token budget, request ids and an event-stream reply, tested on fresh state.
The phase began with bytes and ends with the backend an AI feature sits behind: a service that checks who is calling, limits how much they may spend, calls a model and streams the answer back, with a trail you can follow when something goes wrong. Every piece is one you have built: the parser of HTTP from the wire up, the checks of Project: a reviews API, the limits of Rate limits, retries and idempotency, the stream of Streaming from the server and the budget of Serving a model behind FastAPI. The work here is putting them in order, so that each protects the ones after it, and proving it with tests.
429 with a Retry-After that points at the reset
The key is fine, so it is not 401. The user has used what they may spend, which is a rate-limit style answer: 429, with a Retry-After that says when the budget resets.
This rests on Streaming from the server, which wrote a stream, and Serving a model behind FastAPI, which put limits in front of a model. The brief: "Our app needs a backend for a chat box. A signed-in user sends a prompt and sees the reply appear word by word. Each user may send a few messages a second and spend a fixed number of tokens a day, because every reply costs money. When something goes wrong, support must be able to find the request."
Turn it into a plan before any code, the way you would sketch a floor plan before building: a table of what each route needs and what it answers. One route does the work, POST /chat, with a JSON body {prompt, max_tokens} and the header X-API-Key. It answers with an event stream: data: {"delta": "..."} for each word and data: [DONE] at the end. Every reply carries X-Request-Id.
The cost is known up front, and a call already running cannot be un-spent
Charging first, with the most the reply may use, means a key can never overspend by a call already in flight.
The checks run in a fixed order, and each protects the ones after it: the key first, since an unknown caller must learn nothing; then the per-user bucket, which refuses a burst before any work is done; then the body; then the daily budget, charged before the model is called, since the cost is known as the prompt plus max_tokens; and only then the stream. Pick each case and watch where the request leaves. Two details are worth fixing in the plan now. The reply is capped by max_tokens, with a limit in the body schema, so no request can ask for an unbounded stream. And the per-user state, the buckets and the used tokens, is created lazily and held in one place, so a test can start with none of it and a deployment can move it elsewhere.
The earlier the check, the less a failing caller learns, and the less work is done for a request that will fail.
Write a sentence or two, then compare it with a good answer in the lesson.
The service is one function, make_app, that builds a fresh app and its state. A test calls it and gets an empty bucket table and an empty budget, so no test depends on another, and the clock is an argument, so a test can move time without waiting. The checks are FastAPI dependencies in the order of the plan, each raising an HTTPException, and a middleware adds the request id. The reply is the stream from Streaming from the server, over the tiny chat model loaded in this lesson's setup.
make_app builds the app and its state. The cell sends eight requests with a clock it controls, and prints the status of each, with the header that matters.
The lesson runs this once, before its first cell, and every cell, drill and example starts with what it defines. A cell that changes a setup object changes it for the cells after it, until the page reloads.
import asyncio import json import math import time import uuid import httpx import torch from fastapi import Depends, FastAPI, Header, HTTPException, Request from fastapi.responses import StreamingResponse from pydantic import BaseModel, Field from tinygpt import load_pretrained model, tok = load_pretrained("tiny-gpt-chat") STOP = tok.token_id("<end>") def token_stream(prompt, max_new): """One token at a time: the words a small chat model writes, greedily.""" idx = torch.tensor([tok.encode(prompt)]) for _ in range(max_new): idx = model.generate(idx, 1, temperature=0) nxt = int(idx[0, -1]) if nxt == STOP: return yield tok.vocab[nxt] class TokenBucket: def __init__(self, capacity, rate, now): self.capacity, self.rate = capacity, rate self.tokens, self.updated = capacity, now def allow(self, now): self.tokens = min(self.capacity, self.tokens + (now - self.updated) * self.rate) self.updated = now if self.tokens >= 1: self.tokens -= 1 return True return False def retry_after(self): return max(0.0, (1 - self.tokens) / self.rate)
KEYS = {"key-ana": "ana", "key-raj": "raj"}
class ChatIn(BaseModel):
prompt: str = Field(min_length=1, max_length=200)
max_tokens: int = Field(default=6, ge=1, le=12)
def make_app(clock, daily_budget=40, burst=2, rate=1.0):
app = FastAPI()
state = {"buckets": {}, "used": {}, "log": []}
@app.middleware("http")
async def request_id(request: Request, call_next):
rid = request.headers.get("x-request-id") or uuid.uuid4().hex[:8]
response = await call_next(request)
response.headers["X-Request-Id"] = rid
state["log"].append((rid, response.status_code))
return response
def known_user(x_api_key: str = Header(default="")):
user = KEYS.get(x_api_key)
if user is None:
raise HTTPException(401, "unknown key")
return user
def limited_user(user: str = Depends(known_user)):
fresh = TokenBucket(burst, rate, clock())
bucket = state["buckets"].setdefault(user, fresh)
if not bucket.allow(clock()):
wait = str(max(1, math.ceil(bucket.retry_after())))
raise HTTPException(429, "slow down", headers={"Retry-After": wait})
return user
@app.post("/chat")
async def chat(body: ChatIn, user: str = Depends(limited_user)):
cost = len(tok.encode(body.prompt)) + body.max_tokens
if state["used"].get(user, 0) + cost > daily_budget:
raise HTTPException(
429, "daily budget used", headers={"Retry-After": "3600"}
)
state["used"][user] = state["used"].get(user, 0) + cost
async def events():
for word in token_stream(body.prompt, body.max_tokens):
yield f"data: {json.dumps({'delta': word})}\n\n"
await asyncio.sleep(0)
yield "data: [DONE]\n\n"
return StreamingResponse(events(), media_type="text/event-stream")
return app, state
now = [0.0]
app, state = make_app(lambda: now[0])
PROMPT = "<user> Describe the River tent . <assistant>"
async def post(client, key, **extra):
headers = {"x-api-key": key} if key else {}
r = await client.post("/chat", json={"prompt": PROMPT, **extra}, headers=headers)
return r
async def main():
transport = httpx.ASGITransport(app=app)
async with httpx.AsyncClient(transport=transport, base_url="http://chat") as c:
r = await post(c, "key-evil")
print("unknown key:", r.status_code)
r = await post(c, "key-ana")
events = [line for line in r.text.split("\n\n") if line]
has_id = bool(r.headers["x-request-id"])
print("first:", r.status_code, len(events) > 1, events[-1], has_id)
print("second:", (await post(c, "key-ana")).status_code)
r = await post(c, "key-ana")
print("burst:", r.status_code, r.headers.get("retry-after"))
now[0] = 5.0
print("after 5 s:", (await post(c, "key-ana")).status_code)
r = await post(c, "key-ana")
print("over the day:", r.status_code, r.headers.get("retry-after"))
print("raj is separate:", (await post(c, "key-raj")).status_code)
print("too long:", (await post(c, "key-raj", max_tokens=500)).status_code)
asyncio.run(main())
ids = {rid for rid, _ in state["log"]}
print("log:", len(state["log"]), "requests, one id each:", len(ids))An unknown key got 401 and nothing more. The first reply streamed, and ended with the DONE event. The second used the burst, and the third was refused with Retry-After 1; moving the clock by 5 s refilled the bucket. By then Ana's daily tokens were spent, so the next call was 429 with Retry-After 3600, while Raj's budget was separate. max_tokens of 500 stopped at the body gate. Every request, refused or served, got an id and one log line.
make_app, why is the clock a function the caller passes in, rather than time.monotonic() inside the service?A test can move time with no waiting, so bucket tests are instant and repeatable
A service that reads the real clock makes tests slow and flaky. Passing the clock in lets a test set the time to 0, then to 5, and check exactly what the bucket does.
used, limit, cost = 26, 40, 13 print(used + cost > limit, used + cost, limit - used)
False 39 14
26 + 13 is 39, which is not over 40, so the request fits. The budget has 14 tokens left before it, and 1 after. The check is a plain comparison, written before the call.
def first_failure(request, checks): for check in checks: result = check(request) if result is not None: return result return 200 checks = [lambda r: None, lambda r: 429 if r > 5 else None, lambda r: 401] print(first_failure(3, checks), first_failure(9, checks))
401 429
For 3, the first two checks pass and the third returns 401. For 9, the second check fires first and returns 429, so the third never runs. Order decides which error a client sees.
A bucket tells you wait in seconds as a float. A Retry-After header holds whole seconds, at least 1, rounded up. Write retry_after(wait) returning that integer as a string.
Uses the setup above, 42 lines.
def retry_after(wait): return str(round(wait)) print(retry_after(0.2), retry_after(1.01), retry_after(3))
Write charge(used, user, cost, limit): used maps each user to tokens spent. If the user's spend plus cost is over limit, return False and change nothing. Otherwise add cost to their spend and return True.
Uses the setup above, 42 lines.
def charge(used, user, cost, limit): used[user] = used.get(user, 0) + cost return True used = {} print(charge(used, 'ana', 25, 40), charge(used, 'ana', 25, 40), used)
Write a sentence or two, then compare it with a good answer in the lesson.
A service that passes its tests is not safe until someone has read it as an attacker would. The diff below is a first draft of the chat route, written quickly. Mark the lines you would stop at before it merged.
The route also has two weaknesses a reviewer might add: it never checks whether the client has left, and a token that contains a blank line could forge an event in the stream. Both are worth a comment. The front end that reads this stream is the page from A chat front end in the browser: point its fetch at /chat, send the key in a header from your own server (a page must never hold the key), read the events with the parser you wrote, and show the text with textContent. The same checks guard it however it is called.
Deployment comes last, and is a short checklist, since the service is stateless apart from two dictionaries that a real deployment would move to Redis: read the keys and limits from the environment, never the code; run it under Uvicorn with a health route and a readiness route; put TLS and CORS at the edge, as in Running APIs in production; and send the request id into the logs and the traces.
import uuid def request_id(headers): return headers.get("x-request-id") or uuid.uuid4().hex[:8] print(request_id({"x-request-id": "edge-7f3a"}), len(request_id({})))
edge-7f3a 8
An id that arrives from upstream is kept, so the trail continues across services. With none, a new one is made, and hex[:8] is 8 characters.
Each copy has its own counts, so a user can get four times the limits
A dictionary lives in one process. Four copies each count separately, so a user sent to all four gets four times the allowance. Counts that must be shared live in a shared store, such as Redis.
The prototype works on your laptop, and the app team wants to try it on a staging server. You are asked to make it ready to run there, with its limits, its logs and a way to tell when it is healthy.
The hard part is the state. Two dictionaries held the buckets and the budgets, and a second copy of the service would count separately. Decide what moves out of memory, and what stays.
Uses the setup above, 42 lines.
import os KEYS = dict( pair.split("=") for pair in os.environ.get("API_KEYS", "").split(",") if pair ) DAILY_BUDGET = int(os.environ.get("DAILY_BUDGET", "40")) class MemoryStore: """Counts kept in this process. Replace with one two copies could share.""" def __init__(self): self.used = {} def charge(self, user, tokens, limit): if self.used.get(user, 0) + tokens > limit: return False self.used[user] = self.used.get(user, 0) + tokens return True # your code: the app, /healthz, /readyz, the request id and the tests print(len(KEYS), DAILY_BUDGET, MemoryStore().charge("ana", 13, DAILY_BUDGET))
A streaming chat backend is the shape of most AI products: a gate that decides who may spend, a limit on how much, a model, and a stream. Each piece is a lesson you can reread: Authentication and authorization for the key, Rate limits, retries and idempotency for the bucket, Validating requests with schemas for the body, Streaming from the server for the stream and Running APIs in production for the edge. Goes deeper in: Transports II: Streamable HTTP sends MCP over the same kind of stream, Streaming, interrupts and sessions keeps a conversation's state, and System design fundamentals draws the whole service with its load balancer, queue and cache.
Write a sentence or two, then compare it with a good answer in the lesson.
Why does the key check come before every other check in the chat service?
575 lessons in thirteen phases. 176 are free without an account, and a free account opens the rest. See every lesson.
The checks in front of a streaming chat service: an API key, a token bucket for each user, and a daily token budget, in plain Python you can test.
Go from your first line of code to working as a Python developer: projects, testing, data, databases, web apps and APIs, all in your browser.
Never trained an AI? Teach a small one to write, right in your browser. It takes about 20 seconds and needs no account. Teach it to write