Index Ai Academy
Project tutorial · 3 steps

Build a chat API with keys, rate limits and a token budget

The checks in front of a streaming chat service: an API key, a token bucket for each user, and a daily token budget, in plain Python you can test.

Free to read, with no account. The code comes from the lesson Capstone: a streaming chat service, and has been run.

The steps5

What you'll build

A chat service that calls a language model needs three gates before it answers: who is asking, how fast, and how much they may spend. This tutorial writes those gates as ordinary Python, so each one can be tested without a web server. The lesson wires them into a FastAPI service that streams its reply.

What you'll learn

  • Design a streaming chat service: routes, the order of checks, limits and the stream format
  • Build it in FastAPI with key auth, a per-user token bucket, a daily token budget, request ids and an event-stream reply
  • Test it on fresh state, with a clock you control, through one entry point
  • Review a chat route for security faults and plan its deployment

What you need

Python classes and dictionaries, and what an HTTP status code is. It uses only the standard library, so it runs anywhere with Python 3.

Step 1: A token bucket for each user

The bucket holds up to capacity tokens and gains rate tokens a second. Each request takes one. allow works out how much was refilled since the last request, so nothing runs in the background. retry_after says how long until one whole token is back, which becomes the Retry-After header.

import math


class TokenBucket:
    def __init__(self, capacity, rate, now):
        self.capacity, self.rate = capacity, rate
        self.tokens, self.updated = capacity, now

    def allow(self, now):
        refill = (now - self.updated) * self.rate
        self.tokens = min(self.capacity, self.tokens + refill)
        self.updated = now
        if self.tokens >= 1:
            self.tokens -= 1
            return True
        return False

    def retry_after(self):
        return max(0.0, (1 - self.tokens) / self.rate)

Step 2: Authorize a request

Three checks, in order. An unknown key is 401. A user who is out of tokens is 429 with the wait in seconds. A user whose day's budget would be passed by this request's prompt and reply tokens is 429 with an hour's wait. Only a request that passes all three is counted against the budget.

def authorize(state, key, now, prompt_tokens, max_tokens):
    user = state["keys"].get(key)
    if user is None:
        return 401, {}
    bucket = state["buckets"].setdefault(
        user, TokenBucket(state["burst"], state["rate"], now)
    )
    if not bucket.allow(now):
        return 429, {"Retry-After": str(max(1, math.ceil(bucket.retry_after())))}
    cost = prompt_tokens + max_tokens
    if state["used"].get(user, 0) + cost > state["limit"]:
        return 429, {"Retry-After": "3600"}
    state["used"][user] = state["used"].get(user, 0) + cost
    return 200, {}

Rather write it yourself?

In the lesson you write this from a starter, in your browser with nothing to install, and checks tell you what's wrong as you go. A free account opens it and all 575 lessons.

Step 3: Try it

The first call has a known key and a fresh bucket, so it is allowed. The second has a key nobody holds, so it is refused before any bucket is made.

state = {"keys": {"k1": "ana"}, "buckets": {}, "used": {}, "limit": 40,
         "burst": 2, "rate": 1.0}
print(authorize(state, "k1", 0.0, 7, 6), authorize(state, "nope", 0.0, 7, 6))
What it prints
(200, {}) (401, {})

In the lesson these functions sit behind FastAPI dependencies, a middleware that stamps each reply with a request id, and a route that streams the model's reply as server-sent events.

Where to take it next

  • Call authorize three times in a row at the same moment, and watch the third answer 429 with a Retry-After.
  • Add a second tier of keys with a bigger burst and budget.
  • Test the budget by passing a limit of 10 and a request that costs 13.

Mistakes to watch for

  • Charging the budget after the model replies, so a call in flight can overspend.
  • Logging the API key or the prompt, which leaks a credential or the user's data.
  • Letting an unknown key through as anonymous, so there is no authentication.
  • Keeping counts in memory and running several copies, so each copy counts on its own.
  • Charging a request that was refused by the bucket or the budget.
The complete program
import math


class TokenBucket:
    def __init__(self, capacity, rate, now):
        self.capacity, self.rate = capacity, rate
        self.tokens, self.updated = capacity, now

    def allow(self, now):
        refill = (now - self.updated) * self.rate
        self.tokens = min(self.capacity, self.tokens + refill)
        self.updated = now
        if self.tokens >= 1:
            self.tokens -= 1
            return True
        return False

    def retry_after(self):
        return max(0.0, (1 - self.tokens) / self.rate)


def authorize(state, key, now, prompt_tokens, max_tokens):
    user = state["keys"].get(key)
    if user is None:
        return 401, {}
    bucket = state["buckets"].setdefault(
        user, TokenBucket(state["burst"], state["rate"], now)
    )
    if not bucket.allow(now):
        return 429, {"Retry-After": str(max(1, math.ceil(bucket.retry_after())))}
    cost = prompt_tokens + max_tokens
    if state["used"].get(user, 0) + cost > state["limit"]:
        return 429, {"Retry-After": "3600"}
    state["used"][user] = state["used"].get(user, 0) + cost
    return 200, {}


state = {"keys": {"k1": "ana"}, "buckets": {}, "used": {}, "limit": 40,
         "burst": 2, "rate": 1.0}
print(authorize(state, "k1", 0.0, 7, 6), authorize(state, "nope", 0.0, 7, 6))

Write it yourself

In the lesson you write this from a starter, in your browser with nothing to install, and checks tell you what's wrong as you go. A free account opens it and all 575 lessons.

Keep building

More tutorials.

This is one stage of the AI engineer roadmap: the bigger picture, in order.

5 steps

Build an AI ad-copy reviewer in Python

A small Python tool that checks ad headlines against your rules, saves a JSON report, and asks a language model to rewrite the worst one.

3 steps

Train a neural network classifier in PyTorch

Standardize from the training rows only, train a small network with a validation score after every epoch, and compare it fairly with logistic regression.

6 steps

Build a GPT from scratch in Python

Write a tiny GPT in PyTorch, train it on a product catalog, and watch it write new descriptions. The whole program is about 70 lines.

9 steps

Build a REST API in Python from scratch

Authentication, validation, pagination, idempotency and clean errors, in a working API built with the standard library so you can see how each one works.

6 steps

Build an MCP server in Python from scratch

A working Model Context Protocol server with tools, resources and prompts, written without an SDK so you can see exactly what goes over the wire.

7 steps

Build a coding agent in Python

An agent that reads and edits a project, runs its tests and stops when they pass, with the safety checks that keep it inside the project.

8 steps

Ship a production AI service in Python

API keys, rate limits, validation, caching, model fallbacks, structured logs and an eval, around a service that classifies support tickets.

7 steps

Build a neural network from scratch with NumPy

A two-layer classifier for trail signs written straight from the formulas: forward pass, cross-entropy, a backward pass, training, an honest test score with its error bar, and next steps in PyTorch.

4 steps

Ship a small machine learning library in Python

The core of a typed, tested training package: a config that rejects unknown settings, mini-batches as a generator, a seeded training loop and an evaluation function.

Never trained an AI? Teach a small one to write, right in your browser. It takes about 20 seconds and needs no account. Teach it to write