Shipping an LLM app

Protect the endpoint

Your endpoint spends money every time it's called. Treat it like a payment form:

1. Know who's calling. Require sign-in, or at least an access code for internal tools. An open endpoint will be found and used.
2. Limit input size. Cap characters, files and conversation length. Long inputs cost money and give attackers room for injection.
3. Rate-limit each user. A common method is the token bucket. Each user has a bucket of, say, 5 requests. It refills at one request every 10 seconds. Short bursts are fine, but steady hammering gets a 429 Too Many Requests.
4. Cap spending. Set monthly spending limits in your provider's console, with alerts well below them. A daily budget per user catches the one account that's running a script on your app.
5. Set max_tokens and timeouts on every call, so no single request can run away.

from collections import defaultdict

LIMIT = 3
counts = defaultdict(int)

def allowed(user, minute):
    key = (user, minute)
    counts[key] += 1
    return counts[key] <= LIMIT

for n in range(5):
    print("ann", "minute 1:", allowed("ann", 1))
print("ben", "minute 1:", allowed("ben", 1))
print("ann", "minute 2:", allowed("ann", 2))

import os
os.environ["API_KEY"] = "secret"
print("secret" in "fetch('/api/chat')")

Fail gracefully

The model API will sometimes be slow, rate-limited or overloaded. Plan for it:

• Friendly errors, no internals. Show "The assistant is busy, please try again", and log the details for yourself. Never show users the raw error messages from your code or the API. They leak information and confuse people.
• Fallbacks. If the main model is still overloaded after retries, try a smaller model, a saved answer to a common question, or a link to the help center.
• Stream long answers, so users see progress, and let them cancel.
• Timeouts that match your hosting. Some hosts cut requests off after 10 to 60 seconds, which is shorter than a long answer takes. Streaming, or running the work in the background, avoids that.

Deploying it

Hosting services such as Railway, Render, Fly.io and Vercel run a small backend straight from a Git repository. Push to the main branch, and they build and deploy it. You set ANTHROPIC_API_KEY in the host's environment settings, never in the repository. Add a health check, a simple address the host can call to check the app is running. And keep a staging copy of the app, where you try changes before they reach real users.

Launch checklist

For marketers: before launching a customer-facing AI feature, go through the checklist with your developer and your legal or compliance contact. "What happens when it's wrong?" deserves an answer before launch, not after.

MAX_CHARS = 2000
DAILY_BUDGET = 1.00
spent_today = {"sam": 0.99, "alex": 0.10}

def allowed(user, text, cost=0.02):
    if not text.strip() or len(text) > MAX_CHARS:
        return "400: too long or empty"
    if spent_today.get(user, 0) + cost > DAILY_BUDGET:
        return "429: daily budget used"
    return "ok"

print(allowed("alex", "Do gift cards expire?"))
print(allowed("sam", "Do gift cards expire?"))
print(allowed("alex", "x" * 3000))