Designing AI systems

When the model fails

If your usual restaurant is closed, you try the one next door, and if that's full you cook at home. A request to a model can fail in several ways: an error (429, 529, timeout), a refusal, or output that fails your checks (bad JSON, a missing field, an answer outside the allowed set). Plan a chain:

1. Retry temporary errors with backoff, a couple of times.
2. Fall back to another model, or a smaller one, if the first keeps failing.
3. Degrade gracefully: a cached answer, a rule-based answer, or a clear "try again soon".

A circuit breaker stops you hammering a provider that's down. After several failures in a row, it opens: calls are skipped at once for a cooldown period, and the fallback is used. Then it lets one trial call through (half-open). If that succeeds it closes again; if not, it reopens. This keeps users from waiting on timeouts, and gives the provider room to recover.

In plain words: decide now what users will see when the model is down, so they never just get a spinner.

class Temporary(Exception): pass

def chain(call_main, call_backup, default):
    path = []
    for attempt in range(2):
        try:
            path.append("main")
            return call_main(), path
        except Temporary:
            path.append("retry")
    try:
        path.append("backup")
        return call_backup(), path
    except Temporary:
        path.append("default")
        return default, path

def fails():
    raise Temporary()

print(chain(lambda: "great answer", lambda: "ok answer", "Try again soon"))
print(chain(fails, lambda: "ok answer", "Try again soon"))
print(chain(fails, fails, "Try again soon"))

Guardrails and quality

A new hire's work gets checked before it goes to a customer, especially when mistakes are costly. Do the same with a model:

• Validate inputs: size limits, file types, and treat user content as data, not instructions.
• Validate outputs: use structured outputs or JSON schemas, check values against allowed sets, and handle refusals.
• Humans for high stakes: a person approves refunds, medical or legal advice, or anything irreversible.
• Evals as a gate: an eval set runs before any prompt or model change ships, and production samples are scored to catch drift.
• Privacy: know what data goes to the provider, how long it's kept, and in which region. Strip what the model doesn't need.

import json

ALLOWED = {"billing", "shipping", "returns"}

def route(reply):
    try:
        data = json.loads(reply)
    except ValueError:
        return "human", "not JSON"
    if data.get("team") not in ALLOWED:
        return "human", f"unknown team {data.get('team')!r}"
    if not 0 <= data.get("confidence", -1) <= 1:
        return "human", "bad confidence"
    return data["team"], "ok"

print(route('{"team": "billing", "confidence": 0.9}'))
print(route('Sure! The team is billing.'))
print(route('{"team": "legal", "confidence": 0.9}'))

def route(question):
    hard_words = ("why", "compare", "design", "explain", "trade-off")
    if len(question.split()) > 40 or any(w in question.lower() for w in hard_words):
        return ["large", "small"]
    return ["small", "large"]

for q in ["What are your opening hours?",
          "Compare the two plans and explain which suits a team of five."]:
    print(route(q), "<-", q)

def route(question):
    hard = ("why", "compare", "design")
    return "large" if any(w in question.lower() for w in hard) else "small"

print(route("What time do you open?"), route("Why did my order fail?"))

A model call isn't a function call

Ordinary code gives the same answer each time, in microseconds, for free, and it fails in ways you can list. A model call is slow, costs money, can be refused, and might return something plausible and wrong. Design with all of that in mind, and treat the model like a helpful remote service that sometimes has a bad day.

Use the smallest model that passes your evals

It's natural to pick the most capable model and leave it there. But most requests in a real product are easy. You don't send a surgeon to put on a bandage. Classifying a ticket or pulling out a date rarely needs your largest model, and a smaller one may cost 40 times less. Routing easy work to it is usually the biggest saving available.

Don't retry a failing provider forever

If a provider has an outage, retrying only adds to the pile, and your users wait. Set a limit, and then fall back to another model or a degraded answer. Track how often that happens, since a quiet rise in fallbacks may be your first sign of trouble.

Switching models is a code change

A new model version looks like a one-line edit. It can change tone, length, formatting and what it refuses, and any of those can break your product. Rerun your evals first, and roll it out in stages, as you would with any other risky change.