Capstone: ship a production AI service

import hashlib

def cache_key(version, text):
    return hashlib.sha256(f"{version}|{text}".encode()).hexdigest()[:10]

cache = {cache_key("v1", "Where's my order?"): "shipping"}

print(cache_key("v1", "Where's my order?") in cache)
print(cache_key("v1", "Where is my order?") in cache)
print(cache_key("v2", "Where's my order?") in cache)

Logs that help without hurting

A doctor's appointment book says who came and when. It doesn't say what they talked about. Each request gets exactly one log entry: request_id, the first 6 characters of the key plus "...", the status, the source (which model, or "rules") and whether the cache was used. Never the full key, which would be a leaked credential, and never the ticket text, which is customer data.

In plain words: log enough to find out what happened, and nothing you'd hate to see leaked.

import re, hashlib

def redact(text):
    text = re.sub(r"[\w.+-]+@[\w-]+\.[\w.]+", "[email]", text)
    text = re.sub(r"\b\d(?:[ -]?\d){12,15}\b", "[card]", text)
    return text

def log_line(request_id, ticket_text):
    return {"request_id": request_id, "chars": len(ticket_text),
            "text_hash": hashlib.sha256(ticket_text.encode()).hexdigest()[:8],
            "preview": redact(ticket_text)[:60]}

print(log_line("req-41", "Refund to ann@shop.example, card 4242 4242 4242 4242 please"))

import json

CATEGORIES = {"billing", "bug", "account", "feature", "other"}
PRIORITIES = {"low", "normal", "high", "urgent"}

def parse(reply):
    try:
        out = json.loads(reply)
    except json.JSONDecodeError:
        return None
    if isinstance(out, dict) and out.get("category") in CATEGORIES and out.get("priority") in PRIORITIES:
        return {"category": out["category"], "priority": out["priority"]}
    return None

for reply in ['{"category": "bug", "priority": "high"}',
              "Sure! This looks like a bug.",
              '{"category": "complaint", "priority": "high"}']:
    print(parse(reply))

import hashlib

key = lambda version, text: hashlib.sha256((version + "\n" + text).encode()).hexdigest()[:8]
print(key("triage-v3", "Refund please") == key("triage-v4", "Refund please"))

Why the fallback must not be cached

A subtle bug is waiting in the cache. When the model fails, the service returns a safe fallback answer, and that's correct. But if you cache it as you would a model answer, then that fallback keeps being served for that text long after the model has recovered. A short outage becomes a lasting wrong answer. So cache only real model results, and mark fallback responses so that you can count them.

A model won't always return valid JSON

It's easy to believe that a good prompt guarantees the format. It doesn't. A model may wrap the JSON in a sentence, cut it off at the token limit, invent a category you never listed, or refuse. Every one of those will reach your parser at some point in production. So parse it, check it against the allowed values, and have a plan for each failure. The check isn't there because the model is bad. It's there because the model's output is input from outside your system, and you've learned how to treat that.

Who's calling comes first

Check the API key before you validate the body. Otherwise a stranger can send junk and read your validation messages, or use up your rate limits and your compute with no key at all. Authentication first is also cheaper: rejecting a bad key costs almost nothing, while reading and validating a large body doesn't.

Logging for the person on call

Whoever's on call at 3 a.m. needs to know what happened to a request, not what the customer wrote. A request id, the outcome, the latency and whether it came from the cache will answer almost every question. Full API keys and customer messages won't help, and they turn your logs into a liability. If you do need a key in the logs, keep only the last four characters, enough to tell which one it was.