Cost and latency

Prompt caching

Think of a teacher who photocopies the same worksheet for every class. It's quicker to keep a stack of copies ready than to write it out each time.

When the start of your prompt is the same across requests, the API can keep its processed form for a while and reuse it. That shared start might be instructions, a style guide, documents, tool definitions or the earlier turns of a conversation. You mark where the reusable part ends with cache_control:

"system": [{"type": "text", "text": HELP_CENTER_AND_RULES,
            "cache_control": {"type": "ephemeral"}}],

The rules that make it work:

• Stable first, changing last. The cache matches your prompt from the very beginning up to the marker, exactly, in the order tools, then system, then messages. A timestamp or user name near the top changes the text, so the cache misses every time.
• Writes cost a bit more, reads much less. A write costs 1.25× normal input for the default 5-minute lifetime (2× for the optional 1-hour lifetime). Each read restarts the timer.
• A minimum size. Starts shorter than the model's minimum (512 tokens on current models) aren't cached, and nothing warns you. Check cache_creation_input_tokens to confirm.
• Up to 4 markers per request, for parts that change at different rates. Tools might never change, a document might change daily, and the conversation changes every turn.

Caching also makes replies start sooner, because the model skips re-reading the cached part.

import requests

docs = requests.get("https://api.gradient.test/v1/docs").json()["data"]
help_center = "\n\n".join(f"## {d['title']}\n{d['text']}" for d in docs)
system = [{"type": "text",
           "text": "Answer customer questions using only this help center.\n\n" + help_center,
           "cache_control": {"type": "ephemeral"}}]

for question in ["How long is the return window?", "Do gift cards expire?"]:
    response = requests.post("https://api.anthropic.com/v1/messages",
        headers={"x-api-key": "added-by-the-academy-server", "anthropic-version": "2026-10-21"},
        json={"model": "claude-opus-5-5", "max_tokens": 200, "system": system,
              "messages": [{"role": "user", "content": question}]},
        timeout=60).json()
    u = response["usage"]
    print(question, "| written:", u["cache_creation_input_tokens"], "| read:", u["cache_read_input_tokens"], "| uncached:", u["input_tokens"])

The other cost levers, in order

1. Pick the cheapest model that passes your eval. Haiku 5.5 is 40 times cheaper than Opus 5.5 per token. Classifying, tagging and pulling out fields often work just as well on it. Route by task (the routing pattern), and prove it with an eval.
2. Cap and shrink the output. Set max_tokens sensibly. Ask for the format you need, such as "three bullets" or JSON, instead of essays. Use structured output.
3. Lower the effort for simple tasks. Less thinking means fewer output tokens and faster replies.
4. Batch what can wait. The Message Batches API halves the price of every token, and the discount stacks with caching.
5. Send less. Retrieve the five relevant chunks instead of the whole manual. Return small tool results. Drop old conversation turns you no longer need.

Measure before and after each change. A saving that costs quality will show up in the eval.

What makes replies slow

Two numbers matter:

• time to first token: how long before anything appears,
• total time: how long until the reply is finished.

The first grows with the length of the input and with thinking. The second depends mostly on how much the model writes, because output is produced one token at a time. So:

• Stream the reply, so people see text straight away (the streaming lesson). It doesn't make the answer finish sooner, but it makes it feel faster.
• Write less. Shorter outputs finish sooner.
• Use a smaller model or lower effort where quality allows.
• Cache long, repeated starts.
• Run independent calls at the same time instead of one after another.

For marketers: cost per request × requests per day × 30 is the number your finance team will ask for. Work it out before launch, with caching and model choice already applied.

per_request = 0.0048
print(round(per_request * 10_000 * 30))
print(round(0.0162 * 10_000 * 30))