Managing the context window

Where to cut when compacting

You can't cut anywhere. If you tear pages out of an interview transcript, you mustn't leave an answer without its question. If the recent part starts with a user message of tool_result blocks, their tool_use blocks are in the part you summarized away, and the request is invalid. A safe cut is right before an assistant message: put the summary in a new first user message, then the recent turns from that assistant message on. Roles still alternate, and every result in the recent part has its tool call there too.

The summary prompt matters. Treat it as a handover note at the end of a shift. Ask for the original task, key decisions and their reasons, files touched, errors met and fixed, and the next steps. A summary that loses the task leaves an agent that's lost.

roles = ["user", "assistant", "user", "assistant", "user", "assistant", "user"]
kinds = ["text", "tool_use", "tool_result", "tool_use", "tool_result", "text", "text"]

def safe_cut(roles, keep_recent):
    for i in range(len(roles) - keep_recent, 0, -1):
        if roles[i] == "assistant":
            return i
    return None

cut = safe_cut(roles, keep_recent=3)
print("cut before message", cut, f"({roles[cut]}, {kinds[cut]})")
print("summarized:", kinds[:cut])
print("kept word for word:", kinds[cut:])

cut = safe_cut(roles, keep_recent=2)
print("asking to keep 2 keeps:", kinds[cut:])

The cacheable prefix

If you've already checked the first 50 pages of a contract, you can skip them next time, as long as not one word has changed. Prompt caching does the same thing. It makes a repeated prefix, the unchanged start of a request, cheaper and faster. The order is tools, then system, then messages, so:

• Put stable things first: tool definitions, then the system prompt, then the conversation.
• Don't put the time, a random ID or a changing count in the system prompt; it would change the prefix every call.
• Keep tool order fixed; sort them once.
• To change instructions mid-session, append rather than edit. The current Claude API accepts system role messages later in the conversation for this.

Clearing and compaction do change the prefix, so the next call pays to rebuild the cache. That's why you clear in large batches when crossing a threshold, not a little every turn.

In plain words: keep the start of every request exactly the same, and put whatever changes at the end.

Seeing the cache work

The second call sends the same tools and system prompt, so that prefix is read from the cache. (The practice server follows the same rules: it caches prefixes of 512 tokens or more.)

import requests

SYSTEM = [{"type": "text", "text": "You are a careful coding agent for the gradient-shop project. " * 60,
           "cache_control": {"type": "ephemeral"}}]

def call(question):
    r = requests.post("https://api.anthropic.com/v1/messages",
        headers={"x-api-key": "added-by-the-academy-server", "anthropic-version": "2026-10-21"},
        json={"model": "claude-opus-5-5", "max_tokens": 200, "system": SYSTEM,
              "messages": [{"role": "user", "content": question}]},
        timeout=60)
    r.raise_for_status()
    return r.json()["usage"]

for q in ["List the test files.", "Which tests are slow?"]:
    u = call(q)
    print(q, "| new:", u["input_tokens"], "| written to cache:", u["cache_creation_input_tokens"],
          "| read from cache:", u["cache_read_input_tokens"])

messages = ["task", "a1", "r1", "a2", "r2", "a3", "r3"]
roles = ["user", "assistant", "user", "assistant", "user", "assistant", "user"]
cut = len(messages) - 3
while cut > 0 and roles[cut] != "assistant":
    cut -= 1
print(cut, messages[cut:])

Bigger windows don't remove the work

A million-token window sounds like the end of the problem. It isn't. A bigger desk still gets buried. You pay for every token on every call, so a huge context costs more each turn, and the model pays less attention to details buried deep inside. Managing context is about quality as much as fitting in.

Don't delete results to save space

Removing tool_result blocks breaks the conversation, because each tool_use must have its result. Replace the contents with a placeholder and keep the block.

Keep dynamic values out of the start

Putting today's date or a request id at the top of the system prompt looks harmless. It changes the prefix on every call, so the cache never gets a hit, and you pay full price each time. Put values like that at the end, in the latest message.

Compaction loses things

A movie recap never has every scene, and a summary always drops detail. So write important decisions and plans to files, which survive compaction word for word, and read them back whenever you need them again later on in a session.