Tracing, budgets and guardrails

Loop detection

A common failure: the agent calls the same tool with the same input, gets the same error, and tries again. And again. It's a fly hitting the same window. A turn budget eventually stops it, but expensively. Better: notice repetition. Keep a fingerprint of each call (tool name plus its input as sorted JSON). If the same fingerprint appears, say, 3 times in the last 10 calls, step in:

1. First, tell the model in the tool result: "You've made this exact call 3 times with the same result. Try a different approach."
2. If it carries on, stop the run with a clear status.

In plain words: if the agent keeps doing the same thing and getting the same result, tell it. If that doesn't help, stop it.

import json
from collections import Counter

def fingerprint(tool, args):
    return tool + ":" + json.dumps(args, sort_keys=True)

def watch(calls, window=10, limit=3):
    recent = []
    for n, (tool, args) in enumerate(calls, 1):
        recent = (recent + [fingerprint(tool, args)])[-window:]
        if Counter(recent)[recent[-1]] >= limit:
            return f"step in at call {n}: {tool} repeated {limit} times"
    return "no loop"

calls = [("read", {"path": "a.py", "limit": 10}), ("edit", {"path": "a.py"}),
         ("read", {"limit": 10, "path": "a.py"}), ("run", {"cmd": "test"}),
         ("read", {"path": "a.py", "limit": 10})]
print(watch(calls))
print(watch(calls[:4]))

Shipping a harness

A restaurant tries a new dish on a few tables before it goes on the menu. Ship agents with the same care:

• Version everything: system prompt, tool definitions, model and harness code. Record versions on every trace, so you can tell which change caused what.
• Run the eval suite before each release (previous lesson).
• Roll out gradually: a small share of traffic first, and compare success, cost and complaints.
• Monitor: success rate, cost per task, turns per task, error and refusal rates, and loop stops. Alert on sudden changes.
• Plan for failure: what does the user see when the API is down, a budget is hit, or a run is stopped? Clear statuses beat silent failures.

PRICES = {"input": 5.00, "output": 25.00, "cache_read": 0.50, "cache_write": 6.25}  # $ per million tokens

def cost(usage):
    return (usage["input_tokens"] * PRICES["input"]
            + usage["output_tokens"] * PRICES["output"]
            + usage.get("cache_read_input_tokens", 0) * PRICES["cache_read"]
            + usage.get("cache_creation_input_tokens", 0) * PRICES["cache_write"]) / 1_000_000

first = {"input_tokens": 300, "output_tokens": 400, "cache_creation_input_tokens": 20_000}
later = {"input_tokens": 300, "output_tokens": 400, "cache_read_input_tokens": 20_000}
print(f"first call  ${cost(first):.4f}")
print(f"later calls ${cost(later):.4f}")

prices = {"input": 3.0, "output": 15.0}
usage = {"input_tokens": 2000, "output_tokens": 1000}
cost = (usage["input_tokens"] * prices["input"] + usage["output_tokens"] * prices["output"]) / 1_000_000
print(round(cost, 3))

A turn budget is the last resort

A building has smoke alarms and sprinklers as well as fire doors. A budget will stop a stuck agent eventually, but "eventually" might be $20 and 100 turns later. Loop detection catches it after three identical calls, and a cost limit stops expensive runs whatever they're doing. Use all of them, in layers.

Logs are full of secrets

Agent logs are unusually sensitive. Full prompts contain whatever the agent read: customer records, private code, or keys it found in a config file. So decide on redaction and a retention limit before you start logging, not after. Store the sensitive parts separately, with stricter access, and delete them on a schedule. Metadata such as token counts, tool names and timings can usually be kept for longer.

Prices belong in config

Prices change, and new models arrive. If they're hard-coded across your harness, your cost reports silently drift from reality. Keep one table in config, and record the model id on each span, so that you can recalculate old runs if you need to.

Treat prompts as code

A one-line change to a system prompt can change an agent's behavior more than a thousand lines of code. Yet people often edit prompts directly in production, with no review and no test. So keep prompts in version control. Run your eval suite on every change, and roll out in stages, just as you would with code. Include the prompt version on every trace, so that when behavior shifts, you can tell which change caused it.

When a shipped agent starts misbehaving, ask first what changed: the prompt, the tools, the model, or the data it reads. Those four cover almost every regression you'll see, and your traces should be able to answer it.