Capstone: build a coding agent

import importlib, sys, tempfile
from pathlib import Path

def run_tests(root):
    root = Path(root)
    passed, failures = 0, []
    sys.path.insert(0, str(root))
    old_flag, sys.dont_write_bytecode = sys.dont_write_bytecode, True
    try:
        for test_file in sorted(root.glob("test_*.py")):
            for name in [m for m in list(sys.modules) if (root / f"{m}.py").exists()]:
                del sys.modules[name]
            importlib.invalidate_caches()
            ns = {"__name__": "tests"}
            try:
                exec(compile(test_file.read_text(), str(test_file), "exec"), ns)
            except Exception as err:
                failures.append(f"ERROR {test_file.name}: {type(err).__name__}: {err}")
                continue
            for name, fn in list(ns.items()):
                if name.startswith("test_") and callable(fn):
                    try:
                        fn()
                        passed += 1
                    except Exception as err:
                        failures.append(f"FAIL {name}: {err}")
    finally:
        sys.path.remove(str(root))
        sys.dont_write_bytecode = old_flag
    summary = f"{passed} passed, {len(failures)} failed" + "".join("\n" + f for f in failures)
    return {"passed": passed > 0 and not failures, "summary": summary}

root = Path(tempfile.mkdtemp())
(root / "calc.py").write_text("def add(a, b):\n    return a - b\n")
(root / "test_calc.py").write_text('from calc import add\n\ndef test_add():\n    assert add(2, 2) == 4, f"add(2, 2) gave {add(2, 2)}"\n')
print(run_tests(root))

Plugging in a real model

Remember the flight simulator. The pilot's controls stay the same when they move to the real plane. Because the model is injected, switching from the scripted model to Claude is one small adapter. With an API key on your own machine, using the official SDK:

import anthropic

client = anthropic.Anthropic()   # reads ANTHROPIC_API_KEY from the environment

def claude(system, messages, tools):
    reply = client.messages.create(model="claude-opus-5-5", max_tokens=4000,
                                   system=system, tools=tools, messages=messages)
    return {"stop_reason": reply.stop_reason,
            "content": [block.model_dump(exclude_none=True) for block in reply.content]}

agent = CodingAgent(claude, "path/to/project")
print(agent.run("The tests in test_calc.py fail. Find the bug and fix it."))

model_dump(exclude_none=True) turns the SDK's block objects into the plain dicts the harness uses. From there, add the pieces from this phase one at a time, and run your eval suite after each, to see what actually helps.

Why the prompt isn't enough

A "do not touch" sign isn't a lock. You could write "do not edit the tests" in the system prompt. It would work most of the time, and that's the problem. Under pressure from a failing test, with several attempts already used, a model may decide that the test is what's wrong. It may even be right sometimes. But your check on the agent's work is worthless if the agent can rewrite it.

So the harness enforces the rule. Test files are read-only at the tool level, and no amount of clever reasoning changes that. As a general rule, use prompts to guide and code to guarantee.

What happens when it keeps failing

Without a limit, an agent that can't solve a problem keeps trying forever, spending money each turn. So cap the turns, and when the budget runs out, stop and report honestly: what it tried, which tests still fail, and what state it left the files in. Reporting failure clearly is a feature. A person can pick up from there, which is much better than a silent loop, or a confident claim that everything works.

Don't trust "done"

A model saying it has finished is a statement, not a fact. It may have stopped because it believed it was done, or ran out of ideas, or got confused. The harness runs the tests itself on every end_turn. If they fail, it sends the failures back and continues. Only the test results decide.

The real model changes nothing

When you plug in the real API, the harness stays exactly the same. The scripted model and the real one are both functions that take messages and tools, and return a reply. That's the payoff of injecting the model from the start: every path you tested with a script, including errors, timeouts and bad tool calls, is still tested. Real models do add one thing, which is variety. Run each task several times, as you learned in the eval lesson, since a single pass doesn't mean much.

Where to take it next

From here, you could add a real sandbox for the commands, a smarter way to pick which files to read, or the ability to ask the person a question when it's stuck. Each is a small change to one part. The loop in the middle stays as it is, and that's what makes a well-designed harness easy to grow.