Project tutorials · 7 steps

Build a coding agent in Python

An agent that reads and edits a project, runs its tests and stops when they pass, with the safety checks that keep it inside the project.

What you'll build

A coding agent is a model in a loop with file tools. What makes it trustworthy is the harness around the model: it keeps file access inside the project, refuses to change the tests, and decides for itself when the work is done. This tutorial builds all of that with the standard library.

What you'll learn

Combine the loop, tools, permissions, a stop check and an event log into one coding agent
Give the agent file tools that stay inside the project and refuse to change the tests
Make the harness, not the model, decide when the work is done: when the tests pass
Connect the finished harness to a real model with a small adapter

What you need

Python classes and file handling, and what an agent loop is. It uses only the standard library. The demo runs on a scripted model, so it costs nothing and gives the same result every time, and connecting a real model takes one small function.

Step 1: Define the tools

Four tools are enough to fix code: read a file, write a file, edit a file and run the tests. edit_file replaces an exact string, which is safer than rewriting a whole file and uses far fewer tokens on a big one.

import importlib, sys, tempfile
from pathlib import Path

SYSTEM = "You are a coding agent. Fix the code so the tests pass. Never change test files."
TOOLS = [
    {"name": "read_file", "description": "Read a file in the project.", "input_schema": {"type": "object", "properties": {"path": {"type": "string"}}, "required": ["path"]}},
    {"name": "write_file", "description": "Create or replace a file.", "input_schema": {"type": "object", "properties": {"path": {"type": "string"}, "content": {"type": "string"}}, "required": ["path", "content"]}},
    {"name": "edit_file", "description": "Replace an exact string that appears once.", "input_schema": {"type": "object", "properties": {"path": {"type": "string"}, "old": {"type": "string"}, "new": {"type": "string"}}, "required": ["path", "old", "new"]}},
    {"name": "run_tests", "description": "Run the project's tests.", "input_schema": {"type": "object", "properties": {}}},
]

Step 2: Run the project's tests

This runs each test_*.py file and reports what passed and failed, with no test framework needed. It clears cached modules first, so it always tests the code as it is now. A real harness would run pytest in a sandbox, since running code a model just wrote is exactly the kind of thing to isolate.

def run_tests(root):
    root = Path(root)
    passed, failures = 0, []
    sys.path.insert(0, str(root))
    old_flag, sys.dont_write_bytecode = sys.dont_write_bytecode, True
    try:
        for test_file in sorted(root.glob("test_*.py")):
            for name in [m for m in list(sys.modules) if (root / f"{m}.py").exists()]:
                del sys.modules[name]
            importlib.invalidate_caches()
            ns = {"__name__": "tests"}
            try:
                exec(compile(test_file.read_text(), str(test_file), "exec"), ns)
            except Exception as err:
                failures.append(f"ERROR {test_file.name}: {type(err).__name__}: {err}")
                continue
            for name, fn in list(ns.items()):
                if name.startswith("test_") and callable(fn):
                    try:
                        fn()
                        passed += 1
                    except Exception as err:
                        failures.append(f"FAIL {name}: {err}")
    finally:
        sys.path.remove(str(root))
        sys.dont_write_bytecode = old_flag
    summary = f"{passed} passed, {len(failures)} failed" + "".join("\n" + f for f in failures)
    return {"passed": passed > 0 and not failures, "summary": summary}

Step 3: Keep the agent inside the project

_path resolves the path and refuses anything outside the project, which stops ../../.ssh/id_rsa. _writable refuses test files, because an agent told to make the tests pass will otherwise sometimes edit the tests. Enforcing this in code is what makes it a rule. A sentence in the prompt is only a request.

class ToolError(Exception):
    pass


class CodingAgent:
    def __init__(self, model, root, max_turns=12):
        self.model = model
        self.root = Path(root).resolve()
        self.max_turns = max_turns
        self.events = []

    def _path(self, path):
        full = (self.root / path).resolve()
        if not full.is_relative_to(self.root):
            raise ToolError(f"Path outside project: {path}")
        return full

    def _writable(self, path):
        full = self._path(path)
        if full.name.startswith("test_"):
            raise ToolError("Denied: test files are read-only.")
        return full

Rather write it yourself?

In the lesson you write this from a starter, in your browser with nothing to install, and checks tell you what's wrong as you go. A free account opens it and all 244 lessons.

Create a free account

Step 4: Write the file tools

_edit_file insists that old appears exactly once. If it appears twice, the agent has to be more specific, and it can't change the wrong one by accident. Each error says what went wrong, and the model reads that message.

    def _read_file(self, path):
        full = self._path(path)
        if not full.is_file():
            raise ToolError(f"No such file: {path}")
        return full.read_text()

    def _write_file(self, path, content):
        full = self._writable(path)
        full.parent.mkdir(parents=True, exist_ok=True)
        full.write_text(content)
        return f"Wrote {path}"

    def _edit_file(self, path, old, new):
        full = self._writable(path)
        if not full.is_file():
            raise ToolError(f"No such file: {path}")
        text = full.read_text()
        n = text.count(old)
        if n != 1:
            raise ToolError(f"old must appear exactly once in {path} (found {n})")
        full.write_text(text.replace(old, new))
        return f"Edited {path}"

    def _run_tests(self):
        return run_tests(self.root)["summary"]

Step 5: Turn tool failures into messages

Errors go back to the model as results rather than crashing the run. It can then read No such file and try something else. Only the expected failures are caught here, so a bug in the harness itself still shows up as a bug.

    def run_tool(self, name, args):
        handlers = {"read_file": self._read_file, "write_file": self._write_file,
                    "edit_file": self._edit_file, "run_tests": self._run_tests}
        if name not in handlers:
            return (f"Unknown tool: {name}", True)
        try:
            return (handlers[name](**args), False)
        except ToolError as err:
            return (str(err), True)
        except TypeError:
            return (f"Bad arguments for {name}", True)

Step 6: Write the loop, and let the tests decide

Each turn, the model either asks for tools or says it's finished. The key design choice comes next: when the model says it's done, the harness runs the tests itself. If they fail, the failures go back to the model. It gets two more chances, and the turn limit stops a run that's going nowhere. The model doesn't get to mark its own homework.

    def run(self, task):
        messages = [{"role": "user", "content": task}]
        checks, summary = 0, ""
        for turn in range(1, self.max_turns + 1):
            reply = self.model(SYSTEM, messages, TOOLS)
            reason = reply["stop_reason"]
            messages.append({"role": "assistant", "content": reply["content"]})
            self.events.append(("model", reason))
            if reason == "tool_use":
                results = []
                for block in reply["content"]:
                    if block["type"] != "tool_use":
                        continue
                    self.events.append(("tool", block["name"]))
                    content, is_error = self.run_tool(block["name"], block["input"])
                    result = {"type": "tool_result", "tool_use_id": block["id"], "content": content}
                    if is_error:
                        result["is_error"] = True
                    results.append(result)
                messages.append({"role": "user", "content": results})
                continue
            if reason != "end_turn":
                return {"status": "stopped", "turns": turn, "summary": reason}
            report = run_tests(self.root)
            summary = report["summary"]
            self.events.append(("tests", report["passed"]))
            if report["passed"]:
                return {"status": "passed", "turns": turn, "summary": summary}
            if checks >= 2:
                return {"status": "failed", "turns": turn, "summary": summary}
            checks += 1
            messages.append({"role": "user", "content": "The tests still fail:\n" + summary + "\nKeep working until they pass."})
        return {"status": "max_turns", "turns": self.max_turns, "summary": ""}

Step 7: Try it on a real bug

The demo creates a project where add subtracts, along with a test that catches it. A scripted model plays the agent's part, so you can see the loop work for free. It runs the tests, edits the bug and says it's done, and the harness confirms by running the tests itself.

root = Path(tempfile.mkdtemp())
(root / "calc.py").write_text("def add(a, b):\n    return a - b\n")
(root / "test_calc.py").write_text('from calc import add\n\ndef test_add():\n    assert add(2, 2) == 4, f"add(2, 2) gave {add(2, 2)}"\n')
replies = iter([
    {"stop_reason": "tool_use", "content": [{"type": "tool_use", "id": "1", "name": "run_tests", "input": {}}]},
    {"stop_reason": "tool_use", "content": [{"type": "tool_use", "id": "2", "name": "edit_file", "input": {"path": "calc.py", "old": "a - b", "new": "a + b"}}]},
    {"stop_reason": "end_turn", "content": [{"type": "text", "text": "Fixed add."}]},
])
agent = CodingAgent(lambda system, messages, tools: next(replies), root)
print(agent.run("The tests fail. Fix it."))
print(agent.events)

What it prints

{'status': 'passed', 'turns': 3, 'summary': '1 passed, 0 failed'}
[('model', 'tool_use'), ('tool', 'run_tests'), ('model', 'tool_use'), ('tool', 'edit_file'), ('model', 'end_turn'), ('tests', True)]

To use a real model, pass a function with the same three inputs that calls your provider's API and returns its response. The event log is what you'd read when a run goes wrong.

Where to take it next

Connect a real model, and start on a small project you can afford to break.
Run tests in a container, not in your own process.
Add a permission prompt before anything that changes files outside a scratch copy.
Trim old tool results as the conversation grows, so it fits the context window.

Mistakes to watch for

Trusting the model's "done" without checking.
Relying on the prompt alone to keep the agent away from the tests.
Retrying forever when the tests keep failing.
Thinking a real model needs a different harness than the scripted one.

The complete program

import importlib, sys, tempfile
from pathlib import Path

SYSTEM = "You are a coding agent. Fix the code so the tests pass. Never change test files."
TOOLS = [
    {"name": "read_file", "description": "Read a file in the project.", "input_schema": {"type": "object", "properties": {"path": {"type": "string"}}, "required": ["path"]}},
    {"name": "write_file", "description": "Create or replace a file.", "input_schema": {"type": "object", "properties": {"path": {"type": "string"}, "content": {"type": "string"}}, "required": ["path", "content"]}},
    {"name": "edit_file", "description": "Replace an exact string that appears once.", "input_schema": {"type": "object", "properties": {"path": {"type": "string"}, "old": {"type": "string"}, "new": {"type": "string"}}, "required": ["path", "old", "new"]}},
    {"name": "run_tests", "description": "Run the project's tests.", "input_schema": {"type": "object", "properties": {}}},
]

def run_tests(root):
    root = Path(root)
    passed, failures = 0, []
    sys.path.insert(0, str(root))
    old_flag, sys.dont_write_bytecode = sys.dont_write_bytecode, True
    try:
        for test_file in sorted(root.glob("test_*.py")):
            for name in [m for m in list(sys.modules) if (root / f"{m}.py").exists()]:
                del sys.modules[name]
            importlib.invalidate_caches()
            ns = {"__name__": "tests"}
            try:
                exec(compile(test_file.read_text(), str(test_file), "exec"), ns)
            except Exception as err:
                failures.append(f"ERROR {test_file.name}: {type(err).__name__}: {err}")
                continue
            for name, fn in list(ns.items()):
                if name.startswith("test_") and callable(fn):
                    try:
                        fn()
                        passed += 1
                    except Exception as err:
                        failures.append(f"FAIL {name}: {err}")
    finally:
        sys.path.remove(str(root))
        sys.dont_write_bytecode = old_flag
    summary = f"{passed} passed, {len(failures)} failed" + "".join("\n" + f for f in failures)
    return {"passed": passed > 0 and not failures, "summary": summary}


class ToolError(Exception):
    pass


class CodingAgent:
    def __init__(self, model, root, max_turns=12):
        self.model = model
        self.root = Path(root).resolve()
        self.max_turns = max_turns
        self.events = []

    def _path(self, path):
        full = (self.root / path).resolve()
        if not full.is_relative_to(self.root):
            raise ToolError(f"Path outside project: {path}")
        return full

    def _writable(self, path):
        full = self._path(path)
        if full.name.startswith("test_"):
            raise ToolError("Denied: test files are read-only.")
        return full

    def _read_file(self, path):
        full = self._path(path)
        if not full.is_file():
            raise ToolError(f"No such file: {path}")
        return full.read_text()

    def _write_file(self, path, content):
        full = self._writable(path)
        full.parent.mkdir(parents=True, exist_ok=True)
        full.write_text(content)
        return f"Wrote {path}"

    def _edit_file(self, path, old, new):
        full = self._writable(path)
        if not full.is_file():
            raise ToolError(f"No such file: {path}")
        text = full.read_text()
        n = text.count(old)
        if n != 1:
            raise ToolError(f"old must appear exactly once in {path} (found {n})")
        full.write_text(text.replace(old, new))
        return f"Edited {path}"

    def _run_tests(self):
        return run_tests(self.root)["summary"]

    def run_tool(self, name, args):
        handlers = {"read_file": self._read_file, "write_file": self._write_file,
                    "edit_file": self._edit_file, "run_tests": self._run_tests}
        if name not in handlers:
            return (f"Unknown tool: {name}", True)
        try:
            return (handlers[name](**args), False)
        except ToolError as err:
            return (str(err), True)
        except TypeError:
            return (f"Bad arguments for {name}", True)

    def run(self, task):
        messages = [{"role": "user", "content": task}]
        checks, summary = 0, ""
        for turn in range(1, self.max_turns + 1):
            reply = self.model(SYSTEM, messages, TOOLS)
            reason = reply["stop_reason"]
            messages.append({"role": "assistant", "content": reply["content"]})
            self.events.append(("model", reason))
            if reason == "tool_use":
                results = []
                for block in reply["content"]:
                    if block["type"] != "tool_use":
                        continue
                    self.events.append(("tool", block["name"]))
                    content, is_error = self.run_tool(block["name"], block["input"])
                    result = {"type": "tool_result", "tool_use_id": block["id"], "content": content}
                    if is_error:
                        result["is_error"] = True
                    results.append(result)
                messages.append({"role": "user", "content": results})
                continue
            if reason != "end_turn":
                return {"status": "stopped", "turns": turn, "summary": reason}
            report = run_tests(self.root)
            summary = report["summary"]
            self.events.append(("tests", report["passed"]))
            if report["passed"]:
                return {"status": "passed", "turns": turn, "summary": summary}
            if checks >= 2:
                return {"status": "failed", "turns": turn, "summary": summary}
            checks += 1
            messages.append({"role": "user", "content": "The tests still fail:\n" + summary + "\nKeep working until they pass."})
        return {"status": "max_turns", "turns": self.max_turns, "summary": ""}


root = Path(tempfile.mkdtemp())
(root / "calc.py").write_text("def add(a, b):\n    return a - b\n")
(root / "test_calc.py").write_text('from calc import add\n\ndef test_add():\n    assert add(2, 2) == 4, f"add(2, 2) gave {add(2, 2)}"\n')
replies = iter([
    {"stop_reason": "tool_use", "content": [{"type": "tool_use", "id": "1", "name": "run_tests", "input": {}}]},
    {"stop_reason": "tool_use", "content": [{"type": "tool_use", "id": "2", "name": "edit_file", "input": {"path": "calc.py", "old": "a - b", "new": "a + b"}}]},
    {"stop_reason": "end_turn", "content": [{"type": "text", "text": "Fixed add."}]},
])
agent = CodingAgent(lambda system, messages, tools: next(replies), root)
print(agent.run("The tests fail. Fix it."))
print(agent.events)

Write it yourself

In the lesson you write this from a starter, in your browser with nothing to install, and checks tell you what's wrong as you go. A free account opens it and all 244 lessons.

Create a free account