Index Ai Academy
Project tutorials · 7 steps

How to build an AI agent in Python, from scratch

Build a research agent with tools, an agent loop, citations you can check and an eval, in plain Python with no framework.

What you'll build

An agent is a loop: the model asks for a tool, your code runs it, and the result goes back to the model until it can answer. This tutorial builds one for a shop's marketing team, using nothing but requests. It answers questions from documents and campaign data, cites where each fact came from, and comes with an eval that grades it automatically.

What you'll learn

  • Combine retrieval, tools and an agent loop into one assistant
  • Make every claim traceable to a source the agent actually looked at
  • Grade the agent automatically with code checks on a small eval set
  • Decide what to improve next from the eval's failures

What you need

Python, and what an API call is. In the lesson it runs in your browser with a practice API already there. To run it on your own machine, set API to https://www.indexaiacademy.com/mock/v1, which serves the same practice data, and replace the x-api-key value with your own Anthropic key. API calls cost money.

Step 1: Decide what the agent can do

Start with the system prompt and the tool list, because they're what the model actually reads. The prompt fixes one citation format, like [doc_returns], so code can find citations later. Each tool's description says when to call it: the model chooses tools from those words alone, so vague descriptions lead to wrong choices.

import json
import re
import requests

API = "https://api.gradient.test/v1"
ID_PATTERN = r"\b(?:doc|cmp)_[a-z0-9_]+\b"
CITATION = r"\[((?:doc|cmp)_[a-z0-9_]+)\]"
SYSTEM = ("You are a research assistant for the marketing team of an outdoor clothing shop. "
          "Use the tools to look things up; never answer from memory. "
          "Cite every fact with the id it came from in square brackets, like [doc_returns] or [cmp_104]. "
          "If the tools don't give you the answer, say so.")
TOOLS = [
    {"name": "search_docs", "description": "Search the help center and brand voice guide. Returns the ids, titles and scores of the 3 best matches. Call this for any question about policies, products or how we write.",
     "input_schema": {"type": "object", "properties": {"query": {"type": "string"}}, "required": ["query"]}},
    {"name": "get_doc", "description": "Get the full text of one article by id, such as doc_voice. Call this after search_docs, before relying on a policy.",
     "input_schema": {"type": "object", "properties": {"doc_id": {"type": "string"}}, "required": ["doc_id"]}},
    {"name": "list_campaigns", "description": "List campaign ids and names, optionally by status. Call this to find campaigns to compare.",
     "input_schema": {"type": "object", "properties": {"status": {"type": "string", "enum": ["active", "paused", "draft"]}}}},
    {"name": "get_campaign", "description": "Get one campaign's spend, revenue and ROAS by id, such as cmp_104.",
     "input_schema": {"type": "object", "properties": {"campaign_id": {"type": "string"}}, "required": ["campaign_id"]}},
]

Step 2: Write the tools

Each tool is an ordinary function that calls the practice API. get_campaign works out return on ad spend in code rather than leaving the model to divide two numbers, because code doesn't make arithmetic slips. FUNCTIONS maps the names the model uses to the functions that do the work.

def search_docs(query):
    return requests.get(f"{API}/docs", params={"q": query, "limit": 3}).json()["data"]

def get_doc(doc_id):
    response = requests.get(f"{API}/docs/{doc_id}")
    response.raise_for_status()
    return response.json()

def list_campaigns(status=None):
    params = {"status": status} if status else {}
    return [{"id": c["id"], "name": c["name"]} for c in requests.get(f"{API}/campaigns", params=params).json()["data"]]

def get_campaign(campaign_id):
    response = requests.get(f"{API}/campaigns/{campaign_id}")
    response.raise_for_status()
    c = response.json()
    roas = round(c["revenue"] / c["spend"], 2) if c["spend"] else None
    return {"id": c["id"], "name": c["name"], "spend": c["spend"], "revenue": c["revenue"], "roas": roas}

FUNCTIONS = {"search_docs": search_docs, "get_doc": get_doc, "list_campaigns": list_campaigns, "get_campaign": get_campaign}

Step 3: Call the model

One request to the Messages API, with the system prompt, the tools and the conversation so far. The API keeps no memory between calls, so you send the whole conversation every time.

def call_claude(messages):
    response = requests.post("https://api.anthropic.com/v1/messages",
        headers={"x-api-key": "added-by-the-academy-server", "anthropic-version": "2026-10-21"},
        json={"model": "claude-opus-5-5", "max_tokens": 1500, "system": SYSTEM, "tools": TOOLS, "messages": messages},
        timeout=90)
    response.raise_for_status()
    return response.json()

Rather write it yourself?

In the lesson you write this from a starter, in your browser with nothing to install, and checks tell you what's wrong as you go. A free account opens it and all 244 lessons.

Create a free account

Step 4: Run a tool and record what the agent saw

execute runs the tool the model asked for and returns the result in the format the API expects. Two details matter. A failing tool returns an error the model can read and recover from, rather than crashing the run. And every id in a result goes into seen, which is what makes citations checkable later.

def execute(block, seen):
    try:
        content = json.dumps(FUNCTIONS[block["name"]](**block["input"]))
        result = {"type": "tool_result", "tool_use_id": block["id"], "content": content}
    except Exception as e:
        content = f"Error: {e}"
        result = {"type": "tool_result", "tool_use_id": block["id"], "content": content, "is_error": True}
    seen.update(re.findall(ID_PATTERN, content))
    return result

Step 5: Write the agent loop

This is the whole of an agent. Call the model. If it wants tools, run them, add the results to the conversation and go around again. If it doesn't, you have the answer. The turn limit matters, because a confused model can otherwise loop forever, spending money on every turn.

def research(question, max_turns=8):
    messages = [{"role": "user", "content": question}]
    steps, seen = [], set()
    for _ in range(max_turns):
        response = call_claude(messages)
        messages.append({"role": "assistant", "content": response["content"]})
        if response["stop_reason"] != "tool_use":
            answer = "".join(b["text"] for b in response["content"] if b["type"] == "text")
            return {"answer": answer, "steps": steps, "seen": sorted(seen)}
        results = []
        for block in response["content"]:
            if block["type"] == "tool_use":
                steps.append(block["name"])
                results.append(execute(block, seen))
        messages.append({"role": "user", "content": results})
    return {"answer": "Stopped: reached the turn limit.", "steps": steps, "seen": sorted(seen)}

Step 6: Grade answers with code

Three cheap checks. Every citation must be an id the agent really saw, so an invented source fails right away. The answer must cite the source a correct answer relies on. And it must contain a key fact. They don't prove an answer is perfect, but they catch invented sources and missing evidence on every run.

def grade(result, case):
    cited = re.findall(CITATION, result["answer"])
    checks = {
        "citations_valid": bool(cited) and all(c in result["seen"] for c in cited),
        "cites_source": case["must_cite"] in cited,
        "has_fact": case["must_include"].lower() in result["answer"].lower(),
    }
    checks["passed"] = all(checks.values())
    return checks

Step 7: Run an eval on several questions

One hand-picked question proves very little, so run a small set and report the pass rate and what failed. With a real API key, this is three full agent runs, which takes about a minute.

def run_eval(cases, research_fn):
    failed = [case["question"] for case in cases if not grade(research_fn(case["question"]), case)["passed"]]
    return {"pass_rate": round((len(cases) - len(failed)) / len(cases), 2), "failed": failed}

CASES = [
    {"question": "Can we describe the new fleece as eco-friendly in an ad?", "must_cite": "doc_voice", "must_include": "recycled"},
    {"question": "Which active campaign has the best ROAS?", "must_cite": "cmp_104", "must_include": "cmp_104"},
    {"question": "How long do customers have to return an item?", "must_cite": "doc_returns", "must_include": "60 days"},
]
print(run_eval(CASES, research))

When a case fails, read why before changing anything. If the right source never appears in seen, retrieval or the choice of tool failed. If it was seen but not cited, tighten the citation instruction. A prompt change won't fix a broken tool.

Where to take it next

  • Log every run with its tool calls, so you can see what happened when an answer is wrong.
  • Add an approval step before any tool that changes data.
  • Cache the system prompt and tools to cut cost.
  • Run the eval after every change to the prompt or the model.

Mistakes to watch for

  • Trusting citations without checking them against what was retrieved.
  • Letting the model do arithmetic the code could do.
  • Changing the prompt before reading why each eval case failed.
  • Judging the agent on one hand-picked question.
The complete program
import json
import re
import requests

API = "https://api.gradient.test/v1"
ID_PATTERN = r"\b(?:doc|cmp)_[a-z0-9_]+\b"
CITATION = r"\[((?:doc|cmp)_[a-z0-9_]+)\]"
SYSTEM = ("You are a research assistant for the marketing team of an outdoor clothing shop. "
          "Use the tools to look things up; never answer from memory. "
          "Cite every fact with the id it came from in square brackets, like [doc_returns] or [cmp_104]. "
          "If the tools don't give you the answer, say so.")
TOOLS = [
    {"name": "search_docs", "description": "Search the help center and brand voice guide. Returns the ids, titles and scores of the 3 best matches. Call this for any question about policies, products or how we write.",
     "input_schema": {"type": "object", "properties": {"query": {"type": "string"}}, "required": ["query"]}},
    {"name": "get_doc", "description": "Get the full text of one article by id, such as doc_voice. Call this after search_docs, before relying on a policy.",
     "input_schema": {"type": "object", "properties": {"doc_id": {"type": "string"}}, "required": ["doc_id"]}},
    {"name": "list_campaigns", "description": "List campaign ids and names, optionally by status. Call this to find campaigns to compare.",
     "input_schema": {"type": "object", "properties": {"status": {"type": "string", "enum": ["active", "paused", "draft"]}}}},
    {"name": "get_campaign", "description": "Get one campaign's spend, revenue and ROAS by id, such as cmp_104.",
     "input_schema": {"type": "object", "properties": {"campaign_id": {"type": "string"}}, "required": ["campaign_id"]}},
]

def search_docs(query):
    return requests.get(f"{API}/docs", params={"q": query, "limit": 3}).json()["data"]

def get_doc(doc_id):
    response = requests.get(f"{API}/docs/{doc_id}")
    response.raise_for_status()
    return response.json()

def list_campaigns(status=None):
    params = {"status": status} if status else {}
    return [{"id": c["id"], "name": c["name"]} for c in requests.get(f"{API}/campaigns", params=params).json()["data"]]

def get_campaign(campaign_id):
    response = requests.get(f"{API}/campaigns/{campaign_id}")
    response.raise_for_status()
    c = response.json()
    roas = round(c["revenue"] / c["spend"], 2) if c["spend"] else None
    return {"id": c["id"], "name": c["name"], "spend": c["spend"], "revenue": c["revenue"], "roas": roas}

FUNCTIONS = {"search_docs": search_docs, "get_doc": get_doc, "list_campaigns": list_campaigns, "get_campaign": get_campaign}

def call_claude(messages):
    response = requests.post("https://api.anthropic.com/v1/messages",
        headers={"x-api-key": "added-by-the-academy-server", "anthropic-version": "2026-10-21"},
        json={"model": "claude-opus-5-5", "max_tokens": 1500, "system": SYSTEM, "tools": TOOLS, "messages": messages},
        timeout=90)
    response.raise_for_status()
    return response.json()

def execute(block, seen):
    try:
        content = json.dumps(FUNCTIONS[block["name"]](**block["input"]))
        result = {"type": "tool_result", "tool_use_id": block["id"], "content": content}
    except Exception as e:
        content = f"Error: {e}"
        result = {"type": "tool_result", "tool_use_id": block["id"], "content": content, "is_error": True}
    seen.update(re.findall(ID_PATTERN, content))
    return result

def research(question, max_turns=8):
    messages = [{"role": "user", "content": question}]
    steps, seen = [], set()
    for _ in range(max_turns):
        response = call_claude(messages)
        messages.append({"role": "assistant", "content": response["content"]})
        if response["stop_reason"] != "tool_use":
            answer = "".join(b["text"] for b in response["content"] if b["type"] == "text")
            return {"answer": answer, "steps": steps, "seen": sorted(seen)}
        results = []
        for block in response["content"]:
            if block["type"] == "tool_use":
                steps.append(block["name"])
                results.append(execute(block, seen))
        messages.append({"role": "user", "content": results})
    return {"answer": "Stopped: reached the turn limit.", "steps": steps, "seen": sorted(seen)}

def grade(result, case):
    cited = re.findall(CITATION, result["answer"])
    checks = {
        "citations_valid": bool(cited) and all(c in result["seen"] for c in cited),
        "cites_source": case["must_cite"] in cited,
        "has_fact": case["must_include"].lower() in result["answer"].lower(),
    }
    checks["passed"] = all(checks.values())
    return checks

def run_eval(cases, research_fn):
    failed = [case["question"] for case in cases if not grade(research_fn(case["question"]), case)["passed"]]
    return {"pass_rate": round((len(cases) - len(failed)) / len(cases), 2), "failed": failed}

CASES = [
    {"question": "Can we describe the new fleece as eco-friendly in an ad?", "must_cite": "doc_voice", "must_include": "recycled"},
    {"question": "Which active campaign has the best ROAS?", "must_cite": "cmp_104", "must_include": "cmp_104"},
    {"question": "How long do customers have to return an item?", "must_cite": "doc_returns", "must_include": "60 days"},
]
print(run_eval(CASES, research))

Write it yourself

In the lesson you write this from a starter, in your browser with nothing to install, and checks tell you what's wrong as you go. A free account opens it and all 244 lessons.

Create a free account