Capstone: research agent

Making citations checkable

The system prompt asks for citations in one fixed format, so code can find them:

Cite every fact with the id it came from in square brackets, like [doc_returns] or [cmp_104].

While the agent runs, your code collects every id that appears in any tool result. When it finishes, three checks grade the answer:

1. The citations are valid. There's at least one, and every cited id is in the set the agent actually saw. A citation to a document it never saw is invented. Recording every id the agent looked at is what makes this check possible: each citation is checked against that set.
2. The right source is cited. Each test case names the document or campaign a correct answer must rely on.
3. The key fact is there. Each test case names a phrase a correct answer must contain, such as "recycled polyester" or "cmp_104".

import re

answer = "Say 'made with recycled polyester' instead of eco-friendly [doc_voice]. Our best campaign is cmp_104 [cmp_104] [doc_ads]."
seen = {"doc_voice", "doc_sustainability", "cmp_104"}

cited = re.findall(r"\[((?:doc|cmp)_[a-z0-9_]+)\]", answer)
print("cited:", cited)
print("invented:", [c for c in cited if c not in seen])

It prints cited: ['doc_voice', 'cmp_104', 'doc_ads'], then invented: ['doc_ads']. The agent never saw doc_ads, so that citation is made up. The pattern finds anything in square brackets that starts with doc_ or cmp_.

These checks don't prove an answer is perfect. They catch invented sources, missing evidence and missing facts, cheaply, on every run. For tone or completeness, you'd add an LLM judge.

doc = "Returns are free for 30 days."
cited = "free for 30 days"
print(doc.find(cited) >= 0)

import re
answer = "Say 'made with recycled polyester' instead of eco-friendly [doc_voice]."
seen = {"doc_voice", "doc_sustainability"}
cited = re.findall(r"\[((?:doc|cmp)_[a-z0-9_]+)\]", answer)
checks = {
    "valid citations": bool(cited) and all(c in seen for c in cited),
    "right source": "doc_voice" in cited,
    "key fact": "recycled polyester" in answer,
}
print(checks)
print(all(checks.values()))

After the exercise: what to improve

When your eval reports failures, read them before changing anything. Each kind of failure points somewhere different:

• The right source never appears in "seen". Retrieval or the choice of tool failed. Improve the tool descriptions or the search query, or add hybrid search.
• The source was seen but not cited. Tighten the citation instruction, or add an example.
• The fact is missing. The answer may be right but worded differently. Decide whether the check or the answer is wrong. For open-ended facts, use an LLM judge.
• The turn limit was reached. The agent is going in circles. Look at its steps. Often a tool returns too little information, or an error it can't recover from.

Then extend it with the rest of Phase 1. Cache the system prompt and tools. Log every run with a trace of its tool calls. Add an approval step before any tool that changes data. And put it behind a small backend with rate limits. With a real API key, run the eval after every change to the prompt or the model.

For marketers: you now know the design of most AI products you'll be pitched: a model, a few tools over company data, a loop, citations and an eval. Ask vendors how their version handles each part, especially the eval.

Read the failures before you change anything

When an eval case fails, read why before touching the prompt. The cause may be retrieval, or a tool, and a prompt change won't fix those. And let code do the arithmetic.

Judge on many questions

One hand-picked question proves very little.