Retrieval over a graph
Answer questions that hop between documents: facts as triples, a walk out from the entities a question names, and the hubs and missed facts that limit it.
What it covers7
Why it matters
Retrieval by similarity finds text that looks like the question. Some questions can only be answered by joining facts from pages that look nothing like the question at all: this incident was caused by that service, which belongs to that team, which is led by that person. Agents that work over a company's documents meet these constantly. Retrieval over a graph is the main technique for them. It is also expensive to build and easy to oversell, so this lesson measures both where it wins and where it doesn't.
What you'll be able to do
- Recognize a question that needs several documents joined by things the question doesn't name
- Store facts as triples in a graph, each pointing back to the document it came from
- Retrieve by linking a question to entities and walking a few steps out from them
- Explain why hubs and missed facts limit how far a walk can usefully go
- Decide when a graph is worth building, and combine it with ordinary search
Show the answer
Two of the three pages share almost nothing with the question
The incident report matches the question, since both mention the incident. The catalog page is about a service the question never names, and the team page is about a team it never names. They are connected to the question only through each other. That chain is what a similarity score can't see.
The idea: facts as triples, a walk, and the pages behind them
Recall: Making RAG good showed how search finds pages that look like the question, and Recursion, trees and graphs taught walking a graph. A question that hops between pages that look nothing alike needs both.
Questions that hop
Here are twelve short pages from one company: incident reports, a service catalog, team pages. The search below ranks pages by the words they share with the question, with rare words counting for more. It is a stand-in for the embedding search you built before, and it fails in the same place.
Setup (runs before every cell) · 20 lines
The lesson runs this once, before its first cell, and every cell, drill and example starts with what it defines. A cell that changes a setup object changes it for the cells after it, until the page reloads.
TRIPLES = [ # subject, relation, object, the document it came from ("March outage", "caused by", "ledger-sync", "d1"), ("ledger-sync", "owned by", "Payments Platform", "d2"), ("ledger-sync", "depends on", "invoice-api", "d2"), ("Payments Platform", "led by", "Dana Reyes", "d3"), ("Payments Platform", "located in", "Lisbon office", "d3"), ("Growth", "led by", "Omar Haddad", "d4"), ("notify-hub", "owned by", "Growth", "d4"), ("search-index", "owned by", "Core Data", "d5"), ("Core Data", "led by", "Li Wei", "d6"), ("Core Data", "located in", "Lisbon office", "d6"), ("June outage", "caused by", "search-index", "d7"), ("invoice-api", "owned by", "Payments Platform", "d9"), ("invoice-api", "depends on", "CloudBase", "d9"), ("notify-hub", "depends on", "CloudBase", "d10"), ("CloudBase", "managed by", "Priya Nair", "d10"), ("Growth", "located in", "Lisbon office", "d11"), ("Atlas migration", "moves", "ledger-sync", "d12"), ("Atlas migration", "sponsored by", "Li Wei", "d12"), ]
DOCS = {
"d1": "Incident report. The March outage lasted four hours. The root cause was a "
"failed deploy of ledger-sync.",
"d2": "Service catalog. ledger-sync is owned by the Payments Platform team. It "
"depends on invoice-api.",
"d3": "Team page. The Payments Platform team is led by Dana Reyes and sits in the "
"Lisbon office.",
"d4": "Team page. The Growth team is led by Omar Haddad. The team owns notify-hub "
"and leads the March campaign.",
"d5": "Service catalog. search-index is owned by the Core Data team. Who owns a "
"service is listed in this catalog.",
"d6": "Team page. The Core Data team is led by Li Wei and sits in the Lisbon "
"office.",
"d7": "Incident report. The June outage lasted one hour. The root cause was a full "
"disk on search-index.",
"d8": "Leadership note. Every team that owns a service must name a lead who is on "
"call during an outage.",
"d9": "Service catalog. invoice-api is owned by the Payments Platform team. It "
"depends on CloudBase.",
"d10": "Vendor note. CloudBase is the hosting vendor for invoice-api and "
"notify-hub. The contract is managed by Priya Nair.",
"d11": "Office guide. The Lisbon office hosts the Payments Platform, Core Data and "
"Growth teams.",
"d12": "Project brief. The Atlas migration moves ledger-sync to new hosting. Li "
"Wei sponsors the Atlas migration.",
}
import math
SAME = {
"led": "lead", "leads": "lead", "owned": "own", "owns": "own", "teams": "team",
"hosts": "host", "hosting": "host"}
SKIP = {
"the", "a", "is", "by", "of", "in", "on", "it", "to", "and", "that", "was", "who",
"for", "this", "behind", "what", "which", "how", "did", "does"}
def words(text):
out = set()
for w in text.lower().replace(".", " ").replace(",", " ").replace("?", " ").split():
w = SAME.get(w, w)
if w not in SKIP:
out.add(w)
return out
DOC_WORDS = {d: words(text) for d, text in DOCS.items()}
RARITY = {
w: math.log(
len(DOCS) / sum(w in ws for ws in DOC_WORDS.values())
) for ws in DOC_WORDS.values() for w in ws}
def search(question, k):
"""A stand-in for embedding search: documents that share the question's words,
rare words counting for more."""
q = words(question)
scored = sorted(
DOCS, key=lambda d: (-sum(RARITY[w] for w in q & DOC_WORDS[d]), int(d[1:])))
return scored[:k]
easy = "How long did the March outage last?"
hard = "Who leads the team that owns the service behind the March outage?"
needed = {easy: ["d1"], hard: ["d1", "d2", "d3"]}
for question in (easy, hard):
print(question)
for k in (3, 5):
found = search(question, k)
print(
" top", k, found, "| needed", needed[question], "| found",
sum(d in found for d in needed[question]), "of", len(needed[question]))
print()
print(
"where the three needed documents rank for the hard question:",
{d: search(hard, 12).index(d) + 1 for d in needed[hard]})The easy question needs one page, and it comes first. The hard question needs three, and the top 3 has only one of them. Asking for 5 finds a second. The third, the team page that holds the actual answer, ranks 7th.
Look at what did rank at the top: d8, a leadership note about teams, services, leads and outages. It shares more words with the question than any page that helps. It is about the same things and contains none of the facts.
A better similarity model would not cure this. To know that the page about ledger-sync matters, you must already have read the incident report, since the question never says ledger-sync. The question is a multi-hop question. It has to be answered in hops: from the outage to the service, from the service to the team, from the team to its lead. Each hop goes through something the question didn't name.
There are two ways to make the hops. An agent can search, read, and search again with what it learned, which costs a model call per hop. Or the links can be worked out ahead of time and stored. That store is a graph.
Show the answer
Finds the passages most relevant to the question and puts them in the prompt
RAG answers from your own documents: retrieve the passages that bear on the question, put them in the prompt, and have the model answer from them. Everything in this lesson is about the first step. A graph is another way of choosing what goes into the prompt.
Find the entities a question names
Walking a graph starts from the entities in the question. Write link(question, names). Return the names that appear in the question, ignoring case, as whole words: the character just before and just after the match must not be a letter or a digit. When one matched name lies inside a longer matched name at the same place, as "Payments" does inside "Payments Platform", keep only the longer. Return the names as given in names, in the order they appear in the question.
Uses the setup above, 20 lines.
def link(question, names): return [] print( link( "Who leads Payments Platform, and who owns ledger-sync?", ["ledger-sync", "Payments", "Payments Platform", "Growth"]))
Step the walk. Columns are hops from the starting entity; lit entities have been reached.
Write a sentence or two, then compare it with a good answer in the lesson.
In code: build the graph, link, walk and compare with an agent
Facts as a graph
The links between pages run through the things they mention: a service, a team, a person. Call those entities. A fact connecting two of them can be written as a triple: a subject, a relation, an object. "ledger-sync is owned by the Payments Platform team" becomes (ledger-sync, owned by, Payments Platform).
Collect the triples from every page and you have a knowledge graph: entities as the points, relations as the lines between them. Each triple also remembers which page it came from, which will matter when it is time to give the model something to read.
Uses the setup above, 20 lines.
TRIPLES = [ # subject, relation, object, the document it came from ("March outage", "caused by", "ledger-sync", "d1"), ("ledger-sync", "owned by", "Payments Platform", "d2"), ("ledger-sync", "depends on", "invoice-api", "d2"), ("Payments Platform", "led by", "Dana Reyes", "d3"), ("Payments Platform", "located in", "Lisbon office", "d3"), ("Growth", "led by", "Omar Haddad", "d4"), ("notify-hub", "owned by", "Growth", "d4"), ("search-index", "owned by", "Core Data", "d5"), ("Core Data", "led by", "Li Wei", "d6"), ("Core Data", "located in", "Lisbon office", "d6"), ("June outage", "caused by", "search-index", "d7"), ("invoice-api", "owned by", "Payments Platform", "d9"), ("invoice-api", "depends on", "CloudBase", "d9"), ("notify-hub", "depends on", "CloudBase", "d10"), ("CloudBase", "managed by", "Priya Nair", "d10"), ("Growth", "located in", "Lisbon office", "d11"), ("Atlas migration", "moves", "ledger-sync", "d12"), ("Atlas migration", "sponsored by", "Li Wei", "d12"), ] def build(triples): """entity -> the entities one step away, in either direction.""" graph = {} for subject, relation, obj, source in triples: graph.setdefault(subject, set()).add(obj) graph.setdefault(obj, set()).add(subject) return graph def expand(graph, start, hops): """Every entity within this many steps of the start.""" reached, edge = {start}, [start] for hop in range(hops): edge = sorted({other for node in edge for other in graph[node]} - reached) reached.update(edge) return reached graph = build(TRIPLES) print(len(TRIPLES), "facts about", len(graph), "entities") for entity in ("ledger-sync", "Lisbon office", "Dana Reyes"): print(entity.ljust(14), "->", sorted(graph[entity])) print() print("everything the graph holds about ledger-sync:") for subject, relation, obj, source in TRIPLES: if "ledger-sync" in (subject, obj): print(" ", subject, "|", relation, "|", obj, " (from " + source + ")")
Eighteen facts connect sixteen entities. ledger-sync touches four of them, drawn from three different pages: the incident report, the catalog and a project brief. No single page says all of that. The graph has done the joining ahead of time.
Where do triples come from? In a real system a model reads every page and writes them out. Here they were written by hand, which hides the two hard parts.
- Extraction misses things. Some facts aren't found, and some that are found are wrong.
- Names don't match. One page says "Payments Platform", another "the payments team", a third "PP". Deciding that these are one entity is called entity resolution, and a graph with three separate points for one team has lost exactly the links it was built for.
The page with "lasted four hours" shows a third limit. That duration isn't in any triple. A graph holds the facts someone chose to extract, and prose always holds more.
triples = [
("checkout", "owned by", "Web team"), ("checkout", "depends on", "cart-api"),
("cart-api", "owned by", "Web team")]
graph = {}
for subject, relation, obj in triples:
graph.setdefault(subject, set()).add(obj)
graph.setdefault(obj, set()).add(subject)
print({entity: len(graph[entity]) for entity in sorted(graph)})Show the answer
{'Web team': 2, 'cart-api': 2, 'checkout': 2}
Each triple links its two ends in both directions. The Web team owns two things, so it has two neighbors, even though it is only ever the object. checkout links to the team and to cart-api. cart-api links to checkout and to the team. Storing both directions is what lets a walk go from a team to its services as easily as from a service to its team.
Retrieval by walking
With a graph, retrieval has three steps.
- Link. Find the entities the question names.
- Walk. Go out from them a fixed number of steps, collecting every entity on the way.
- Collect. Take the facts between the entities reached, and the pages those facts came from.
What goes into the prompt is the facts, the pages behind them, or both.
Uses the setup above, 20 lines.
def build(triples): """entity -> the entities one step away, in either direction.""" graph = {} for subject, relation, obj, source in triples: graph.setdefault(subject, set()).add(obj) graph.setdefault(obj, set()).add(subject) return graph def expand(graph, start, hops): """Every entity within this many steps of the start.""" reached, edge = {start}, [start] for hop in range(hops): edge = sorted({other for node in edge for other in graph[node]} - reached) reached.update(edge) return reached graph = build(TRIPLES) question = "Who leads the team that owns the service behind the March outage?" linked = [entity for entity in graph if entity.lower() in question.lower()] print("entities named in the question:", linked) start = linked[0] for hops in (1, 2, 3): reached = expand(graph, start, hops) facts = [t for t in TRIPLES if t[0] in reached and t[2] in reached] print( "hops:", hops, "| entities:", len(reached), "| facts:", len(facts), "| documents:", sorted({t[3] for t in facts}, key=lambda d: int(d[1:]))) def paths(node, steps, seen): """Every way of walking this many steps from a node, as readable text.""" if steps == 0: return [node] out = [] for subject, relation, obj, source in TRIPLES: if subject == node and obj not in seen: out += [ node + " -" + relation + "-> " + rest for rest in paths( obj, steps - 1, seen | {obj})] if obj == node and subject not in seen: out += [ node + " <-" + relation + "- " + rest for rest in paths( subject, steps - 1, seen | {subject})] return out print() people = ("Dana Reyes", "Li Wei", "Omar Haddad", "Priya Nair") for path in paths(start, 3, {start}): if path.endswith(people): print(path)
The question names one entity, the March outage. One hop reaches the service. Two hops reach the team, and a couple of things beside it. At three hops the walk has 9 entities and 9 facts drawn from five pages, and for the first time d1, d2 and d3 are all there: the three pages the search could not put together.
The last two lines show something a pile of pages can't. Three steps from the outage there are two people. One is reached through "caused by", "owned by", "led by". The other is reached because a migration project moves the same service and she sponsors it. The relations say which chain answers the question. Hand the model the facts with their relations, and it can tell the team's lead from someone who merely appears nearby.
That second path is also a warning. The walk brought back d9 and d12, which have nothing to do with the question. A walk gathers everything within reach, and the further it goes the more of that there is.
graph = {
"outage": {"api"}, "api": {"outage", "team", "db"}, "team": {"api", "lead"},
"db": {"api"}, "lead": {"team"}}
reached, edge = {"outage"}, ["outage"]
for hop in range(3):
edge = sorted({other for node in edge for other in graph[node]} - reached)
reached.update(edge)
print(hop + 1, edge)Show the answer
1 ['api'] 2 ['db', 'team'] 3 ['lead']
Step 1 reaches the outage's only neighbor, api. Step 2 goes out from api to db and team, and not back to outage, which was already reached. Step 3 goes from team to lead, and db leads nowhere new. This is a breadth-first search: everything one step away, then everything two steps away.
How far is everything?
Write distances(graph, start, hops). graph maps each entity to the set of its neighbors. Return a dictionary from each entity within hops steps of start to the smallest number of steps needed to reach it. The start itself is at 0. If start isn't in the graph, return an empty dictionary.
Uses the setup above, 20 lines.
def distances(graph, start, hops): return {} graph = { "outage": {"api"}, "api": {"outage", "team", "db"}, "team": {"api", "lead"}, "db": {"api"}, "lead": {"team"}} print(distances(graph, "outage", 2))
The agent's alternative: search, read, search again
Without a graph, an agent can follow the same chain by searching, reading what it finds and searching again, one model call per hop. The stand-in below does the thinking with a rule.
Uses the setup above, 20 lines.
PAGES = {
"d1": "March outage caused by ledger-sync",
"d2": "ledger-sync owned by Payments Platform",
"d3": "Payments Platform led by Dana Reyes",
}
def search(query):
words = set(query.lower().split())
return max(PAGES, key=lambda d: len(words & set(PAGES[d].lower().split())))
def next_query(page):
# a stand-in for one model call: read the page, name the next thing to look up
entity = page.split(" by ")[-1]
return {
"ledger-sync": "ledger-sync owned", "Payments Platform": "Payments Platform led"
}.get(entity, "")
query, calls = "March outage caused", 0
while query:
page = PAGES[search(query)]
calls += 1
print(calls, "|", query, "->", page)
query = next_query(page)
print("model calls to follow the chain:", calls)Write a sentence or two, then compare it with a good answer in the lesson.
In a production harness: costs, networkx and when a graph is worth it
What it costs, and when it is worth it
Two things limit how far a walk can go.
The first is hubs. Real graphs are uneven: most entities have a few links and a handful have hundreds. "The Lisbon office" connects every team that sits there. One step through a hub and the walk has left the subject of the question entirely.
The second is the extractor. A chain is only in the graph if every fact along it was found.
Uses the setup above, 20 lines.
import random import matplotlib.pyplot as plt def grow(n, seed): """A graph that grows the way real ones do: a new entity is more likely to link to one that already has many links.""" rng = random.Random(seed) graph, ends = {0: {1}, 1: {0}}, [0, 1] for new in range(2, n): graph[new] = set() for other in {rng.choice(ends), rng.choice(ends)}: graph[new].add(other) graph[other].add(new) ends += [new, other] return graph graph = grow(5000, 0) sizes = sorted(len(v) for v in graph.values()) print( "5,000 entities | a typical entity has", sizes[len(sizes) // 2], "links | the busiest has", sizes[-1]) def reach(graph, start, hops, cap): reached, edge = {start}, [start] for hop in range(hops): edge = { other for node in edge if len(graph[node]) <= cap for other in graph[node] } - reached reached |= edge return len(reached) - 1 rng = random.Random(1) starts = [rng.randrange(5000) for i in range(200)] curves = {} for cap, name in ( (10 ** 9, "follow every link "), (20, "don't expand through hubs ")): curve = [round(sum(reach(graph, s, hops, cap) for s in starts) / len(starts)) for hops in (1, 2, 3, 4)] curves[name.strip()] = curve print(name, "| entities reached, on average, within 1, 2, 3, 4 hops:", curve) for label, curve in curves.items(): plt.plot([1, 2, 3, 4], curve, marker="o", label=label) plt.yscale("log") plt.xlabel("hops") plt.ylabel("entities reached (log scale)") plt.legend() plt.show() print() rng = random.Random(2) for hops in (1, 2, 3, 4): whole = sum( all(rng.random() < 0.9 for fact in range(hops)) for trial in range(10000)) print( "facts in the chain:", hops, "| each is found by the extractor 9 times in 10 | the whole chain is in the " "graph", round(100 * whole / 10000), "% of the time")
In this graph of 5,000 entities, a typical one has 3 links and the busiest has 301. Following every link, a walk reaches 4 entities in one hop, 61 in two, 487 in three and 1,950 in four: more than a third of the graph, from one starting point. That is no longer retrieval. Refusing to walk through hubs, here anything with more than 20 links, keeps four hops to 213.
The last four lines are plain arithmetic made visible. An extractor that finds 9 facts in 10 leaves a three-fact chain whole 73% of the time. The longer the chain a question needs, the less likely the graph holds all of it.
And there is the bill. Building the graph means a model reading every page. Keeping it right means doing that again when pages change, and removing facts that stopped being true.
So a graph earns its place when questions really do hop through things they don't name, over documents that change slowly: who owns what, what depends on what, who reports to whom. For questions that one passage answers, plain search is cheaper and often better. Most systems keep both. The approach is often called GraphRAG, after a Microsoft research project that also summarizes whole regions of the graph.
Show the answer
Reach the office, and not continue the walk through it
The office is a legitimate fact about the Growth team, so it can be reached. Walking on through it leads to every other team, and from them to everything they own, none of which bears on the question. Capping expansion at hubs keeps the walk near its subject. Deleting the hub would lose real facts, and stopping the walk would lose the rest of the neighborhood.
chance = 1.0 for fact in range(3): chance *= 0.9 print(round(chance, 3))
Show the answer
0.729
Every fact on the chain has to be there, so the chances multiply: 0.9 × 0.9 × 0.9 = 0.729. More than a quarter of three-hop questions can't be answered from this graph at all, however good the walk. This is the main reason to keep ordinary search beside it.
Worked example: eight questions, three retrievers
The honest test of a retriever is a set of questions with the pages each answer needs. Here are eight. Text search returns its top 3. The graph links, walks three hops, and returns the pages behind the facts it reached. "Both" passes if the needed pages are in either set.
Uses the setup above, 20 lines.
DOCS = {
"d1": "Incident report. The March outage lasted four hours. The root cause was a "
"failed deploy of ledger-sync.",
"d2": "Service catalog. ledger-sync is owned by the Payments Platform team. It "
"depends on invoice-api.",
"d3": "Team page. The Payments Platform team is led by Dana Reyes and sits in the "
"Lisbon office.",
"d4": "Team page. The Growth team is led by Omar Haddad. The team owns notify-hub "
"and leads the March campaign.",
"d5": "Service catalog. search-index is owned by the Core Data team. Who owns a "
"service is listed in this catalog.",
"d6": "Team page. The Core Data team is led by Li Wei and sits in the Lisbon "
"office.",
"d7": "Incident report. The June outage lasted one hour. The root cause was a full "
"disk on search-index.",
"d8": "Leadership note. Every team that owns a service must name a lead who is on "
"call during an outage.",
"d9": "Service catalog. invoice-api is owned by the Payments Platform team. It "
"depends on CloudBase.",
"d10": "Vendor note. CloudBase is the hosting vendor for invoice-api and "
"notify-hub. The contract is managed by Priya Nair.",
"d11": "Office guide. The Lisbon office hosts the Payments Platform, Core Data and "
"Growth teams.",
"d12": "Project brief. The Atlas migration moves ledger-sync to new hosting. Li "
"Wei sponsors the Atlas migration.",
}
import math
SAME = {
"led": "lead", "leads": "lead", "owned": "own", "owns": "own", "teams": "team",
"hosts": "host", "hosting": "host"}
SKIP = {
"the", "a", "is", "by", "of", "in", "on", "it", "to", "and", "that", "was", "who",
"for", "this", "behind", "what", "which", "how", "did", "does"}
def words(text):
out = set()
for w in text.lower().replace(".", " ").replace(",", " ").replace("?", " ").split():
w = SAME.get(w, w)
if w not in SKIP:
out.add(w)
return out
DOC_WORDS = {d: words(text) for d, text in DOCS.items()}
RARITY = {
w: math.log(
len(DOCS) / sum(w in ws for ws in DOC_WORDS.values())
) for ws in DOC_WORDS.values() for w in ws}
def search(question, k):
"""A stand-in for embedding search: documents that share the question's words,
rare words counting for more."""
q = words(question)
scored = sorted(
DOCS, key=lambda d: (-sum(RARITY[w] for w in q & DOC_WORDS[d]), int(d[1:])))
return scored[:k]
TRIPLES = [ # subject, relation, object, the document it came from
("March outage", "caused by", "ledger-sync", "d1"),
("ledger-sync", "owned by", "Payments Platform", "d2"),
("ledger-sync", "depends on", "invoice-api", "d2"),
("Payments Platform", "led by", "Dana Reyes", "d3"),
("Payments Platform", "located in", "Lisbon office", "d3"),
("Growth", "led by", "Omar Haddad", "d4"),
("notify-hub", "owned by", "Growth", "d4"),
("search-index", "owned by", "Core Data", "d5"),
("Core Data", "led by", "Li Wei", "d6"),
("Core Data", "located in", "Lisbon office", "d6"),
("June outage", "caused by", "search-index", "d7"),
("invoice-api", "owned by", "Payments Platform", "d9"),
("invoice-api", "depends on", "CloudBase", "d9"),
("notify-hub", "depends on", "CloudBase", "d10"),
("CloudBase", "managed by", "Priya Nair", "d10"),
("Growth", "located in", "Lisbon office", "d11"),
("Atlas migration", "moves", "ledger-sync", "d12"),
("Atlas migration", "sponsored by", "Li Wei", "d12"),
]
def build(triples):
"""entity -> the entities one step away, in either direction."""
graph = {}
for subject, relation, obj, source in triples:
graph.setdefault(subject, set()).add(obj)
graph.setdefault(obj, set()).add(subject)
return graph
def expand(graph, start, hops):
"""Every entity within this many steps of the start."""
reached, edge = {start}, [start]
for hop in range(hops):
edge = sorted({other for node in edge for other in graph[node]} - reached)
reached.update(edge)
return reached
graph = build(TRIPLES)
def graph_docs(question, hops):
linked = [entity for entity in graph if entity.lower() in question.lower()]
reached = set()
for entity in linked:
reached |= expand(graph, entity, hops)
return sorted(
{t[3] for t in TRIPLES if t[0] in reached and t[2] in reached},
key=lambda d: int(d[1:]))
questions = [ # the question, and the documents an answer needs
("How long did the March outage last?", ["d1"]),
("What must every team that owns a service name?", ["d8"]),
("Which vendor does notify-hub depend on?", ["d10"]),
(
"Who leads the team that owns the service behind the March outage?",
["d1", "d2", "d3"]),
(
"Who leads the team that owns the service behind the June outage?",
["d7", "d5", "d6"]),
("Who manages the vendor that invoice-api depends on?", ["d9", "d10"]),
("In which office is the team that owns ledger-sync?", ["d2", "d3"]),
(
"How long was the outage caused by the service the Atlas migration moves?",
["d12", "d1"]),
]
score = {"text": 0, "graph": 0, "both": 0}
for question, needed in questions:
text, walk = search(question, 3), graph_docs(question, 3)
got = {
"text": all(d in text for d in needed), "graph": all(d in walk for d in needed),
"both": all(d in text + walk for d in needed)}
for name in score:
score[name] += got[name]
print(
"text", "yes" if got["text"] else "no ", "| graph",
"yes" if got["graph"] else "no ", "(" + str(len(walk)).rjust(2), "docs) | both",
"yes" if got["both"] else "no ", "|", question)
print("questions with every needed document retrieved, of", len(questions), ":", score)Text search gets 6 of 8 and the graph gets 7, and neither is simply better. Text search fails on exactly the two questions where the chain runs through unnamed things: outage, service, team, lead. It passes several other multi-step questions, because those name something on each page they need, such as "vendor" and "invoice-api". The graph fails on the leadership-note question, where nothing in the question is an entity, so it has nowhere to start and returns 0 pages. Using both retrieves everything, 8 of 8. Look also at the page counts: the graph returned up to 9 of the 12 pages. On a collection this small, three hops reach most of it. The graph bought its recall with a great deal of extra reading, which is the hub problem from the last cell at toy size.
Write a sentence or two, then compare it with a good answer in the lesson.
Practice what you just read
- 6 code examples you can edit and run
- a practice exercise that checks your code
- a 5-question quiz
- 11 review cards for spaced repetition
Try a quiz question
What kind of question is hardest for retrieval by similarity alone?
Before this lesson
Where this lesson leads.
575 lessons in thirteen phases. 176 are free without an account, and a free account opens the rest. See every lesson.
Phase 10 · about 20 hours of study
Agent harnesses
Build the software around a model that turns it into a capable, safe agent
24 lessons 1 free
- What an agent harness is FreeThe software around a model that turns it into an agent.35 min
- The core loop, built properlyStop reasons, parallel tool calls, errors and turn budgets.40 min
- A tool registry for agentsSchemas from functions, validation, output limits and helpful errors.40 min
- File and search toolsRead, write, edit by exact replacement, glob and grep inside one folder.45 min
- Running code safelyExecution tools, step limits, output caps and real sandboxes.45 min
- Giving an agent a databaseSchema as context, a query tool with caps and readable errors, guards in layers, and execution accuracy.45 min
- Permissions and approvalsRules, modes and approval prompts that keep a person in control.40 min
- Hooks and skillsExtend a harness at lifecycle events, and load know-how on demand.40 min
- Multi-agent systemsCoordinating many agents on one large job.35 min
- Managing the context windowBudgets, clearing old tool results, compaction and a cacheable prefix.45 min
- Memory, plans and long tasksTo-do lists, progress notes, memory files and checkpoints.45 min
- SubagentsHand a focused task to a fresh context with fewer tools.40 min
- Streaming, interrupts and sessionsLive output, stopping safely, and saving and resuming transcripts.40 min
- Agents as graphsState, reducers and routing, a checkpoint after every step, and pausing for a person's approval.45 min
- Evaluation harnessesRun agents on tasks in fresh environments and grade what they leave behind.45 min
- Tracing, budgets and guardrailsSee what an agent did, cap what it spends, and stop it when it loops.40 min
- Attack your own harnessRun fourteen attacks against a harness with switches, read which layer stops each one, score attacks stopped against ordinary tasks done, and keep every finding as a release test.40 min
- Capstone: build a coding agentA harness that edits a project, runs its tests and stops when they pass.45 min
- The same agent on the real SDKsPut the parts you built beside the tool runner and the Agent SDK, choose a surface for a job, and take an MCP server's tools into your registry with prefixed names, one rule per server and a capped result.35 min
- Agents that use a computerAn agent with a screen, a mouse and a keyboard: the observe-decide-act loop, what a click lands on, why a wrong hit is worse than a miss, what happens when the screen changes under a plan, and what screenshots cost.45 min
- Sandboxes for agents that browseTreat every page as untrusted input: check hosts properly, gate actions by their effect, and layer the defenses.45 min
- Protocols between agentsHand a task to an agent someone else built: agent cards, the life of a task, the messages on the wire, safe retries, and whom to trust.45 min
- Tracing a run end to endFollow one run across an orchestrator, subagents and tool servers: trace ids, the traceparent header, the critical path, tokens by branch, and which traces to keep.45 min
- Retrieval over a graphAnswer questions that hop between documents: facts as triples, a walk out from the entities a question names, and the hubs and missed facts that limit it.45 min
Never trained an AI? Teach a small one to write, right in your browser. It takes about 20 seconds and needs no account. Teach it to write