Index Ai Academy
Phase 10 · The agent frontier

Retrieval over a graph

Answer questions that hop between documents: facts as triples, a walk out from the entities a question names, and the hubs and missed facts that limit it.

About 45 minutes · Model names checked November 2026 · A free account opens the whole lesson.
Already have an account? Sign in to open this lesson

What it covers7

Why it matters

Retrieval by similarity finds text that looks like the question. Some questions can only be answered by joining facts from pages that look nothing like the question at all: this incident was caused by that service, which belongs to that team, which is led by that person. Agents that work over a company's documents meet these constantly. Retrieval over a graph is the main technique for them. It is also expensive to build and easy to oversell, so this lesson measures both where it wins and where it doesn't.

What you'll be able to do

  • Recognize a question that needs several documents joined by things the question doesn't name
  • Store facts as triples in a graph, each pointing back to the document it came from
  • Retrieve by linking a question to entities and walking a few steps out from them
  • Explain why hubs and missed facts limit how far a walk can usefully go
  • Decide when a graph is worth building, and combine it with ordinary search
Quick check
A guess before you start. An assistant has three pages: an incident report naming the service that failed, a catalog page saying which team owns that service, and a team page naming the team's lead. Someone asks "who leads the team behind the incident?". Why might search by similarity struggle?
  1. Two of the three pages share almost nothing with the question
  2. Similarity search can only ever return one page
  3. The pages are too short to be turned into embeddings
  4. Questions about people are blocked by most retrievers
Show the answer

Two of the three pages share almost nothing with the question

The incident report matches the question, since both mention the incident. The catalog page is about a service the question never names, and the team page is about a team it never names. They are connected to the question only through each other. That chain is what a similarity score can't see.

The idea: facts as triples, a walk, and the pages behind them

Recall: Making RAG good showed how search finds pages that look like the question, and Recursion, trees and graphs taught walking a graph. A question that hops between pages that look nothing alike needs both.

Questions that hop

Here are twelve short pages from one company: incident reports, a service catalog, team pages. The search below ranks pages by the words they share with the question, with rare words counting for more. It is a stand-in for the embedding search you built before, and it fails in the same place.

Setup (runs before every cell) · 20 lines

The lesson runs this once, before its first cell, and every cell, drill and example starts with what it defines. A cell that changes a setup object changes it for the cells after it, until the page reloads.

TRIPLES = [  # subject, relation, object, the document it came from
    ("March outage", "caused by", "ledger-sync", "d1"),
    ("ledger-sync", "owned by", "Payments Platform", "d2"),
    ("ledger-sync", "depends on", "invoice-api", "d2"),
    ("Payments Platform", "led by", "Dana Reyes", "d3"),
    ("Payments Platform", "located in", "Lisbon office", "d3"),
    ("Growth", "led by", "Omar Haddad", "d4"),
    ("notify-hub", "owned by", "Growth", "d4"),
    ("search-index", "owned by", "Core Data", "d5"),
    ("Core Data", "led by", "Li Wei", "d6"),
    ("Core Data", "located in", "Lisbon office", "d6"),
    ("June outage", "caused by", "search-index", "d7"),
    ("invoice-api", "owned by", "Payments Platform", "d9"),
    ("invoice-api", "depends on", "CloudBase", "d9"),
    ("notify-hub", "depends on", "CloudBase", "d10"),
    ("CloudBase", "managed by", "Priya Nair", "d10"),
    ("Growth", "located in", "Lisbon office", "d11"),
    ("Atlas migration", "moves", "ledger-sync", "d12"),
    ("Atlas migration", "sponsored by", "Li Wei", "d12"),
]
DOCS = {
    "d1": "Incident report. The March outage lasted four hours. The root cause was a "
        "failed deploy of ledger-sync.",
    "d2": "Service catalog. ledger-sync is owned by the Payments Platform team. It "
        "depends on invoice-api.",
    "d3": "Team page. The Payments Platform team is led by Dana Reyes and sits in the "
        "Lisbon office.",
    "d4": "Team page. The Growth team is led by Omar Haddad. The team owns notify-hub "
        "and leads the March campaign.",
    "d5": "Service catalog. search-index is owned by the Core Data team. Who owns a "
        "service is listed in this catalog.",
    "d6": "Team page. The Core Data team is led by Li Wei and sits in the Lisbon "
        "office.",
    "d7": "Incident report. The June outage lasted one hour. The root cause was a full "
        "disk on search-index.",
    "d8": "Leadership note. Every team that owns a service must name a lead who is on "
        "call during an outage.",
    "d9": "Service catalog. invoice-api is owned by the Payments Platform team. It "
        "depends on CloudBase.",
    "d10": "Vendor note. CloudBase is the hosting vendor for invoice-api and "
        "notify-hub. The contract is managed by Priya Nair.",
    "d11": "Office guide. The Lisbon office hosts the Payments Platform, Core Data and "
        "Growth teams.",
    "d12": "Project brief. The Atlas migration moves ledger-sync to new hosting. Li "
        "Wei sponsors the Atlas migration.",
}
import math

SAME = {
    "led": "lead", "leads": "lead", "owned": "own", "owns": "own", "teams": "team",
    "hosts": "host", "hosting": "host"}
SKIP = {
    "the", "a", "is", "by", "of", "in", "on", "it", "to", "and", "that", "was", "who",
    "for", "this", "behind", "what", "which", "how", "did", "does"}

def words(text):
    out = set()
    for w in text.lower().replace(".", " ").replace(",", " ").replace("?", " ").split():
        w = SAME.get(w, w)
        if w not in SKIP:
            out.add(w)
    return out

DOC_WORDS = {d: words(text) for d, text in DOCS.items()}
RARITY = {
    w: math.log(
        len(DOCS) / sum(w in ws for ws in DOC_WORDS.values())
    ) for ws in DOC_WORDS.values() for w in ws}

def search(question, k):
    """A stand-in for embedding search: documents that share the question's words,
    rare words counting for more."""
    q = words(question)
    scored = sorted(
        DOCS, key=lambda d: (-sum(RARITY[w] for w in q & DOC_WORDS[d]), int(d[1:])))
    return scored[:k]

easy = "How long did the March outage last?"
hard = "Who leads the team that owns the service behind the March outage?"
needed = {easy: ["d1"], hard: ["d1", "d2", "d3"]}

for question in (easy, hard):
    print(question)
    for k in (3, 5):
        found = search(question, k)
        print(
            "   top", k, found, "| needed", needed[question], "| found",
            sum(d in found for d in needed[question]), "of", len(needed[question]))
print()
print(
    "where the three needed documents rank for the hard question:",
    {d: search(hard, 12).index(d) + 1 for d in needed[hard]})
It runs right here in your browser, with nothing to install.

The easy question needs one page, and it comes first. The hard question needs three, and the top 3 has only one of them. Asking for 5 finds a second. The third, the team page that holds the actual answer, ranks 7th.

Look at what did rank at the top: d8, a leadership note about teams, services, leads and outages. It shares more words with the question than any page that helps. It is about the same things and contains none of the facts.

A better similarity model would not cure this. To know that the page about ledger-sync matters, you must already have read the incident report, since the question never says ledger-sync. The question is a multi-hop question. It has to be answered in hops: from the outage to the service, from the service to the team, from the team to its lead. Each hop goes through something the question didn't name.

There are two ways to make the hops. An agent can search, read, and search again with what it learned, which costs a model call per hop. Or the links can be worked out ahead of time and stored. That store is a graph.

Quick check
From the lesson on retrieval-augmented generation: what does the retrieval step do?
  1. Trains the model on the company's documents overnight
  2. Finds the passages most relevant to the question and puts them in the prompt
  3. Checks the model's answer against the web after it is written
  4. Shortens the question so that it uses fewer tokens
Show the answer

Finds the passages most relevant to the question and puts them in the prompt

RAG answers from your own documents: retrieve the passages that bear on the question, put them in the prompt, and have the model answer from them. Everything in this lesson is about the first step. A graph is another way of choosing what goes into the prompt.

Drill

Find the entities a question names

Walking a graph starts from the entities in the question. Write link(question, names). Return the names that appear in the question, ignoring case, as whole words: the character just before and just after the match must not be a letter or a digit. When one matched name lies inside a longer matched name at the same place, as "Payments" does inside "Payments Platform", keep only the longer. Return the names as given in names, in the order they appear in the question.

Uses the setup above, 20 lines.

def link(question, names):
    return []


print(
    link(
        "Who leads Payments Platform, and who owns ledger-sync?",
        ["ledger-sync", "Payments", "Payments Platform", "Growth"]))
In the lesson, your code is checked as you write it.
Try it now

Step the walk. Columns are hops from the starting entity; lit entities have been reached.

In your own words
Why can't a better embedding model fix the question 'who leads the team that owns the service behind the March outage'?

Write a sentence or two, then compare it with a good answer in the lesson.

Facts as a graph

The links between pages run through the things they mention: a service, a team, a person. Call those entities. A fact connecting two of them can be written as a triple: a subject, a relation, an object. "ledger-sync is owned by the Payments Platform team" becomes (ledger-sync, owned by, Payments Platform).

Collect the triples from every page and you have a knowledge graph: entities as the points, relations as the lines between them. Each triple also remembers which page it came from, which will matter when it is time to give the model something to read.

Uses the setup above, 20 lines.

TRIPLES = [  # subject, relation, object, the document it came from
    ("March outage", "caused by", "ledger-sync", "d1"),
    ("ledger-sync", "owned by", "Payments Platform", "d2"),
    ("ledger-sync", "depends on", "invoice-api", "d2"),
    ("Payments Platform", "led by", "Dana Reyes", "d3"),
    ("Payments Platform", "located in", "Lisbon office", "d3"),
    ("Growth", "led by", "Omar Haddad", "d4"),
    ("notify-hub", "owned by", "Growth", "d4"),
    ("search-index", "owned by", "Core Data", "d5"),
    ("Core Data", "led by", "Li Wei", "d6"),
    ("Core Data", "located in", "Lisbon office", "d6"),
    ("June outage", "caused by", "search-index", "d7"),
    ("invoice-api", "owned by", "Payments Platform", "d9"),
    ("invoice-api", "depends on", "CloudBase", "d9"),
    ("notify-hub", "depends on", "CloudBase", "d10"),
    ("CloudBase", "managed by", "Priya Nair", "d10"),
    ("Growth", "located in", "Lisbon office", "d11"),
    ("Atlas migration", "moves", "ledger-sync", "d12"),
    ("Atlas migration", "sponsored by", "Li Wei", "d12"),
]

def build(triples):
    """entity -> the entities one step away, in either direction."""
    graph = {}
    for subject, relation, obj, source in triples:
        graph.setdefault(subject, set()).add(obj)
        graph.setdefault(obj, set()).add(subject)
    return graph

def expand(graph, start, hops):
    """Every entity within this many steps of the start."""
    reached, edge = {start}, [start]
    for hop in range(hops):
        edge = sorted({other for node in edge for other in graph[node]} - reached)
        reached.update(edge)
    return reached

graph = build(TRIPLES)
print(len(TRIPLES), "facts about", len(graph), "entities")
for entity in ("ledger-sync", "Lisbon office", "Dana Reyes"):
    print(entity.ljust(14), "->", sorted(graph[entity]))

print()
print("everything the graph holds about ledger-sync:")
for subject, relation, obj, source in TRIPLES:
    if "ledger-sync" in (subject, obj):
        print("  ", subject, "|", relation, "|", obj, " (from " + source + ")")

Eighteen facts connect sixteen entities. ledger-sync touches four of them, drawn from three different pages: the incident report, the catalog and a project brief. No single page says all of that. The graph has done the joining ahead of time.

Where do triples come from? In a real system a model reads every page and writes them out. Here they were written by hand, which hides the two hard parts.

  • Extraction misses things. Some facts aren't found, and some that are found are wrong.
  • Names don't match. One page says "Payments Platform", another "the payments team", a third "PP". Deciding that these are one entity is called entity resolution, and a graph with three separate points for one team has lost exactly the links it was built for.

The page with "lasted four hours" shows a third limit. That duration isn't in any triple. A graph holds the facts someone chose to extract, and prose always holds more.

Quick check
Three triples go into a graph that links each pair in both directions. What does this print: each entity's neighbors, counted?
triples = [
    ("checkout", "owned by", "Web team"), ("checkout", "depends on", "cart-api"),
    ("cart-api", "owned by", "Web team")]
graph = {}
for subject, relation, obj in triples:
    graph.setdefault(subject, set()).add(obj)
    graph.setdefault(obj, set()).add(subject)
print({entity: len(graph[entity]) for entity in sorted(graph)})
  1. {'Web team': 2, 'cart-api': 1, 'checkout': 2}
  2. {'Web team': 0, 'cart-api': 1, 'checkout': 2}
  3. {'Web team': 3, 'cart-api': 3, 'checkout': 3}
  4. {'Web team': 2, 'cart-api': 2, 'checkout': 2}
Show the answer

{'Web team': 2, 'cart-api': 2, 'checkout': 2}

Each triple links its two ends in both directions. The Web team owns two things, so it has two neighbors, even though it is only ever the object. checkout links to the team and to cart-api. cart-api links to checkout and to the team. Storing both directions is what lets a walk go from a team to its services as easily as from a service to its team.

Retrieval by walking

With a graph, retrieval has three steps.

  1. Link. Find the entities the question names.
  2. Walk. Go out from them a fixed number of steps, collecting every entity on the way.
  3. Collect. Take the facts between the entities reached, and the pages those facts came from.

What goes into the prompt is the facts, the pages behind them, or both.

Uses the setup above, 20 lines.

def build(triples):
    """entity -> the entities one step away, in either direction."""
    graph = {}
    for subject, relation, obj, source in triples:
        graph.setdefault(subject, set()).add(obj)
        graph.setdefault(obj, set()).add(subject)
    return graph

def expand(graph, start, hops):
    """Every entity within this many steps of the start."""
    reached, edge = {start}, [start]
    for hop in range(hops):
        edge = sorted({other for node in edge for other in graph[node]} - reached)
        reached.update(edge)
    return reached

graph = build(TRIPLES)
question = "Who leads the team that owns the service behind the March outage?"
linked = [entity for entity in graph if entity.lower() in question.lower()]
print("entities named in the question:", linked)

start = linked[0]
for hops in (1, 2, 3):
    reached = expand(graph, start, hops)
    facts = [t for t in TRIPLES if t[0] in reached and t[2] in reached]
    print(
        "hops:", hops, "| entities:", len(reached), "| facts:", len(facts),
        "| documents:", sorted({t[3] for t in facts}, key=lambda d: int(d[1:])))

def paths(node, steps, seen):
    """Every way of walking this many steps from a node, as readable text."""
    if steps == 0:
        return [node]
    out = []
    for subject, relation, obj, source in TRIPLES:
        if subject == node and obj not in seen:
            out += [
                node + " -" + relation + "-> " + rest for rest in paths(
                    obj, steps - 1, seen | {obj})]
        if obj == node and subject not in seen:
            out += [
                node + " <-" + relation + "- " + rest for rest in paths(
                    subject, steps - 1, seen | {subject})]
    return out

print()
people = ("Dana Reyes", "Li Wei", "Omar Haddad", "Priya Nair")
for path in paths(start, 3, {start}):
    if path.endswith(people):
        print(path)

The question names one entity, the March outage. One hop reaches the service. Two hops reach the team, and a couple of things beside it. At three hops the walk has 9 entities and 9 facts drawn from five pages, and for the first time d1, d2 and d3 are all there: the three pages the search could not put together.

The last two lines show something a pile of pages can't. Three steps from the outage there are two people. One is reached through "caused by", "owned by", "led by". The other is reached because a migration project moves the same service and she sponsors it. The relations say which chain answers the question. Hand the model the facts with their relations, and it can tell the team's lead from someone who merely appears nearby.

That second path is also a warning. The walk brought back d9 and d12, which have nothing to do with the question. A walk gathers everything within reach, and the further it goes the more of that there is.

Quick check
A walk starts at "outage" in this graph. What does this print: the entities first reached at each step?
graph = {
    "outage": {"api"}, "api": {"outage", "team", "db"}, "team": {"api", "lead"},
    "db": {"api"}, "lead": {"team"}}
reached, edge = {"outage"}, ["outage"]
for hop in range(3):
    edge = sorted({other for node in edge for other in graph[node]} - reached)
    reached.update(edge)
    print(hop + 1, edge)
  1. 1 ['api', 'db', 'team'] 2 ['lead'] 3 []
  2. 1 ['api'] 2 ['db', 'team'] 3 ['lead']
  3. 1 ['api'] 2 ['team'] 3 ['lead']
  4. 1 ['api'] 2 ['db', 'outage', 'team'] 3 ['api', 'lead']
Show the answer

1 ['api'] 2 ['db', 'team'] 3 ['lead']

Step 1 reaches the outage's only neighbor, api. Step 2 goes out from api to db and team, and not back to outage, which was already reached. Step 3 goes from team to lead, and db leads nowhere new. This is a breadth-first search: everything one step away, then everything two steps away.

Drill

How far is everything?

Write distances(graph, start, hops). graph maps each entity to the set of its neighbors. Return a dictionary from each entity within hops steps of start to the smallest number of steps needed to reach it. The start itself is at 0. If start isn't in the graph, return an empty dictionary.

Uses the setup above, 20 lines.

def distances(graph, start, hops):
    return {}


graph = {
    "outage": {"api"}, "api": {"outage", "team", "db"}, "team": {"api", "lead"},
    "db": {"api"}, "lead": {"team"}}
print(distances(graph, "outage", 2))
In the lesson, your code is checked as you write it.
Worked example

The agent's alternative: search, read, search again

Without a graph, an agent can follow the same chain by searching, reading what it finds and searching again, one model call per hop. The stand-in below does the thinking with a rule.

Uses the setup above, 20 lines.

PAGES = {
    "d1": "March outage caused by ledger-sync",
    "d2": "ledger-sync owned by Payments Platform",
    "d3": "Payments Platform led by Dana Reyes",
}

def search(query):
    words = set(query.lower().split())
    return max(PAGES, key=lambda d: len(words & set(PAGES[d].lower().split())))

def next_query(page):
    # a stand-in for one model call: read the page, name the next thing to look up
    entity = page.split(" by ")[-1]
    return {
        "ledger-sync": "ledger-sync owned", "Payments Platform": "Payments Platform led"
    }.get(entity, "")

query, calls = "March outage caused", 0
while query:
    page = PAGES[search(query)]
    calls += 1
    print(calls, "|", query, "->", page)
    query = next_query(page)
print("model calls to follow the chain:", calls)
In the lesson you can change this and run it.
In your own words
Two pages call the same team 'Payments Platform' and 'the payments team'. What happens to a walk?

Write a sentence or two, then compare it with a good answer in the lesson.

In a production harness: costs, networkx and when a graph is worth it

What it costs, and when it is worth it

Two things limit how far a walk can go.

The first is hubs. Real graphs are uneven: most entities have a few links and a handful have hundreds. "The Lisbon office" connects every team that sits there. One step through a hub and the walk has left the subject of the question entirely.

The second is the extractor. A chain is only in the graph if every fact along it was found.

Uses the setup above, 20 lines.

import random
import matplotlib.pyplot as plt

def grow(n, seed):
    """A graph that grows the way real ones do: a new entity is more likely to link
    to one that already has many links."""
    rng = random.Random(seed)
    graph, ends = {0: {1}, 1: {0}}, [0, 1]
    for new in range(2, n):
        graph[new] = set()
        for other in {rng.choice(ends), rng.choice(ends)}:
            graph[new].add(other)
            graph[other].add(new)
            ends += [new, other]
    return graph

graph = grow(5000, 0)
sizes = sorted(len(v) for v in graph.values())
print(
    "5,000 entities | a typical entity has", sizes[len(sizes) // 2],
    "links | the busiest has", sizes[-1])

def reach(graph, start, hops, cap):
    reached, edge = {start}, [start]
    for hop in range(hops):
        edge = {
            other for node in edge if len(graph[node]) <= cap for other in graph[node]
        } - reached
        reached |= edge
    return len(reached) - 1

rng = random.Random(1)
starts = [rng.randrange(5000) for i in range(200)]
curves = {}
for cap, name in (
    (10 ** 9, "follow every link         "), (20, "don't expand through hubs ")):
    curve = [round(sum(reach(graph, s, hops, cap) for s in starts) / len(starts))
             for hops in (1, 2, 3, 4)]
    curves[name.strip()] = curve
    print(name, "| entities reached, on average, within 1, 2, 3, 4 hops:", curve)

for label, curve in curves.items():
    plt.plot([1, 2, 3, 4], curve, marker="o", label=label)
plt.yscale("log")
plt.xlabel("hops")
plt.ylabel("entities reached (log scale)")
plt.legend()
plt.show()

print()
rng = random.Random(2)
for hops in (1, 2, 3, 4):
    whole = sum(
        all(rng.random() < 0.9 for fact in range(hops)) for trial in range(10000))
    print(
        "facts in the chain:", hops,
        "| each is found by the extractor 9 times in 10 | the whole chain is in the "
            "graph",
        round(100 * whole / 10000), "% of the time")

In this graph of 5,000 entities, a typical one has 3 links and the busiest has 301. Following every link, a walk reaches 4 entities in one hop, 61 in two, 487 in three and 1,950 in four: more than a third of the graph, from one starting point. That is no longer retrieval. Refusing to walk through hubs, here anything with more than 20 links, keeps four hops to 213.

The last four lines are plain arithmetic made visible. An extractor that finds 9 facts in 10 leaves a three-fact chain whole 73% of the time. The longer the chain a question needs, the less likely the graph holds all of it.

And there is the bill. Building the graph means a model reading every page. Keeping it right means doing that again when pages change, and removing facts that stopped being true.

So a graph earns its place when questions really do hop through things they don't name, over documents that change slowly: who owns what, what depends on what, who reports to whom. For questions that one passage answers, plain search is cheaper and often better. Most systems keep both. The approach is often called GraphRAG, after a Microsoft research project that also summarizes whole regions of the graph.

Quick check
A walk for the question "which services does the Growth team own?" passes through the entity "Lisbon office", which is linked to every team in the building. What should the retriever do?
  1. Delete the office from the graph for good
  2. Stop the whole walk as soon as a hub is met
  3. Follow all of its links, to be sure nothing is missed
  4. Reach the office, and not continue the walk through it
Show the answer

Reach the office, and not continue the walk through it

The office is a legitimate fact about the Growth team, so it can be reached. Walking on through it leads to every other team, and from them to everything they own, none of which bears on the question. Capping expansion at hubs keeps the walk near its subject. Deleting the hub would lose real facts, and stopping the walk would lose the rest of the neighborhood.

Quick check
An extractor finds each fact with probability 0.9, and a question needs a chain of three facts. What does this print: the chance that the whole chain is in the graph?
chance = 1.0
for fact in range(3):
    chance *= 0.9
print(round(chance, 3))
  1. 0.7
  2. 0.27
  3. 0.729
  4. 0.9
Show the answer

0.729

Every fact on the chain has to be there, so the chances multiply: 0.9 × 0.9 × 0.9 = 0.729. More than a quarter of three-hop questions can't be answered from this graph at all, however good the walk. This is the main reason to keep ordinary search beside it.

Worked example

Worked example: eight questions, three retrievers

The honest test of a retriever is a set of questions with the pages each answer needs. Here are eight. Text search returns its top 3. The graph links, walks three hops, and returns the pages behind the facts it reached. "Both" passes if the needed pages are in either set.

Uses the setup above, 20 lines.

DOCS = {
    "d1": "Incident report. The March outage lasted four hours. The root cause was a "
        "failed deploy of ledger-sync.",
    "d2": "Service catalog. ledger-sync is owned by the Payments Platform team. It "
        "depends on invoice-api.",
    "d3": "Team page. The Payments Platform team is led by Dana Reyes and sits in the "
        "Lisbon office.",
    "d4": "Team page. The Growth team is led by Omar Haddad. The team owns notify-hub "
        "and leads the March campaign.",
    "d5": "Service catalog. search-index is owned by the Core Data team. Who owns a "
        "service is listed in this catalog.",
    "d6": "Team page. The Core Data team is led by Li Wei and sits in the Lisbon "
        "office.",
    "d7": "Incident report. The June outage lasted one hour. The root cause was a full "
        "disk on search-index.",
    "d8": "Leadership note. Every team that owns a service must name a lead who is on "
        "call during an outage.",
    "d9": "Service catalog. invoice-api is owned by the Payments Platform team. It "
        "depends on CloudBase.",
    "d10": "Vendor note. CloudBase is the hosting vendor for invoice-api and "
        "notify-hub. The contract is managed by Priya Nair.",
    "d11": "Office guide. The Lisbon office hosts the Payments Platform, Core Data and "
        "Growth teams.",
    "d12": "Project brief. The Atlas migration moves ledger-sync to new hosting. Li "
        "Wei sponsors the Atlas migration.",
}
import math

SAME = {
    "led": "lead", "leads": "lead", "owned": "own", "owns": "own", "teams": "team",
    "hosts": "host", "hosting": "host"}
SKIP = {
    "the", "a", "is", "by", "of", "in", "on", "it", "to", "and", "that", "was", "who",
    "for", "this", "behind", "what", "which", "how", "did", "does"}

def words(text):
    out = set()
    for w in text.lower().replace(".", " ").replace(",", " ").replace("?", " ").split():
        w = SAME.get(w, w)
        if w not in SKIP:
            out.add(w)
    return out

DOC_WORDS = {d: words(text) for d, text in DOCS.items()}
RARITY = {
    w: math.log(
        len(DOCS) / sum(w in ws for ws in DOC_WORDS.values())
    ) for ws in DOC_WORDS.values() for w in ws}

def search(question, k):
    """A stand-in for embedding search: documents that share the question's words,
    rare words counting for more."""
    q = words(question)
    scored = sorted(
        DOCS, key=lambda d: (-sum(RARITY[w] for w in q & DOC_WORDS[d]), int(d[1:])))
    return scored[:k]
TRIPLES = [  # subject, relation, object, the document it came from
    ("March outage", "caused by", "ledger-sync", "d1"),
    ("ledger-sync", "owned by", "Payments Platform", "d2"),
    ("ledger-sync", "depends on", "invoice-api", "d2"),
    ("Payments Platform", "led by", "Dana Reyes", "d3"),
    ("Payments Platform", "located in", "Lisbon office", "d3"),
    ("Growth", "led by", "Omar Haddad", "d4"),
    ("notify-hub", "owned by", "Growth", "d4"),
    ("search-index", "owned by", "Core Data", "d5"),
    ("Core Data", "led by", "Li Wei", "d6"),
    ("Core Data", "located in", "Lisbon office", "d6"),
    ("June outage", "caused by", "search-index", "d7"),
    ("invoice-api", "owned by", "Payments Platform", "d9"),
    ("invoice-api", "depends on", "CloudBase", "d9"),
    ("notify-hub", "depends on", "CloudBase", "d10"),
    ("CloudBase", "managed by", "Priya Nair", "d10"),
    ("Growth", "located in", "Lisbon office", "d11"),
    ("Atlas migration", "moves", "ledger-sync", "d12"),
    ("Atlas migration", "sponsored by", "Li Wei", "d12"),
]

def build(triples):
    """entity -> the entities one step away, in either direction."""
    graph = {}
    for subject, relation, obj, source in triples:
        graph.setdefault(subject, set()).add(obj)
        graph.setdefault(obj, set()).add(subject)
    return graph

def expand(graph, start, hops):
    """Every entity within this many steps of the start."""
    reached, edge = {start}, [start]
    for hop in range(hops):
        edge = sorted({other for node in edge for other in graph[node]} - reached)
        reached.update(edge)
    return reached

graph = build(TRIPLES)

def graph_docs(question, hops):
    linked = [entity for entity in graph if entity.lower() in question.lower()]
    reached = set()
    for entity in linked:
        reached |= expand(graph, entity, hops)
    return sorted(
        {t[3] for t in TRIPLES if t[0] in reached and t[2] in reached},
        key=lambda d: int(d[1:]))

questions = [  # the question, and the documents an answer needs
    ("How long did the March outage last?", ["d1"]),
    ("What must every team that owns a service name?", ["d8"]),
    ("Which vendor does notify-hub depend on?", ["d10"]),
    (
        "Who leads the team that owns the service behind the March outage?",
        ["d1", "d2", "d3"]),
    (
        "Who leads the team that owns the service behind the June outage?",
        ["d7", "d5", "d6"]),
    ("Who manages the vendor that invoice-api depends on?", ["d9", "d10"]),
    ("In which office is the team that owns ledger-sync?", ["d2", "d3"]),
    (
        "How long was the outage caused by the service the Atlas migration moves?",
        ["d12", "d1"]),
]
score = {"text": 0, "graph": 0, "both": 0}
for question, needed in questions:
    text, walk = search(question, 3), graph_docs(question, 3)
    got = {
        "text": all(d in text for d in needed), "graph": all(d in walk for d in needed),
        "both": all(d in text + walk for d in needed)}
    for name in score:
        score[name] += got[name]
    print(
        "text", "yes" if got["text"] else "no ", "| graph",
        "yes" if got["graph"] else "no ", "(" + str(len(walk)).rjust(2), "docs) | both",
        "yes" if got["both"] else "no ", "|", question)
print("questions with every needed document retrieved, of", len(questions), ":", score)
In the lesson you can change this and run it.

Text search gets 6 of 8 and the graph gets 7, and neither is simply better. Text search fails on exactly the two questions where the chain runs through unnamed things: outage, service, team, lead. It passes several other multi-step questions, because those name something on each page they need, such as "vendor" and "invoice-api". The graph fails on the leadership-note question, where nothing in the question is an entity, so it has nowhere to start and returns 0 pages. Using both retrieves everything, 8 of 8. Look also at the page counts: the graph returned up to 9 of the 12 pages. On a collection this small, three hops reach most of it. The graph bought its recall with a great deal of extra reading, which is the hub problem from the last cell at toy size.

In your own words
A team asks whether to add a knowledge graph to their document assistant. What do you ask first?

Write a sentence or two, then compare it with a good answer in the lesson.

Practice what you just read

  • 6 code examples you can edit and run
  • a practice exercise that checks your code
  • a 5-question quiz
  • 11 review cards for spaced repetition

Try a quiz question

What kind of question is hardest for retrieval by similarity alone?

  1. One that reaches its answer through unnamed things
  2. One that uses rare words found in only one document
  3. One that is short and asks for a single plain fact
  4. One that names the very document that it wants

Answer it at the end of the lesson to see if you're right, and why.

Before this lesson

Keep learning

Where this lesson leads.

575 lessons in thirteen phases. 176 are free without an account, and a free account opens the rest. See every lesson.

Phase 10 · about 20 hours of study

Agent harnesses

Build the software around a model that turns it into a capable, safe agent 24 lessons 1 free
  1. What an agent harness is FreeThe software around a model that turns it into an agent.35 min
  2. The core loop, built properlyStop reasons, parallel tool calls, errors and turn budgets.40 min
  3. A tool registry for agentsSchemas from functions, validation, output limits and helpful errors.40 min
  4. File and search toolsRead, write, edit by exact replacement, glob and grep inside one folder.45 min
  5. Running code safelyExecution tools, step limits, output caps and real sandboxes.45 min
  6. Giving an agent a databaseSchema as context, a query tool with caps and readable errors, guards in layers, and execution accuracy.45 min
  7. Permissions and approvalsRules, modes and approval prompts that keep a person in control.40 min
  8. Hooks and skillsExtend a harness at lifecycle events, and load know-how on demand.40 min
  9. Multi-agent systemsCoordinating many agents on one large job.35 min
  10. Managing the context windowBudgets, clearing old tool results, compaction and a cacheable prefix.45 min
  11. Memory, plans and long tasksTo-do lists, progress notes, memory files and checkpoints.45 min
  12. SubagentsHand a focused task to a fresh context with fewer tools.40 min
  13. Streaming, interrupts and sessionsLive output, stopping safely, and saving and resuming transcripts.40 min
  14. Agents as graphsState, reducers and routing, a checkpoint after every step, and pausing for a person's approval.45 min
  15. Evaluation harnessesRun agents on tasks in fresh environments and grade what they leave behind.45 min
  16. Tracing, budgets and guardrailsSee what an agent did, cap what it spends, and stop it when it loops.40 min
  17. Attack your own harnessRun fourteen attacks against a harness with switches, read which layer stops each one, score attacks stopped against ordinary tasks done, and keep every finding as a release test.40 min
  18. Capstone: build a coding agentA harness that edits a project, runs its tests and stops when they pass.45 min
  19. The same agent on the real SDKsPut the parts you built beside the tool runner and the Agent SDK, choose a surface for a job, and take an MCP server's tools into your registry with prefixed names, one rule per server and a capped result.35 min
  20. Agents that use a computerAn agent with a screen, a mouse and a keyboard: the observe-decide-act loop, what a click lands on, why a wrong hit is worse than a miss, what happens when the screen changes under a plan, and what screenshots cost.45 min
  21. Sandboxes for agents that browseTreat every page as untrusted input: check hosts properly, gate actions by their effect, and layer the defenses.45 min
  22. Protocols between agentsHand a task to an agent someone else built: agent cards, the life of a task, the messages on the wire, safe retries, and whom to trust.45 min
  23. Tracing a run end to endFollow one run across an orchestrator, subagents and tool servers: trace ids, the traceparent header, the critical path, tokens by branch, and which traces to keep.45 min
  24. Retrieval over a graphAnswer questions that hop between documents: facts as triples, a walk out from the entities a question names, and the hubs and missed facts that limit it.45 min

Never trained an AI? Teach a small one to write, right in your browser. It takes about 20 seconds and needs no account. Teach it to write