Tracing a run end to end
Follow one run across an orchestrator, subagents and tool servers: trace ids, the traceparent header, the critical path, tokens by branch, and which traces to keep.
What it covers7
Why it matters
You have traced an agent inside one program. Real agent systems are several programs: an orchestrator, subagents on other machines, tool servers, sometimes another company's agent. When a run is slow or expensive, each program's own log shows its own small part and none shows the run. Tracing end to end is how the parts are joined into one picture, and reading that picture well, which spans mattered and which didn't, is one of the most useful skills in running agents.
What you'll be able to do
- Rebuild one run's tree from spans recorded by several separate programs
- Carry a trace across a network hop, and recognize what a dropped hop looks like
- Find the critical path: the spans a run's finishing time was waiting on
- Attribute tokens to the part of the run that spent them
- Choose which traces to keep, and decide what a span should and shouldn't record
Show the answer
One id that every program attaches to its records of that run
Each program can only record what it did. What joins the records is a shared id for the run, passed along with every request, so that a collector can gather everything with that id from all four. That id is called the trace id, and passing it along is the whole trick.
The idea: an id crosses every hop, and parents rebuild the tree
Recall: Tracing, budgets and guardrails recorded spans inside one program. Here the run crosses programs, so each hop must carry the trace id and its own span id as the next hop's parent.
One run, many programs
In the lesson on tracing you recorded spans, timed steps that nest, inside one program. Across programs the idea is the same, and three ids hold it together.
- The trace id names the run. Every span of the run carries it, whichever program made it.
- The span id names one span.
- The parent id is the span id of whatever started this span.
Each program sends its spans to a collector when it gets round to it. They arrive in no particular order, as a flat pile. The parent ids are what turn the pile back into a tree.
Setup (runs before every cell) · 30 lines
The lesson runs this once, before its first cell, and every cell, drill and example starts with what it defines. A cell that changes a setup object changes it for the cells after it, until the page reloads.
TRACE = "4bf92f3577b34da6a3ce929d0e0e4736" # span id, parent, service, name, start ms, end ms, input tokens, output tokens ROWS = [ ("a1", None, "orchestrator", "invoke_agent research", 0, 9400, 0, 0), ( "a2", "a1", "orchestrator", "chat", 20, 1250, 2100, 180), ("b1", "a1", "worker-a", "invoke_agent find-sources", 1300, 5200, 0, 0), ( "b2", "b1", "worker-a", "chat", 1320, 2100, 1800, 90), ("b3", "b1", "worker-a", "execute_tool web_search", 2110, 3900, 0, 0), ("s1", "b3", "search-server", "tools/call web_search", 2130, 3880, 0, 0), ( "b4", "b1", "worker-a", "chat", 3910, 5180, 6400, 420), ("c1", "a1", "worker-b", "invoke_agent check-figures", 1300, 7900, 0, 0), ( "c2", "c1", "worker-b", "chat", 1320, 2300, 1900, 110), ("c3", "c1", "worker-b", "execute_tool read_report", 2310, 2600, 0, 0), ( "c4", "c1", "worker-b", "chat", 2610, 4700, 24500, 300), ("c5", "c1", "worker-b", "execute_tool ask_agent", 4710, 7300, 0, 0), ("f1", "c5", "figures-agent", "invoke_agent figures-checker", 4760, 7250, 0, 0), ( "f2", "f1", "figures-agent", "chat", 4780, 7200, 9800, 650), ( "c6", "c1", "worker-b", "chat", 7310, 7880, 26100, 240), ( "a3", "a1", "orchestrator", "chat", 7950, 9380, 5200, 900), ] KEYS = ("id", "parent", "service", "name", "start", "end", "tokens_in", "tokens_out") spans = [dict(zip(KEYS, row), trace=TRACE) for row in ROWS]
TRACE = "4bf92f3577b34da6a3ce929d0e0e4736" # span id, parent, service, name, start ms, end ms, input tokens, output tokens ROWS = [ ("a1", None, "orchestrator", "invoke_agent research", 0, 9400, 0, 0), ( "a2", "a1", "orchestrator", "chat", 20, 1250, 2100, 180), ("b1", "a1", "worker-a", "invoke_agent find-sources", 1300, 5200, 0, 0), ( "b2", "b1", "worker-a", "chat", 1320, 2100, 1800, 90), ("b3", "b1", "worker-a", "execute_tool web_search", 2110, 3900, 0, 0), ("s1", "b3", "search-server", "tools/call web_search", 2130, 3880, 0, 0), ( "b4", "b1", "worker-a", "chat", 3910, 5180, 6400, 420), ("c1", "a1", "worker-b", "invoke_agent check-figures", 1300, 7900, 0, 0), ( "c2", "c1", "worker-b", "chat", 1320, 2300, 1900, 110), ("c3", "c1", "worker-b", "execute_tool read_report", 2310, 2600, 0, 0), ( "c4", "c1", "worker-b", "chat", 2610, 4700, 24500, 300), ("c5", "c1", "worker-b", "execute_tool ask_agent", 4710, 7300, 0, 0), ("f1", "c5", "figures-agent", "invoke_agent figures-checker", 4760, 7250, 0, 0), ( "f2", "f1", "figures-agent", "chat", 4780, 7200, 9800, 650), ( "c6", "c1", "worker-b", "chat", 7310, 7880, 26100, 240), ( "a3", "a1", "orchestrator", "chat", 7950, 9380, 5200, 900), ] KEYS = ("id", "parent", "service", "name", "start", "end", "tokens_in", "tokens_out") spans = [dict(zip(KEYS, row), trace=TRACE) for row in ROWS] def children(spans): """parent id -> its child spans, in the order they started.""" kids = {} for s in sorted(spans, key=lambda s: s["start"]): kids.setdefault(s["parent"], []).append(s) return kids import random arrived = spans[:] # each service sends its spans when it gets round to it random.Random(3).shuffle(arrived) print( len(arrived), "spans from", len({s["service"] for s in arrived}), "services. The first five to arrive:") for s in arrived[:5]: print( " ", s["id"], "parent", str(s["parent"]).ljust(4), s["service"].ljust(13), s["name"]) def show(span, kids, depth=0): tokens = span["tokens_in"] + span["tokens_out"] print( (" " * depth + span["name"]).ljust(36), span["service"].ljust(13), str(span["end"] - span["start"]).rjust(5), "ms", (str(tokens) + " tokens") if tokens else "") for child in kids.get(span["id"], []): show(child, kids, depth + 1) print() kids = children(arrived) for root in kids[None]: show(root, kids)
The first five to arrive are from three programs and mean nothing on their own. Grouped by parent, the 16 spans from 5 services become a readable story: the orchestrator planned, started two workers, and wrote up. One worker searched the web through a tool server. The other read a report and asked an outside agent to check its figures.
Notice the span that crosses a boundary. execute_tool web_search was recorded by worker-a, and its child tools/call web_search by the search server. Two programs, one parent link. That link is only there because worker-a told the server its span id. The next part is about how.
The span names follow OpenTelemetry's conventions for generative AI: invoke_agent, chat, execute_tool. Those conventions are still marked as in development, so check the current names before you depend on them. Real span ids are 16 hex characters. The short ones here are for reading.
Show the answer
One timed step of a run, which can contain other steps
A span is one timed step: a model call, a tool call, a whole agent turn. Spans nest, so a run is a tree of them, and the tree as a whole is the trace. End-to-end tracing keeps that definition and lets the tree's branches live in different programs.
spans = [("a1", None), ("b1", "a1"), ("b2", "b1"), ("c1", "a1"), ("x9", "q7")] ids = {s for s, p in spans} print( [s for s, p in spans if p is None], [s for s, p in spans if p is not None and p not in ids])
Show the answer
['a1'] ['x9']
A root is a span with no parent: a1. An orphan has a parent id that matches nothing the collector holds: x9 points at q7, which isn't there. b2's parent is b1, which is present. Orphans mean a span was lost on the way, or hasn't arrived yet.
Step through the spans. Switch to the broken view: the header is dropped at one hop, and everything below it floats free.
Write a sentence or two, then compare it with a good answer in the lesson.
In code: carry the id, rebuild the tree, find the critical path
Carrying the id across
When one program calls another, the receiver needs two facts: which trace this is, and which span is calling. The standard way to send them over HTTP is one header, traceparent, defined by the W3C. It has four parts joined by dashes.
- A version,
00. - The trace id: 32 hex characters.
- The caller's span id: 16 hex characters. It becomes the parent id of whatever the receiver records.
- Flags: 2 hex characters. The last bit says whether this trace is being sampled, that is, kept.
Sending it is called context propagation, and every hop has to do it: orchestrator to subagent, subagent to tool server, agent to outside agent. The cell builds a header, and then shows the run from before with one hop that forgot.
Uses the setup above, 30 lines.
TRACE = "4bf92f3577b34da6a3ce929d0e0e4736" # span id, parent, service, name, start ms, end ms, input tokens, output tokens ROWS = [ ("a1", None, "orchestrator", "invoke_agent research", 0, 9400, 0, 0), ( "a2", "a1", "orchestrator", "chat", 20, 1250, 2100, 180), ("b1", "a1", "worker-a", "invoke_agent find-sources", 1300, 5200, 0, 0), ( "b2", "b1", "worker-a", "chat", 1320, 2100, 1800, 90), ("b3", "b1", "worker-a", "execute_tool web_search", 2110, 3900, 0, 0), ("s1", "b3", "search-server", "tools/call web_search", 2130, 3880, 0, 0), ( "b4", "b1", "worker-a", "chat", 3910, 5180, 6400, 420), ("c1", "a1", "worker-b", "invoke_agent check-figures", 1300, 7900, 0, 0), ( "c2", "c1", "worker-b", "chat", 1320, 2300, 1900, 110), ("c3", "c1", "worker-b", "execute_tool read_report", 2310, 2600, 0, 0), ( "c4", "c1", "worker-b", "chat", 2610, 4700, 24500, 300), ("c5", "c1", "worker-b", "execute_tool ask_agent", 4710, 7300, 0, 0), ("f1", "c5", "figures-agent", "invoke_agent figures-checker", 4760, 7250, 0, 0), ( "f2", "f1", "figures-agent", "chat", 4780, 7200, 9800, 650), ( "c6", "c1", "worker-b", "chat", 7310, 7880, 26100, 240), ( "a3", "a1", "orchestrator", "chat", 7950, 9380, 5200, 900), ] KEYS = ("id", "parent", "service", "name", "start", "end", "tokens_in", "tokens_out") spans = [dict(zip(KEYS, row), trace=TRACE) for row in ROWS] def make_traceparent(trace_id, span_id, sampled=True): return "00-" + trace_id + "-" + span_id + "-" + ("01" if sampled else "00") header = make_traceparent(TRACE, "00f067aa0ba902b7") print("sent with the request:", header) version, trace_id, parent_id, flags = header.split("-") print( "the receiver reads : trace", trace_id, "| parent span", parent_id, "| sampled", flags == "01") # The same run, where worker-b called the figures agent and forgot to send the header. broken = [dict(s) for s in spans] for s in broken: if s["service"] == "figures-agent": # the figures agent started a trace of its own s["trace"] = "9c1d7e0a55f24b0e8a61d3c2b7f09e15" if s["id"] == "f1": s["parent"] = None by_trace = {} for s in broken: by_trace.setdefault(s["trace"], []).append(s) print() for trace_id, found in by_trace.items(): tokens = sum(s["tokens_in"] + s["tokens_out"] for s in found) print( "trace", trace_id[:8] + "...", "|", str(len(found)).rjust(2), "spans |", str(tokens).rjust(5), "tokens | services:", sorted({s["service"] for s in found})) ask = next(s for s in broken if s["name"] == "execute_tool ask_agent") under = [s for s in broken if s["parent"] == ask["id"] and s["trace"] == TRACE] print( "in the main trace,", ask["name"], "took", ask["end"] - ask["start"], "ms and has", len(under), "spans under it")
Nothing crashed, and that is the danger. The figures agent got a request with no header, so it started a trace of its own. The collector now has two traces. The main one has 14 spans and looks complete. The other has 2 spans that nobody will think to look for.
Read the main trace the way an engineer would on a bad day. execute_tool ask_agent took 2,590 ms and has 0 spans under it. The time is there, and the explanation is gone. So are 10,450 tokens, which the main trace no longer counts.
A long span with nothing inside it, at a boundary between programs, usually means a dropped header. The usual culprits are a hand-written HTTP call, a queue, or a tool server that receives the header and doesn't pass it on.
b7ad6b7169203331. Which parent id does it record on that span?
traceparent: 00-0af7651916cd43dd8448eb211c80319c-00f067aa0ba902b7-01Show the answer
00f067aa0ba902b7
The third part of the header is the caller's span id, and it becomes the parent id of the receiver's span. The second part is the trace id, which the receiver copies onto its span unchanged. When this subagent calls something else, it sends a new header with the same trace id and its own span id.
Read a traceparent header
Write parse_traceparent(header). A valid header has exactly four parts joined by -: the version 00; a trace id of 32 characters; a parent id of 16 characters; flags of 2 characters. The ids and the flags use only the characters 0123456789abcdef, and an id made only of zeros is invalid. Return a dictionary with "trace_id", "parent_id" and "sampled" (True when the last bit of the flags is set: int(flags, 16) & 1). Return None for anything that isn't valid.
Uses the setup above, 30 lines.
def parse_traceparent(header): return None print(parse_traceparent("00-4bf92f3577b34da6a3ce929d0e0e4736-00f067aa0ba902b7-01"))
The run as a waterfall
The same sixteen spans, drawn. Each bar is a span, indented under its parent, colored by service, with its time and tokens at the end.
Uses the setup above, 30 lines.
import matplotlib.pyplot as plt from agentlab import spans_plot plot = [{"id": s["id"], "parent": s["parent"], "name": s["name"], "service": s["service"], "start_ms": s["start"], "end_ms": s["end"], "tokens_in": s["tokens_in"], "tokens_out": s["tokens_out"]} for s in spans] spans_plot(plot, unit="ms") plt.show()
Reading the tree
A complete tree answers the two questions people actually ask: why was it slow, and why did it cost so much. Both have a wrong way to read them.
For time, the wrong way is to add spans up. The two workers ran side by side, so their durations overlap. The useful question is: which spans was the finish waiting on? Start at the end of the run and walk backward. The span that finished last was on the path. Before it started, which span had the run been waiting for? Then that one, and so on. What you collect is the critical path, which you met when splitting work between agents. Speeding up a span on it makes the run faster. Speeding up any other span changes nothing.
For tokens, each model call's count is added to everything above it, so each branch shows what it spent.
Uses the setup above, 30 lines.
def children(spans): """parent id -> its child spans, in the order they started.""" kids = {} for s in sorted(spans, key=lambda s: s["start"]): kids.setdefault(s["parent"], []).append(s) return kids def critical(span, kids): """The leaf spans this span's end time was waiting on, in time order.""" below = kids.get(span["id"], []) if not below: return [span] path, cursor = [], span["end"] for child in sorted(below, key=lambda s: (-s["end"], s["start"])): # it finished before the next one on the path began if child["end"] <= cursor: path = critical(child, kids) + path cursor = child["start"] return path kids = children(spans) root = kids[None][0] wall = root["end"] - root["start"] workers = [s for s in kids[root["id"]] if s["name"] != "chat"] print("the run took", wall, "ms") print( "its two workers took", " + ".join(str(s["end"] - s["start"]) for s in workers), "=", sum(s["end"] - s["start"] for s in workers), "ms, in", max( s["end"] for s in workers ) - min(s["start"] for s in workers), "ms on the clock: they ran side by side") path = critical(root, kids) print() print("the critical path:") for s in path: print( " ", s["service"].ljust(13), s["name"].ljust(26), str(s["end"] - s["start"]).rjust(5), "ms") on_path = sum(s["end"] - s["start"] for s in path) print( "on the path:", on_path, "ms of", wall, "| services on it:", sorted({s["service"] for s in path})) def subtree_tokens(span, kids): return span["tokens_in"] + span[ "tokens_out" ] + sum(subtree_tokens(c, kids) for c in kids.get(span["id"], [])) total = subtree_tokens(root, kids) print() print("tokens, by what the orchestrator started:") own = sum(subtree_tokens(s, kids) for s in kids[root["id"]] if s["name"] == "chat") print( " ", "its own model calls".ljust(28), str(own).rjust(6), str(round(100 * own / total)).rjust(3) + "%") for s in kids[root["id"]]: if s["name"] != "chat": print( " ", s["name"].ljust(28), str(subtree_tokens(s, kids)).rjust(6), str(round(100 * subtree_tokens(s, kids) / total)).rjust(3) + "%") print(" ", "total".ljust(28), str(total).rjust(6)) import matplotlib.pyplot as plt branches = {"orchestrator's own calls": own} for s in kids[root["id"]]: if s["name"] != "chat": branches[s["name"]] = subtree_tokens(s, kids) plt.barh(list(branches), list(branches.values())) plt.xlabel("tokens (input + output)") plt.title("Tokens by the branch that spent them") plt.tight_layout() plt.show()
The run took 9,400 ms, and the critical path accounts for 9,010 of them. The rest is the small gaps between spans: handoffs and network.
Now look at who isn't on the path. worker-a doesn't appear at all. It finished at 5,200 ms, while worker-b ran until 7,900, so the orchestrator was never waiting for it. Making the web search twice as fast would not change this run by a millisecond. The largest single item is the outside agent's model call, at 2,420 ms.
The tokens tell a second story. The check-figures branch spent 63,600 of 80,690, which is 79%. The tree shows why: after read_report, the next model call had 24,500 input tokens, and the one after had 26,100. The whole report went into context and was sent again on every later call.
So this run has one fix for speed and a different one for cost, both in the same branch, and neither is where a glance at the list of spans would send you.
calls = [("search", 0, 400), ("read", 0, 900)] print( [ max( end - 300 if name == faster else end for name, start, end in calls ) for faster in ("search", "read")])
Show the answer
[900, 600]
The pair is finished when the slower call is. Making the search faster leaves the read still ending at 900, so nothing changes. Making the read faster moves the finish to 600. Only the read is on the critical path.
What did this branch spend?
Write subtree_tokens(spans, span_id). Each span is a dictionary with "id", "parent", "tokens_in" and "tokens_out". Return the total of input and output tokens for the span with that id and everything below it: its children, their children, and so on. Return 0 for an id that isn't among the spans.
Uses the setup above, 30 lines.
def subtree_tokens(spans, span_id): return 0 spans = [ {"id": "a1", "parent": None, "tokens_in": 0, "tokens_out": 0}, {"id": "a2", "parent": "a1", "tokens_in": 2100, "tokens_out": 180}, {"id": "b1", "parent": "a1", "tokens_in": 0, "tokens_out": 0}, {"id": "b2", "parent": "b1", "tokens_in": 1800, "tokens_out": 90}, ] print(subtree_tokens(spans, "a1"), subtree_tokens(spans, "b1"))
Write a sentence or two, then compare it with a good answer in the lesson.
In a production harness: OpenTelemetry and which traces to keep
Which traces to keep, and what goes in them
A busy system produces far more traces than anyone will read, and storing them costs money. So traces are sampled: only some are kept. When the choice is made matters a great deal.
- Head sampling decides when the run starts, before anything is known about it. It is cheap, since an unsampled run records nothing. The sampled flag in the traceparent header carries the decision to every program.
- Tail sampling decides when the run is over. Every program records its spans, the collector holds them until the trace is complete, and then keeps the interesting ones. It costs more to run.
Uses the setup above, 30 lines.
import random rng = random.Random(0) runs = [ { "trace": "%032x" % rng.getrandbits(128), "error": rng.random() < 0.02, "slow": rng.random() < 0.03 } for i in range(10000)] def chosen(trace_id, percent): """Decided from the trace id alone, so every service makes the same choice without talking to the others.""" return int(trace_id[-8:], 16) % 100 < percent # decided when the run starts head = [r for r in runs if chosen(r["trace"], 5)] # decided when the run is over tail = [r for r in runs if r["error"] or r["slow"] or chosen(r["trace"], 5)] errors = sum(r["error"] for r in runs) slow = sum(r["slow"] for r in runs) print("10,000 runs:", errors, "ended in an error,", slow, "were slow") for name, kept in ( ("keep 5%, chosen at the start", head), ("keep errors, slow runs and 5%", tail)): print( name.ljust(30), "| stored:", str(len(kept)).rjust(4), "| errors kept:", str(sum(r["error"] for r in kept)).rjust(3), "of", errors, "| slow runs kept:", str(sum(r["slow"] for r in kept)).rjust(3), "of", slow)
Keeping 5% from the start stores 478 traces and holds 12 of the 196 errors. When a user reports a failed run, its trace has almost certainly been thrown away. Keeping every error, every slow run and 5% of the rest stores 950, about twice as many, and has all 196 errors and all 314 slow runs.
The chosen function is worth a look. It decides from the trace id alone, so every program reaches the same answer without asking anyone. A trace kept by one service and dropped by another would have holes in it.
Then there is what a span holds. A span needs names and sizes: which model, which tool, how many tokens, how long, whether it failed. It rarely needs the content. Prompts and tool results are full of customer data, and a trace store is read by many people and kept for a long time. Record content only where you have decided to, with the same redaction and retention rules as any other log.
Show the answer
Decide after the run, and keep every failed trace
Failures are rare, so a small random sample misses nearly all of them. Doubling the rate doubles the cost and still loses nine in ten. Deciding at the end keeps exactly the traces people will ask for, plus a small sample of ordinary ones to compare against.
Show the answer
Every program then makes the same choice for the same run
If each program rolled its own dice, a run would be kept by some and dropped by others, and the stored trace would be a tree with branches missing. A rule based on the id gives every program the same answer with no coordination. In practice the first program decides and the flag in the header carries it, and an id-based rule is the fallback that still agrees.
Worked example: thirty-five seconds for one order
A support agent took 34.9 seconds to answer a question about an order. Three programs recorded spans: the orchestrator, the orders tool server, and the database behind it. The code rebuilds the tree, takes the critical path, and lists it longest first.
Uses the setup above, 30 lines.
ROWS = [ # span id, parent, service, name, start ms, end ms, status ("a1", None, "orchestrator", "invoke_agent support", 0, 34900, "ok"), ("a2", "a1", "orchestrator", "chat", 30, 1400, "ok"), ("a3", "a1", "orchestrator", "execute_tool lookup_order", 1410, 31900, "ok"), ( "t1", "a3", "orders-server", "tools/call lookup_order", 1430, 11430, "error: timeout"), ( "t2", "a3", "orders-server", "tools/call lookup_order", 11630, 21630, "error: timeout"), ("t3", "a3", "orders-server", "tools/call lookup_order", 22030, 31880, "ok"), ( "d1", "t1", "orders-db", "query orders", 1460, 11430, "error: canceled"), ( "d2", "t2", "orders-db", "query orders", 11660, 21630, "error: canceled"), ("d3", "t3", "orders-db", "query orders", 22060, 31850, "ok"), ("a4", "a1", "orchestrator", "chat", 31950, 33100, "ok"), ("a5", "a1", "orchestrator", "execute_tool send_reply", 33150, 33600, "ok"), ("a6", "a1", "orchestrator", "chat", 33650, 34880, "ok"), ] spans = [ dict( zip(("id", "parent", "service", "name", "start", "end", "status"), row) ) for row in ROWS] kids = {} for s in sorted(spans, key=lambda s: s["start"]): kids.setdefault(s["parent"], []).append(s) def critical(span): below = kids.get(span["id"], []) if not below: return [span] path, cursor = [], span["end"] for child in sorted(below, key=lambda s: (-s["end"], s["start"])): if child["end"] <= cursor: path = critical(child) + path cursor = child["start"] return path root = kids[None][0] path = critical(root) print("the run took", root["end"], "ms. The critical path, longest first:") for s in sorted(path, key=lambda s: s["start"] - s["end"]): print( " ", str(s["end"] - s["start"]).rjust(5), "ms", s["service"].ljust(13), s["name"].ljust(26), s["status"]) by_service = {} for s in path: by_service[s["service"]] = by_service.get(s["service"], 0) + s["end"] - s["start"] print("time on the path, by service:", by_service) print( "the tool server's three attempts:", [ ( s["end"] - s["start"], s["status"] ) for s in spans if s["service"] == "orders-server"])
Of 34,900 ms, the path spends 29,730 in the database and 4,200 in the orchestrator, so the model calls are not the problem. The three longest spans are the same query, run three times. The first two ran for 9,970 ms and were canceled, and the last line shows the tool call above each lasting exactly 10,000 ms: the tool server gives up after ten seconds and tries again. The third attempt finished in 9,790 ms, just inside the limit. So one slow query, sitting right at a timeout, was paid for three times. The fix is in the database, an index or a narrower query, and nothing about the agent needs to change. Notice what made that visible: the database's spans. Had the tool server not passed the trace along, the trace would have ended at three slow tool calls, and the tool server would have taken the blame.
Write a sentence or two, then compare it with a good answer in the lesson.
Practice what you just read
- 6 code examples you can edit and run
- a practice exercise that checks your code
- a 5-question quiz
- 12 review cards for spaced repetition
Try a quiz question
Three ids appear on every span in a distributed trace. What does each one do?
Before this lesson
Where this lesson leads.
575 lessons in thirteen phases. 176 are free without an account, and a free account opens the rest. See every lesson.
Phase 10 · about 20 hours of study
Agent harnesses
Build the software around a model that turns it into a capable, safe agent
24 lessons 1 free
- What an agent harness is FreeThe software around a model that turns it into an agent.35 min
- The core loop, built properlyStop reasons, parallel tool calls, errors and turn budgets.40 min
- A tool registry for agentsSchemas from functions, validation, output limits and helpful errors.40 min
- File and search toolsRead, write, edit by exact replacement, glob and grep inside one folder.45 min
- Running code safelyExecution tools, step limits, output caps and real sandboxes.45 min
- Giving an agent a databaseSchema as context, a query tool with caps and readable errors, guards in layers, and execution accuracy.45 min
- Permissions and approvalsRules, modes and approval prompts that keep a person in control.40 min
- Hooks and skillsExtend a harness at lifecycle events, and load know-how on demand.40 min
- Multi-agent systemsCoordinating many agents on one large job.35 min
- Managing the context windowBudgets, clearing old tool results, compaction and a cacheable prefix.45 min
- Memory, plans and long tasksTo-do lists, progress notes, memory files and checkpoints.45 min
- SubagentsHand a focused task to a fresh context with fewer tools.40 min
- Streaming, interrupts and sessionsLive output, stopping safely, and saving and resuming transcripts.40 min
- Agents as graphsState, reducers and routing, a checkpoint after every step, and pausing for a person's approval.45 min
- Evaluation harnessesRun agents on tasks in fresh environments and grade what they leave behind.45 min
- Tracing, budgets and guardrailsSee what an agent did, cap what it spends, and stop it when it loops.40 min
- Attack your own harnessRun fourteen attacks against a harness with switches, read which layer stops each one, score attacks stopped against ordinary tasks done, and keep every finding as a release test.40 min
- Capstone: build a coding agentA harness that edits a project, runs its tests and stops when they pass.45 min
- The same agent on the real SDKsPut the parts you built beside the tool runner and the Agent SDK, choose a surface for a job, and take an MCP server's tools into your registry with prefixed names, one rule per server and a capped result.35 min
- Agents that use a computerAn agent with a screen, a mouse and a keyboard: the observe-decide-act loop, what a click lands on, why a wrong hit is worse than a miss, what happens when the screen changes under a plan, and what screenshots cost.45 min
- Sandboxes for agents that browseTreat every page as untrusted input: check hosts properly, gate actions by their effect, and layer the defenses.45 min
- Protocols between agentsHand a task to an agent someone else built: agent cards, the life of a task, the messages on the wire, safe retries, and whom to trust.45 min
- Tracing a run end to endFollow one run across an orchestrator, subagents and tool servers: trace ids, the traceparent header, the critical path, tokens by branch, and which traces to keep.45 min
- Retrieval over a graphAnswer questions that hop between documents: facts as triples, a walk out from the entities a question names, and the hubs and missed facts that limit it.45 min
Never trained an AI? Teach a small one to write, right in your browser. It takes about 20 seconds and needs no account. Teach it to write