Index Ai Academy
Phase 10 · The agent frontier

Tracing a run end to end

Follow one run across an orchestrator, subagents and tool servers: trace ids, the traceparent header, the critical path, tokens by branch, and which traces to keep.

About 45 minutes · Model names checked November 2026 · A free account opens the whole lesson.
Already have an account? Sign in to open this lesson

What it covers7

Why it matters

You have traced an agent inside one program. Real agent systems are several programs: an orchestrator, subagents on other machines, tool servers, sometimes another company's agent. When a run is slow or expensive, each program's own log shows its own small part and none shows the run. Tracing end to end is how the parts are joined into one picture, and reading that picture well, which spans mattered and which didn't, is one of the most useful skills in running agents.

What you'll be able to do

  • Rebuild one run's tree from spans recorded by several separate programs
  • Carry a trace across a network hop, and recognize what a dropped hop looks like
  • Find the critical path: the spans a run's finishing time was waiting on
  • Attribute tokens to the part of the run that spent them
  • Choose which traces to keep, and decide what a span should and shouldn't record
Quick check
A guess before you start. One agent run passes through an orchestrator, two subagents and a tool server: four programs, each writing its own records. A user says the run was slow. What do you need to see the whole run?
  1. The clocks on all four machines set to the same second
  2. A single log file that all four programs write into
  3. The orchestrator's log, which contains the others
  4. One id that every program attaches to its records of that run
Show the answer

One id that every program attaches to its records of that run

Each program can only record what it did. What joins the records is a shared id for the run, passed along with every request, so that a collector can gather everything with that id from all four. That id is called the trace id, and passing it along is the whole trick.

The idea: an id crosses every hop, and parents rebuild the tree

Recall: Tracing, budgets and guardrails recorded spans inside one program. Here the run crosses programs, so each hop must carry the trace id and its own span id as the next hop's parent.

One run, many programs

In the lesson on tracing you recorded spans, timed steps that nest, inside one program. Across programs the idea is the same, and three ids hold it together.

  • The trace id names the run. Every span of the run carries it, whichever program made it.
  • The span id names one span.
  • The parent id is the span id of whatever started this span.

Each program sends its spans to a collector when it gets round to it. They arrive in no particular order, as a flat pile. The parent ids are what turn the pile back into a tree.

Setup (runs before every cell) · 30 lines

The lesson runs this once, before its first cell, and every cell, drill and example starts with what it defines. A cell that changes a setup object changes it for the cells after it, until the page reloads.

TRACE = "4bf92f3577b34da6a3ce929d0e0e4736"
# span id, parent, service, name, start ms, end ms, input tokens, output tokens
ROWS = [
    ("a1", None, "orchestrator",  "invoke_agent research",        0,    9400, 0,     0),
    (
        "a2", "a1", "orchestrator", "chat", 20, 1250, 2100, 180),
    ("b1", "a1", "worker-a",      "invoke_agent find-sources",    1300, 5200, 0,     0),
    (
        "b2", "b1", "worker-a", "chat", 1320, 2100, 1800, 90),
    ("b3", "b1", "worker-a",      "execute_tool web_search",      2110, 3900, 0,     0),
    ("s1", "b3", "search-server", "tools/call web_search",        2130, 3880, 0,     0),
    (
        "b4", "b1", "worker-a", "chat", 3910, 5180, 6400, 420),
    ("c1", "a1", "worker-b",      "invoke_agent check-figures",   1300, 7900, 0,     0),
    (
        "c2", "c1", "worker-b", "chat", 1320, 2300, 1900, 110),
    ("c3", "c1", "worker-b",      "execute_tool read_report",     2310, 2600, 0,     0),
    (
        "c4", "c1", "worker-b", "chat", 2610, 4700, 24500, 300),
    ("c5", "c1", "worker-b",      "execute_tool ask_agent",       4710, 7300, 0,     0),
    ("f1", "c5", "figures-agent", "invoke_agent figures-checker", 4760, 7250, 0,     0),
    (
        "f2", "f1", "figures-agent", "chat", 4780, 7200, 9800, 650),
    (
        "c6", "c1", "worker-b", "chat", 7310, 7880, 26100, 240),
    (
        "a3", "a1", "orchestrator", "chat", 7950, 9380, 5200, 900),
]
KEYS = ("id", "parent", "service", "name", "start", "end", "tokens_in", "tokens_out")
spans = [dict(zip(KEYS, row), trace=TRACE) for row in ROWS]
TRACE = "4bf92f3577b34da6a3ce929d0e0e4736"
# span id, parent, service, name, start ms, end ms, input tokens, output tokens
ROWS = [
    ("a1", None, "orchestrator",  "invoke_agent research",        0,    9400, 0,     0),
    (
        "a2", "a1", "orchestrator", "chat", 20, 1250, 2100, 180),
    ("b1", "a1", "worker-a",      "invoke_agent find-sources",    1300, 5200, 0,     0),
    (
        "b2", "b1", "worker-a", "chat", 1320, 2100, 1800, 90),
    ("b3", "b1", "worker-a",      "execute_tool web_search",      2110, 3900, 0,     0),
    ("s1", "b3", "search-server", "tools/call web_search",        2130, 3880, 0,     0),
    (
        "b4", "b1", "worker-a", "chat", 3910, 5180, 6400, 420),
    ("c1", "a1", "worker-b",      "invoke_agent check-figures",   1300, 7900, 0,     0),
    (
        "c2", "c1", "worker-b", "chat", 1320, 2300, 1900, 110),
    ("c3", "c1", "worker-b",      "execute_tool read_report",     2310, 2600, 0,     0),
    (
        "c4", "c1", "worker-b", "chat", 2610, 4700, 24500, 300),
    ("c5", "c1", "worker-b",      "execute_tool ask_agent",       4710, 7300, 0,     0),
    ("f1", "c5", "figures-agent", "invoke_agent figures-checker", 4760, 7250, 0,     0),
    (
        "f2", "f1", "figures-agent", "chat", 4780, 7200, 9800, 650),
    (
        "c6", "c1", "worker-b", "chat", 7310, 7880, 26100, 240),
    (
        "a3", "a1", "orchestrator", "chat", 7950, 9380, 5200, 900),
]
KEYS = ("id", "parent", "service", "name", "start", "end", "tokens_in", "tokens_out")
spans = [dict(zip(KEYS, row), trace=TRACE) for row in ROWS]

def children(spans):
    """parent id -> its child spans, in the order they started."""
    kids = {}
    for s in sorted(spans, key=lambda s: s["start"]):
        kids.setdefault(s["parent"], []).append(s)
    return kids

import random
arrived = spans[:]
# each service sends its spans when it gets round to it
random.Random(3).shuffle(arrived)
print(
    len(arrived), "spans from", len({s["service"] for s in arrived}),
    "services. The first five to arrive:")
for s in arrived[:5]:
    print(
        "  ", s["id"], "parent", str(s["parent"]).ljust(4), s["service"].ljust(13),
        s["name"])

def show(span, kids, depth=0):
    tokens = span["tokens_in"] + span["tokens_out"]
    print(
        ("  " * depth + span["name"]).ljust(36), span["service"].ljust(13),
        str(span["end"] - span["start"]).rjust(5), "ms",
        (str(tokens) + " tokens") if tokens else "")
    for child in kids.get(span["id"], []):
        show(child, kids, depth + 1)

print()
kids = children(arrived)
for root in kids[None]:
    show(root, kids)
It runs right here in your browser, with nothing to install.

The first five to arrive are from three programs and mean nothing on their own. Grouped by parent, the 16 spans from 5 services become a readable story: the orchestrator planned, started two workers, and wrote up. One worker searched the web through a tool server. The other read a report and asked an outside agent to check its figures.

Notice the span that crosses a boundary. execute_tool web_search was recorded by worker-a, and its child tools/call web_search by the search server. Two programs, one parent link. That link is only there because worker-a told the server its span id. The next part is about how.

The span names follow OpenTelemetry's conventions for generative AI: invoke_agent, chat, execute_tool. Those conventions are still marked as in development, so check the current names before you depend on them. Real span ids are 16 hex characters. The short ones here are for reading.

Quick check
From the lesson on tracing, budgets and guardrails: what is a span?
  1. One timed step of a run, which can contain other steps
  2. The whole record of a run from start to finish
  3. A limit on how many tokens a run may spend
  4. A copy of the prompt sent to the model
Show the answer

One timed step of a run, which can contain other steps

A span is one timed step: a model call, a tool call, a whole agent turn. Spans nest, so a run is a tree of them, and the tree as a whole is the trace. End-to-end tracing keeps that definition and lets the tree's branches live in different programs.

Quick check
A collector has these (span id, parent id) pairs. What does this print: the roots, and the spans whose parent never arrived?
spans = [("a1", None), ("b1", "a1"), ("b2", "b1"), ("c1", "a1"), ("x9", "q7")]
ids = {s for s, p in spans}
print(
    [s for s, p in spans if p is None],
    [s for s, p in spans if p is not None and p not in ids])
  1. ['a1'] []
  2. ['a1'] ['b2', 'x9']
  3. ['a1'] ['x9']
  4. ['a1', 'x9'] []
Show the answer

['a1'] ['x9']

A root is a span with no parent: a1. An orphan has a parent id that matches nothing the collector holds: x9 points at q7, which isn't there. b2's parent is b1, which is present. Orphans mean a span was lost on the way, or hasn't arrived yet.

Try it now

Step through the spans. Switch to the broken view: the header is dropped at one hop, and everything below it floats free.

In your own words
Why does every span carry the trace id, when the parent ids alone could link the spans into a tree?

Write a sentence or two, then compare it with a good answer in the lesson.

In code: carry the id, rebuild the tree, find the critical path

Carrying the id across

When one program calls another, the receiver needs two facts: which trace this is, and which span is calling. The standard way to send them over HTTP is one header, traceparent, defined by the W3C. It has four parts joined by dashes.

  • A version, 00.
  • The trace id: 32 hex characters.
  • The caller's span id: 16 hex characters. It becomes the parent id of whatever the receiver records.
  • Flags: 2 hex characters. The last bit says whether this trace is being sampled, that is, kept.

Sending it is called context propagation, and every hop has to do it: orchestrator to subagent, subagent to tool server, agent to outside agent. The cell builds a header, and then shows the run from before with one hop that forgot.

Uses the setup above, 30 lines.

TRACE = "4bf92f3577b34da6a3ce929d0e0e4736"
# span id, parent, service, name, start ms, end ms, input tokens, output tokens
ROWS = [
    ("a1", None, "orchestrator",  "invoke_agent research",        0,    9400, 0,     0),
    (
        "a2", "a1", "orchestrator", "chat", 20, 1250, 2100, 180),
    ("b1", "a1", "worker-a",      "invoke_agent find-sources",    1300, 5200, 0,     0),
    (
        "b2", "b1", "worker-a", "chat", 1320, 2100, 1800, 90),
    ("b3", "b1", "worker-a",      "execute_tool web_search",      2110, 3900, 0,     0),
    ("s1", "b3", "search-server", "tools/call web_search",        2130, 3880, 0,     0),
    (
        "b4", "b1", "worker-a", "chat", 3910, 5180, 6400, 420),
    ("c1", "a1", "worker-b",      "invoke_agent check-figures",   1300, 7900, 0,     0),
    (
        "c2", "c1", "worker-b", "chat", 1320, 2300, 1900, 110),
    ("c3", "c1", "worker-b",      "execute_tool read_report",     2310, 2600, 0,     0),
    (
        "c4", "c1", "worker-b", "chat", 2610, 4700, 24500, 300),
    ("c5", "c1", "worker-b",      "execute_tool ask_agent",       4710, 7300, 0,     0),
    ("f1", "c5", "figures-agent", "invoke_agent figures-checker", 4760, 7250, 0,     0),
    (
        "f2", "f1", "figures-agent", "chat", 4780, 7200, 9800, 650),
    (
        "c6", "c1", "worker-b", "chat", 7310, 7880, 26100, 240),
    (
        "a3", "a1", "orchestrator", "chat", 7950, 9380, 5200, 900),
]
KEYS = ("id", "parent", "service", "name", "start", "end", "tokens_in", "tokens_out")
spans = [dict(zip(KEYS, row), trace=TRACE) for row in ROWS]

def make_traceparent(trace_id, span_id, sampled=True):
    return "00-" + trace_id + "-" + span_id + "-" + ("01" if sampled else "00")

header = make_traceparent(TRACE, "00f067aa0ba902b7")
print("sent with the request:", header)
version, trace_id, parent_id, flags = header.split("-")
print(
    "the receiver reads   : trace", trace_id, "| parent span", parent_id, "| sampled",
    flags == "01")

# The same run, where worker-b called the figures agent and forgot to send the header.
broken = [dict(s) for s in spans]
for s in broken:
    if s["service"] == "figures-agent":
        # the figures agent started a trace of its own
        s["trace"] = "9c1d7e0a55f24b0e8a61d3c2b7f09e15"
        if s["id"] == "f1":
            s["parent"] = None

by_trace = {}
for s in broken:
    by_trace.setdefault(s["trace"], []).append(s)
print()
for trace_id, found in by_trace.items():
    tokens = sum(s["tokens_in"] + s["tokens_out"] for s in found)
    print(
        "trace", trace_id[:8] + "...", "|", str(len(found)).rjust(2), "spans |",
        str(tokens).rjust(5), "tokens | services:",
        sorted({s["service"] for s in found}))
ask = next(s for s in broken if s["name"] == "execute_tool ask_agent")
under = [s for s in broken if s["parent"] == ask["id"] and s["trace"] == TRACE]
print(
    "in the main trace,", ask["name"], "took", ask["end"] - ask["start"], "ms and has",
    len(under), "spans under it")

Nothing crashed, and that is the danger. The figures agent got a request with no header, so it started a trace of its own. The collector now has two traces. The main one has 14 spans and looks complete. The other has 2 spans that nobody will think to look for.

Read the main trace the way an engineer would on a bad day. execute_tool ask_agent took 2,590 ms and has 0 spans under it. The time is there, and the explanation is gone. So are 10,450 tokens, which the main trace no longer counts.

A long span with nothing inside it, at a boundary between programs, usually means a dropped header. The usual culprits are a hand-written HTTP call, a queue, or a tool server that receives the header and doesn't pass it on.

Quick check
A subagent receives a request with this header, and starts a span of its own with id b7ad6b7169203331. Which parent id does it record on that span? traceparent: 00-0af7651916cd43dd8448eb211c80319c-00f067aa0ba902b7-01
  1. 00
  2. 0af7651916cd43dd8448eb211c80319c
  3. b7ad6b7169203331
  4. 00f067aa0ba902b7
Show the answer

00f067aa0ba902b7

The third part of the header is the caller's span id, and it becomes the parent id of the receiver's span. The second part is the trace id, which the receiver copies onto its span unchanged. When this subagent calls something else, it sends a new header with the same trace id and its own span id.

Drill

Read a traceparent header

Write parse_traceparent(header). A valid header has exactly four parts joined by -: the version 00; a trace id of 32 characters; a parent id of 16 characters; flags of 2 characters. The ids and the flags use only the characters 0123456789abcdef, and an id made only of zeros is invalid. Return a dictionary with "trace_id", "parent_id" and "sampled" (True when the last bit of the flags is set: int(flags, 16) & 1). Return None for anything that isn't valid.

Uses the setup above, 30 lines.

def parse_traceparent(header):
    return None


print(parse_traceparent("00-4bf92f3577b34da6a3ce929d0e0e4736-00f067aa0ba902b7-01"))
In the lesson, your code is checked as you write it.
Worked example

The run as a waterfall

The same sixteen spans, drawn. Each bar is a span, indented under its parent, colored by service, with its time and tokens at the end.

Uses the setup above, 30 lines.

import matplotlib.pyplot as plt
from agentlab import spans_plot

plot = [{"id": s["id"], "parent": s["parent"], "name": s["name"],
         "service": s["service"], "start_ms": s["start"], "end_ms": s["end"],
         "tokens_in": s["tokens_in"], "tokens_out": s["tokens_out"]}
        for s in spans]
spans_plot(plot, unit="ms")
plt.show()
In the lesson you can change this and run it.

Reading the tree

A complete tree answers the two questions people actually ask: why was it slow, and why did it cost so much. Both have a wrong way to read them.

For time, the wrong way is to add spans up. The two workers ran side by side, so their durations overlap. The useful question is: which spans was the finish waiting on? Start at the end of the run and walk backward. The span that finished last was on the path. Before it started, which span had the run been waiting for? Then that one, and so on. What you collect is the critical path, which you met when splitting work between agents. Speeding up a span on it makes the run faster. Speeding up any other span changes nothing.

For tokens, each model call's count is added to everything above it, so each branch shows what it spent.

Uses the setup above, 30 lines.

def children(spans):
    """parent id -> its child spans, in the order they started."""
    kids = {}
    for s in sorted(spans, key=lambda s: s["start"]):
        kids.setdefault(s["parent"], []).append(s)
    return kids

def critical(span, kids):
    """The leaf spans this span's end time was waiting on, in time order."""
    below = kids.get(span["id"], [])
    if not below:
        return [span]
    path, cursor = [], span["end"]
    for child in sorted(below, key=lambda s: (-s["end"], s["start"])):
        # it finished before the next one on the path began
        if child["end"] <= cursor:
            path = critical(child, kids) + path
            cursor = child["start"]
    return path

kids = children(spans)
root = kids[None][0]
wall = root["end"] - root["start"]
workers = [s for s in kids[root["id"]] if s["name"] != "chat"]
print("the run took", wall, "ms")
print(
    "its two workers took", " + ".join(str(s["end"] - s["start"]) for s in workers),
    "=", sum(s["end"] - s["start"] for s in workers), "ms, in",
      max(
          s["end"] for s in workers
      ) - min(s["start"] for s in workers), "ms on the clock: they ran side by side")

path = critical(root, kids)
print()
print("the critical path:")
for s in path:
    print(
        "  ", s["service"].ljust(13), s["name"].ljust(26),
        str(s["end"] - s["start"]).rjust(5), "ms")
on_path = sum(s["end"] - s["start"] for s in path)
print(
    "on the path:", on_path, "ms of", wall, "| services on it:",
    sorted({s["service"] for s in path}))

def subtree_tokens(span, kids):
    return span["tokens_in"] + span[
        "tokens_out"
    ] + sum(subtree_tokens(c, kids) for c in kids.get(span["id"], []))

total = subtree_tokens(root, kids)
print()
print("tokens, by what the orchestrator started:")
own = sum(subtree_tokens(s, kids) for s in kids[root["id"]] if s["name"] == "chat")
print(
    "  ", "its own model calls".ljust(28), str(own).rjust(6),
    str(round(100 * own / total)).rjust(3) + "%")
for s in kids[root["id"]]:
    if s["name"] != "chat":
        print(
            "  ", s["name"].ljust(28), str(subtree_tokens(s, kids)).rjust(6),
            str(round(100 * subtree_tokens(s, kids) / total)).rjust(3) + "%")
print("  ", "total".ljust(28), str(total).rjust(6))

import matplotlib.pyplot as plt

branches = {"orchestrator's own calls": own}
for s in kids[root["id"]]:
    if s["name"] != "chat":
        branches[s["name"]] = subtree_tokens(s, kids)
plt.barh(list(branches), list(branches.values()))
plt.xlabel("tokens (input + output)")
plt.title("Tokens by the branch that spent them")
plt.tight_layout()
plt.show()

The run took 9,400 ms, and the critical path accounts for 9,010 of them. The rest is the small gaps between spans: handoffs and network.

Now look at who isn't on the path. worker-a doesn't appear at all. It finished at 5,200 ms, while worker-b ran until 7,900, so the orchestrator was never waiting for it. Making the web search twice as fast would not change this run by a millisecond. The largest single item is the outside agent's model call, at 2,420 ms.

The tokens tell a second story. The check-figures branch spent 63,600 of 80,690, which is 79%. The tree shows why: after read_report, the next model call had 24,500 input tokens, and the one after had 26,100. The whole report went into context and was sent again on every later call.

So this run has one fix for speed and a different one for cost, both in the same branch, and neither is where a glance at the list of spans would send you.

Quick check
Two tool calls start together under one span: a search that ends at 400 ms and a file read that ends at 900 ms. What does this print: when the pair finishes if the search gets 300 ms faster, and then if the read does?
calls = [("search", 0, 400), ("read", 0, 900)]
print(
    [
        max(
            end - 300 if name == faster else end for name, start, end in calls
        ) for faster in ("search", "read")])
  1. [900, 900]
  2. [600, 600]
  3. [600, 900]
  4. [900, 600]
Show the answer

[900, 600]

The pair is finished when the slower call is. Making the search faster leaves the read still ending at 900, so nothing changes. Making the read faster moves the finish to 600. Only the read is on the critical path.

Drill

What did this branch spend?

Write subtree_tokens(spans, span_id). Each span is a dictionary with "id", "parent", "tokens_in" and "tokens_out". Return the total of input and output tokens for the span with that id and everything below it: its children, their children, and so on. Return 0 for an id that isn't among the spans.

Uses the setup above, 30 lines.

def subtree_tokens(spans, span_id):
    return 0


spans = [
    {"id": "a1", "parent": None, "tokens_in": 0, "tokens_out": 0},
    {"id": "a2", "parent": "a1", "tokens_in": 2100, "tokens_out": 180},
    {"id": "b1", "parent": "a1", "tokens_in": 0, "tokens_out": 0},
    {"id": "b2", "parent": "b1", "tokens_in": 1800, "tokens_out": 90},
]
print(subtree_tokens(spans, "a1"), subtree_tokens(spans, "b1"))
In the lesson, your code is checked as you write it.
In your own words
A teammate sees that a subagent took 3.9 seconds and proposes a week of work to speed it up. What do you check first?

Write a sentence or two, then compare it with a good answer in the lesson.

In a production harness: OpenTelemetry and which traces to keep

Which traces to keep, and what goes in them

A busy system produces far more traces than anyone will read, and storing them costs money. So traces are sampled: only some are kept. When the choice is made matters a great deal.

  • Head sampling decides when the run starts, before anything is known about it. It is cheap, since an unsampled run records nothing. The sampled flag in the traceparent header carries the decision to every program.
  • Tail sampling decides when the run is over. Every program records its spans, the collector holds them until the trace is complete, and then keeps the interesting ones. It costs more to run.

Uses the setup above, 30 lines.

import random

rng = random.Random(0)
runs = [
    {
        "trace": "%032x" % rng.getrandbits(128), "error": rng.random() < 0.02,
        "slow": rng.random() < 0.03
    } for i in range(10000)]

def chosen(trace_id, percent):
    """Decided from the trace id alone, so every service makes the same choice
    without talking to the others."""
    return int(trace_id[-8:], 16) % 100 < percent

# decided when the run starts
head = [r for r in runs if chosen(r["trace"], 5)]
# decided when the run is over
tail = [r for r in runs if r["error"] or r["slow"] or chosen(r["trace"], 5)]

errors = sum(r["error"] for r in runs)
slow = sum(r["slow"] for r in runs)
print("10,000 runs:", errors, "ended in an error,", slow, "were slow")
for name, kept in (
    ("keep 5%, chosen at the start", head), ("keep errors, slow runs and 5%", tail)):
    print(
        name.ljust(30), "| stored:", str(len(kept)).rjust(4), "| errors kept:",
        str(sum(r["error"] for r in kept)).rjust(3), "of", errors,
          "| slow runs kept:", str(sum(r["slow"] for r in kept)).rjust(3), "of", slow)

Keeping 5% from the start stores 478 traces and holds 12 of the 196 errors. When a user reports a failed run, its trace has almost certainly been thrown away. Keeping every error, every slow run and 5% of the rest stores 950, about twice as many, and has all 196 errors and all 314 slow runs.

The chosen function is worth a look. It decides from the trace id alone, so every program reaches the same answer without asking anyone. A trace kept by one service and dropped by another would have holes in it.

Then there is what a span holds. A span needs names and sizes: which model, which tool, how many tokens, how long, whether it failed. It rarely needs the content. Prompts and tool results are full of customer data, and a trace store is read by many people and kept for a long time. Record content only where you have decided to, with the same redaction and retention rules as any other log.

Quick check
A team keeps 5% of traces, chosen when each run starts. Users report failures that the team can almost never find a trace for. What is the best change?
  1. Ask users to send a screenshot with each report
  2. Keep 10% of traces, still chosen at the very start
  3. Record the full prompt in every span that is kept
  4. Decide after the run, and keep every failed trace
Show the answer

Decide after the run, and keep every failed trace

Failures are rare, so a small random sample misses nearly all of them. Doubling the rate doubles the cost and still loses nine in ten. Deciding at the end keeps exactly the traces people will ask for, plus a small sample of ordinary ones to compare against.

Quick check
Why should the decision to keep a trace be worked out from the trace id, and not by each program picking a random number?
  1. Trace ids are secret and random numbers are not
  2. The trace id records whether the run had an error
  3. Random numbers are too slow to generate per request
  4. Every program then makes the same choice for the same run
Show the answer

Every program then makes the same choice for the same run

If each program rolled its own dice, a run would be kept by some and dropped by others, and the stored trace would be a tree with branches missing. A rule based on the id gives every program the same answer with no coordination. In practice the first program decides and the flag in the header carries it, and an id-based rule is the fallback that still agrees.

Worked example

Worked example: thirty-five seconds for one order

A support agent took 34.9 seconds to answer a question about an order. Three programs recorded spans: the orchestrator, the orders tool server, and the database behind it. The code rebuilds the tree, takes the critical path, and lists it longest first.

Uses the setup above, 30 lines.

ROWS = [  # span id, parent, service, name, start ms, end ms, status
    ("a1", None, "orchestrator",  "invoke_agent support",       0,     34900, "ok"),
    ("a2", "a1", "orchestrator",  "chat",                       30,    1400,  "ok"),
    ("a3", "a1", "orchestrator",  "execute_tool lookup_order",  1410,  31900, "ok"),
    (
        "t1", "a3", "orders-server", "tools/call lookup_order", 1430, 11430,
        "error: timeout"),
    (
        "t2", "a3", "orders-server", "tools/call lookup_order", 11630, 21630,
        "error: timeout"),
    ("t3", "a3", "orders-server", "tools/call lookup_order",    22030, 31880, "ok"),
    (
        "d1", "t1", "orders-db", "query orders", 1460, 11430, "error: canceled"),
    (
        "d2", "t2", "orders-db", "query orders", 11660, 21630, "error: canceled"),
    ("d3", "t3", "orders-db",     "query orders",               22060, 31850, "ok"),
    ("a4", "a1", "orchestrator",  "chat",                       31950, 33100, "ok"),
    ("a5", "a1", "orchestrator",  "execute_tool send_reply",    33150, 33600, "ok"),
    ("a6", "a1", "orchestrator",  "chat",                       33650, 34880, "ok"),
]
spans = [
    dict(
        zip(("id", "parent", "service", "name", "start", "end", "status"), row)
    ) for row in ROWS]
kids = {}
for s in sorted(spans, key=lambda s: s["start"]):
    kids.setdefault(s["parent"], []).append(s)

def critical(span):
    below = kids.get(span["id"], [])
    if not below:
        return [span]
    path, cursor = [], span["end"]
    for child in sorted(below, key=lambda s: (-s["end"], s["start"])):
        if child["end"] <= cursor:
            path = critical(child) + path
            cursor = child["start"]
    return path

root = kids[None][0]
path = critical(root)
print("the run took", root["end"], "ms. The critical path, longest first:")
for s in sorted(path, key=lambda s: s["start"] - s["end"]):
    print(
        "  ", str(s["end"] - s["start"]).rjust(5), "ms", s["service"].ljust(13),
        s["name"].ljust(26), s["status"])
by_service = {}
for s in path:
    by_service[s["service"]] = by_service.get(s["service"], 0) + s["end"] - s["start"]
print("time on the path, by service:", by_service)
print(
    "the tool server's three attempts:",
    [
        (
            s["end"] - s["start"], s["status"]
        ) for s in spans if s["service"] == "orders-server"])
In the lesson you can change this and run it.

Of 34,900 ms, the path spends 29,730 in the database and 4,200 in the orchestrator, so the model calls are not the problem. The three longest spans are the same query, run three times. The first two ran for 9,970 ms and were canceled, and the last line shows the tool call above each lasting exactly 10,000 ms: the tool server gives up after ten seconds and tries again. The third attempt finished in 9,790 ms, just inside the limit. So one slow query, sitting right at a timeout, was paid for three times. The fix is in the database, an index or a narrower query, and nothing about the agent needs to change. Notice what made that visible: the database's spans. Had the tool server not passed the trace along, the trace would have ended at three slow tool calls, and the tool server would have taken the blame.

In your own words
You are adding tracing to a system with an orchestrator, subagents and tool servers. What do you check first?

Write a sentence or two, then compare it with a good answer in the lesson.

Practice what you just read

  • 6 code examples you can edit and run
  • a practice exercise that checks your code
  • a 5-question quiz
  • 12 review cards for spaced repetition

Try a quiz question

Three ids appear on every span in a distributed trace. What does each one do?

  1. Trace id names the run, span id the step, parent id what started it
  2. Trace id names the program, span id the run, parent id the user
  3. Trace id names the step, span id the run, parent id the program
  4. Trace id names the user, span id the model, parent id the tool

Answer it at the end of the lesson to see if you're right, and why.

Before this lesson

Keep learning

Where this lesson leads.

575 lessons in thirteen phases. 176 are free without an account, and a free account opens the rest. See every lesson.

Phase 10 · about 20 hours of study

Agent harnesses

Build the software around a model that turns it into a capable, safe agent 24 lessons 1 free
  1. What an agent harness is FreeThe software around a model that turns it into an agent.35 min
  2. The core loop, built properlyStop reasons, parallel tool calls, errors and turn budgets.40 min
  3. A tool registry for agentsSchemas from functions, validation, output limits and helpful errors.40 min
  4. File and search toolsRead, write, edit by exact replacement, glob and grep inside one folder.45 min
  5. Running code safelyExecution tools, step limits, output caps and real sandboxes.45 min
  6. Giving an agent a databaseSchema as context, a query tool with caps and readable errors, guards in layers, and execution accuracy.45 min
  7. Permissions and approvalsRules, modes and approval prompts that keep a person in control.40 min
  8. Hooks and skillsExtend a harness at lifecycle events, and load know-how on demand.40 min
  9. Multi-agent systemsCoordinating many agents on one large job.35 min
  10. Managing the context windowBudgets, clearing old tool results, compaction and a cacheable prefix.45 min
  11. Memory, plans and long tasksTo-do lists, progress notes, memory files and checkpoints.45 min
  12. SubagentsHand a focused task to a fresh context with fewer tools.40 min
  13. Streaming, interrupts and sessionsLive output, stopping safely, and saving and resuming transcripts.40 min
  14. Agents as graphsState, reducers and routing, a checkpoint after every step, and pausing for a person's approval.45 min
  15. Evaluation harnessesRun agents on tasks in fresh environments and grade what they leave behind.45 min
  16. Tracing, budgets and guardrailsSee what an agent did, cap what it spends, and stop it when it loops.40 min
  17. Attack your own harnessRun fourteen attacks against a harness with switches, read which layer stops each one, score attacks stopped against ordinary tasks done, and keep every finding as a release test.40 min
  18. Capstone: build a coding agentA harness that edits a project, runs its tests and stops when they pass.45 min
  19. The same agent on the real SDKsPut the parts you built beside the tool runner and the Agent SDK, choose a surface for a job, and take an MCP server's tools into your registry with prefixed names, one rule per server and a capped result.35 min
  20. Agents that use a computerAn agent with a screen, a mouse and a keyboard: the observe-decide-act loop, what a click lands on, why a wrong hit is worse than a miss, what happens when the screen changes under a plan, and what screenshots cost.45 min
  21. Sandboxes for agents that browseTreat every page as untrusted input: check hosts properly, gate actions by their effect, and layer the defenses.45 min
  22. Protocols between agentsHand a task to an agent someone else built: agent cards, the life of a task, the messages on the wire, safe retries, and whom to trust.45 min
  23. Tracing a run end to endFollow one run across an orchestrator, subagents and tool servers: trace ids, the traceparent header, the critical path, tokens by branch, and which traces to keep.45 min
  24. Retrieval over a graphAnswer questions that hop between documents: facts as triples, a walk out from the entities a question names, and the hubs and missed facts that limit it.45 min

Never trained an AI? Teach a small one to write, right in your browser. It takes about 20 seconds and needs no account. Teach it to write