Index Ai Academy
Phase 10 · The agent frontier

Agents that use a computer

An agent with a screen, a mouse and a keyboard: the observe-decide-act loop, what a click lands on, why a wrong hit is worse than a miss, what happens when the screen changes under a plan, and what screenshots cost.

About 45 minutes · A free account opens the whole lesson.
Already have an account? Sign in to open this lesson

What it covers7

Why it matters

Most of the software people use has no API for an agent to call. It has a screen, a mouse and a keyboard. A computer-use agent works that way: it is shown the screen, decides on a click or some typing, and is shown the screen again. That lets a model operate almost anything a person can, and it brings problems that tool calls never had. A click can miss. It can land, convincingly, on the wrong thing. The screen can change between looking and acting. This lesson builds the loop on a small simulated application and measures each of those problems.

What you'll be able to do

  • Describe the loop of an agent that operates a screen: observe, decide, act, observe again
  • Turn a target on screen into a click, and work out what a click at a point lands on
  • Explain why an agent must verify the result of its actions, and measure what happens when it doesn't
  • Show why a plan made from one observation goes wrong when the screen changes
  • Choose between an API, a structured view of the page, and raw pixels, and estimate what each costs
Quick check
A guess before you start. An agent looks at one screenshot of a booking form, works out five clicks and keystrokes, and carries them all out without looking again. What is the main risk?
  1. Typing is slower than clicking, so the plan times out
  2. Five actions are too many for a model to plan
  3. The model forgets the screenshot after the first click
  4. The screen changes partway, and later clicks land elsewhere
Show the answer

The screen changes partway, and later clicks land elsewhere

The plan is a list of screen positions that were right when the screenshot was taken. If a dialog appears, a page loads late or a list reorders, the same positions now hold different things, and the agent presses them anyway. Looking again before each action is the basic defense, and you will see below what happens without it.

The idea: look, decide, act, look again

Recall: The core loop, built properly ran a loop with tool results. A computer-use agent runs the same loop with the screen in place of the result, and Sandboxes for agents that browse shows what such an agent can reach.

A model with a mouse

An agent with tools calls functions: search_flights(from, to). A computer-use agent has one very general tool instead: the computer. Its actions are the ones a person has.

  • Click at a position on the screen, given as x and y in pixels from the top left corner.
  • Type some text, into whatever has the keyboard's focus.
  • Press a key, scroll, or wait.

And its observation is the screen. Usually that is a screenshot, an image the model reads. Sometimes the harness can also supply the screen as a list of its elements: each button, text box and label, with its position. Browsers and operating systems keep such a list for screen readers, where it is called the accessibility tree.

The application below is a small flight-booking screen, simulated in Python. elements() returns what is on the screen, and hit(x, y) says what a click at a point would land on.

Setup (runs before every cell) · 140 lines

The lesson runs this once, before its first cell, and every cell, drill and example starts with what it defines. A cell that changes a setup object changes it for the cells after it, until the page reloads.

import random

FLIGHTS = {"book-1": "09:10 direct $240", "book-2": "13:45 one stop $180"}

class App:
    """A tiny flight-booking screen, 400 by 200: a stand-in for a real application."""
    def __init__(self, popup_after=None):
        self.page, self.focus, self.clicks, self.booked = "search", None, 0, None
        self.values = {"from": "", "to": ""}
        self.popup_after, self.popup, self.subscribed = popup_after, False, False

    def box(self, id, role, label, x, y, w, h):
        return {"id": id, "role": role, "label": label, "x": x, "y": y, "w": w, "h": h}

    def elements(self):
        """The boxes on screen now. Later ones are drawn on top of earlier ones."""
        if self.page == "search":
            out = [
                self.box(
                    "from", "textbox", "From: " + self.values["from"], 20, 40, 160, 30),
                   self.box(
                       "to", "textbox", "To: " + self.values["to"], 200, 40, 160, 30),
                   self.box("search", "button", "Search", 150, 100, 80, 30)]
        elif self.page == "results":
            out = [
                self.box("flight-1", "text", FLIGHTS["book-1"], 20, 40, 280, 30),
                self.box("book-1", "button", "Book", 320, 44, 44, 22),
                   self.box(
                       "flight-2", "text", FLIGHTS["book-2"], 20, 80, 280, 30
                   ), self.box("book-2", "button", "Book", 320, 84, 44, 22)]
        elif self.page == "confirm":
            out = [
                self.box(
                    "summary", "text", "Book " + self.booked + "?", 60, 60, 280, 30),
                   self.box(
                       "cancel", "button", "Cancel", 110, 120, 70, 30
                   ), self.box("confirm", "button", "Confirm", 210, 120, 80, 30)]
        else:
            out = [self.box("done", "text", "Booked: " + self.booked, 60, 60, 280, 30)]
        if self.popup:
            out += [
                self.box("popup", "dialog", "Sign up for deals!", 100, 30, 220, 140),
                    self.box(
                        "popup-yes", "button", "Subscribe", 130, 110, 90, 30
                    ), self.box("popup-no", "button", "No thanks", 230, 110, 80, 30)]
        return out

    def hit(self, x, y):
        """The topmost element under a point, or None."""
        found = None
        for e in self.elements():
            if e["x"] <= x < e["x"] + e["w"] and e["y"] <= y < e["y"] + e["h"]:
                found = e
        return found

    def click(self, x, y):
        self.clicks += 1
        target = self.hit(x, y)
        if self.clicks == self.popup_after:
            self.popup = True  # a dialog slides in just after this click
        if target is None:
            self.focus = None
            return "clicked empty space"
        id = target["id"]
        if id == "popup":
            return "clicked the dialog, nothing happened"
        if id in ("popup-yes", "popup-no"):
            self.popup, self.subscribed = False, self.subscribed or id == "popup-yes"
            if id == "popup-yes":
                return "SUBSCRIBED to marketing email"
            return "closed the dialog"
        self.focus = id if target["role"] == "textbox" else None
        if id == "search" and self.values["from"] and self.values["to"]:
            self.page = "results"
        elif id in FLIGHTS:
            self.page, self.booked = "confirm", FLIGHTS[id]
        elif id == "confirm":
            self.page = "done"
        elif id == "cancel":
            self.page, self.booked = "results", None
        return "clicked " + id

    def type(self, text):
        if self.focus is None:
            return "typed, with nothing focused"
        self.values[self.focus] += text
        return "typed into " + self.focus

def center(e):
    return e["x"] + e["w"] // 2, e["y"] + e["h"] // 2

def decide(elements, goal, check=True):
    """A stand-in for the model: rules where a real model would reason."""
    on = {e["id"]: e for e in elements}
    if "popup-no" in on:
        return ("click", "popup-no")
    if "from" in on:
        for field in ("from", "to"):
            if on[field]["label"] != field.capitalize() + ": " + goal[field]:
                return ("fill", field)
        return ("click", "search")
    if "book-1" in on:
        return ("click", "book-2")  # the goal asks for the cheapest flight
    if "confirm" in on:
        wrong = check and goal["flight"] not in on["summary"]["label"]
        return ("click", "cancel" if wrong else "confirm")
    return ("stop", None)

def run(app, goal, noise=0.0, seed=0, check=True, max_actions=40, log=False):
    """Look, decide, act, and look again, until the model says stop."""
    rng, actions = random.Random(seed), 0
    while actions < max_actions:
        elements = app.elements()                               # 1. observe
        kind, target = decide(elements, goal, check)            # 2. decide
        if kind == "stop":
            break
        x, y = center(next(e for e in elements if e["id"] == target))
        x = round(x + rng.gauss(0, noise))  # the aim is a little off
        y = round(y + rng.gauss(0, noise))
        result = app.click(x, y)                                # 3. act
        actions += 1
        if log:
            print(str(actions).rjust(2), "click", str((x, y)).ljust(10),
                      "aiming at", target.ljust(8), "->", result)
        if kind == "fill" and app.focus == target:
            result = app.type(goal[target])
            actions += 1
            if log:
                print(str(actions).rjust(2), "type ", repr(goal[target]).ljust(10),
                          " " * 18, "->", result)
    return app, actions

GOAL = {"from": "Lisbon", "to": "Oslo", "flight": "$180"}

def outcome(app):
    if app.page != "done":
        return "unfinished"
    right = GOAL["flight"] in app.booked
    right = right and app.values == {"from": "Lisbon", "to": "Oslo"}
    return "right" if right else "wrong booking"
app = App()
print("the screen, as a list of elements:")
for e in app.elements():
    print(
        " ", e["id"].ljust(7), e["role"].ljust(8), repr(e["label"]).ljust(10), "| left",
        str(e["x"]).rjust(3), "top", str(e["y"]).rjust(3), "| width",
        str(e["w"]).rjust(3), "height", e["h"])

print()
for point in ((100, 55), (190, 115), (190, 90), (179, 69), (180, 70)):
    target = app.hit(*point)
    print(
        "a click at", str(point).ljust(10), "lands on:",
        target["id"] if target else "nothing")

Three elements, each a box. A click at (100, 55) lands inside the "From" box, and one at (190, 115) on the Search button. A click at (190, 90) is between them and lands on nothing.

The last two lines are a pixel apart. (179, 69) is the bottom right corner of the "From" box, and (180, 70) is just outside it. To the application, a click is a point, and a point is either inside a box or it isn't.

Quick check
Two elements overlap on screen: a wide row, and a small button drawn on top of it. What does this print: what a click at each point lands on?
elements = [
    {"id": "row", "x": 0, "y": 0, "w": 300, "h": 40},
    # drawn later, so it is on top
    {"id": "delete", "x": 250, "y": 5, "w": 40, "h": 30},
]

def hit(x, y):
    found = None
    for e in elements:
        if e["x"] <= x < e["x"] + e["w"] and e["y"] <= y < e["y"] + e["h"]:
            found = e["id"]
    return found

print(hit(100, 20), hit(260, 20), hit(260, 37), hit(310, 20))
  1. row delete delete None
  2. row delete row None
  3. delete delete row row
  4. row row row None
Show the answer

row delete row None

At (100, 20) only the row is under the point. At (260, 20) both are, and the button, drawn later, is on top. At (260, 37) the point is below the button's bottom edge at 35, so it falls through to the row. (310, 20) is past the row's right edge. A slip of a few pixels near a Delete button changes what gets pressed.

Drill

What does a click land on?

Write hit(elements, x, y). Each element is a dictionary with id, x, y, w and h: its left edge, top edge, width and height. Elements later in the list are drawn on top of earlier ones. Return the id of the topmost element that contains the point, or None if there is none. A box contains its left and top edges and not its right and bottom ones.

Uses the setup above, 140 lines.

def hit(elements, x, y):
    return None


screen = [
    {"id": "row", "x": 0, "y": 0, "w": 300, "h": 40},
    {"id": "delete", "x": 250, "y": 5, "w": 40, "h": 30},
]
print(hit(screen, 100, 20), hit(screen, 260, 20), hit(screen, 310, 20))
In the lesson, your code is checked as you write it.
Try it now

Look, decide, act, look again, until the model says it is finished or a limit is reached.

In your own words
How does a computer-use agent's interface to the world differ from that of an agent with one tool per task?

Write a sentence or two, then compare it with a good answer in the lesson.

In code: aim, verification and stale screens

Look, decide, act, look again

The harness for such an agent is the loop you know, with the screen in place of tool results.

  1. Observe. Capture the screen.
  2. Decide. The model reads it, with the goal and the history, and picks one action.
  3. Act. The harness performs that action.
  4. Go back to step 1, until the model says it is finished or a limit is reached.

In the code below, decide is a stand-in for the model: a few rules where a real model would reason about the screenshot. Everything else is the real loop. The goal is to book the cheapest flight from Lisbon to Oslo.

The loop is run, and decide is the stand-in model. Both sit in the lesson's setup above, so the cell below only starts the run. This is the loop in full:

def run(app, goal, noise=0.0, seed=0, check=True, max_actions=40, log=False):
    """Look, decide, act, and look again, until the model says stop."""
    rng, actions = random.Random(seed), 0
    while actions < max_actions:
        elements = app.elements()                               # 1. observe
        kind, target = decide(elements, goal, check)            # 2. decide
        if kind == "stop":
            break
        x, y = center(next(e for e in elements if e["id"] == target))
        # the model's aim is a little off
        x, y = round(x + rng.gauss(0, noise)), round(y + rng.gauss(0, noise))
        result = app.click(x, y)                                # 3. act
        actions += 1
        if log:
            print(
                str(actions).rjust(2), "click", str((x, y)).ljust(10), "aiming at",
                target.ljust(8), "->", result)
        if kind == "fill" and app.focus == target:
            result = app.type(goal[target])
            actions += 1
            if log:
                print(
                    str(actions).rjust(2), "type ", repr(goal[target]).ljust(10),
                    " " * 18, "->", result)
    return app, actions

Uses the setup above, 140 lines.

app, actions = run(App(), GOAL, log=True)
print("result:", outcome(app), "|", app.elements()[0]["label"], "| actions:", actions)

Seven actions. Each click is aimed at the center of the element the "model" chose, and each line shows what the application did in response. Typing goes to whichever box was clicked last, since that box has the focus.

Notice what the loop does not do: it never assumes an action worked. After the click on "From", it looks at the screen again before typing. That costs an observation per action, and the next two sections show why it is worth it.

Quick check
From the lesson on the core loop: an agent harness always sets a limit on the number of turns. What is that limit for?
  1. To keep the model from using more than one tool
  2. To stop a run that isn't making progress from going on forever
  3. To make sure every run takes the same number of turns
  4. To make the model answer faster on each turn
Show the answer

To stop a run that isn't making progress from going on forever

A turn limit is the last line of defense against a run that loops: an agent retrying a click that keeps missing, say, will otherwise spend tokens until someone notices. The computer-use loop above has the same guard, as a cap on actions.

Worked example

500 clicks on the screen, colored by what they hit

The lower Book button is the target. Each click has an error of 20 pixels, and the dot's color says what it landed on.

Uses the setup above, 140 lines.

import matplotlib.pyplot as plt
from matplotlib.patches import Rectangle

app = App()
app.page = "results"
boxes = app.elements()
cx, cy = center(next(e for e in boxes if e["id"] == "book-2"))
rng = random.Random(1)
colors = {"book-2": "tab:green", "book-1": "tab:red", None: "tab:gray"}
counts = {"book-2": 0, "book-1": 0, None: 0}
fig, ax = plt.subplots(figsize=(6, 3.4))
for e in boxes:
    ax.add_patch(Rectangle((e["x"], e["y"]), e["w"], e["h"], fill=False))
for _ in range(500):
    x, y = cx + rng.gauss(0, 20), cy + rng.gauss(0, 20)
    hit = app.hit(x, y)
    key = hit["id"] if hit and hit["id"] in counts else None
    counts[key] += 1
    ax.plot(x, y, ".", color=colors[key], markersize=3)
ax.set_xlim(250, 420)
ax.set_ylim(130, 20)
ax.set_title("500 clicks aimed at the lower Book button, error 20 pixels")
plt.show()
print(counts)
In the lesson you can change this and run it.

Aim is imperfect

A model reading a screenshot has to turn "the Book button" into two numbers. It is good at that and not exact: its coordinates can be off by some pixels, more on small or crowded targets.

The cell adds random error to every click, and runs the task 500 times at each level. It also compares two versions of the agent. One confirms a booking as soon as the confirmation page appears. The other reads the summary on that page first, and cancels if it isn't the flight it meant to book.

Uses the setup above, 140 lines.

print("aim error | no check: right, wrong, unfinished |")
print("          | check: right, wrong, unfinished | actions")
import matplotlib.pyplot as plt

wrong = {False: [], True: []}
for noise in (0, 5, 10, 20, 30):
    cells, mean_actions = [], 0
    for check in (False, True):
        tally = {"right": 0, "wrong booking": 0, "unfinished": 0}
        total = 0
        for seed in range(500):
            app, actions = run(App(), GOAL, noise=noise, seed=seed, check=check)
            tally[outcome(app)] += 1
            total += actions
        wrong[check].append(tally["wrong booking"])
        cells.append(
            (
                str(
                    tally["right"]
                ) + ", " + str(tally["wrong booking"]) + ", " + str(tally["unfinished"])
            ).ljust(13))
        mean_actions = round(total / 500, 1)
    print(
        (str(noise) + " pixels").ljust(9), "|", cells[0].ljust(59), "|",
        cells[1].ljust(44), "|", mean_actions)
plt.plot([0, 5, 10, 20, 30], wrong[False], marker="o", label="confirms at once")
plt.plot([0, 5, 10, 20, 30], wrong[True], marker="s", label="reads the summary first")
plt.xlabel("aim error (pixels)")
plt.ylabel("wrong bookings in 500 runs")
plt.title("Wrong bookings against aim error")
plt.legend()
plt.show()

With an error of 5 or 10 pixels the task nearly always succeeds: the targets are big enough, and a click that misses does nothing, so the agent simply looks and tries again.

At 20 pixels something worse than failure appears. Without reading the summary, 69 runs in 500 end with the wrong flight booked. The two Book buttons are 22 pixels tall and 40 apart, so a click aimed at one sometimes lands on the other. The application responds exactly as if that had been intended, and the agent confirms.

Reading the summary removes those: 500 right, none wrong, for about 13 actions instead of 7. A miss costs a retry. A hit on the wrong thing costs whatever that thing does, so the rule for these agents is to check the state of the world before any step that can't be undone.

When the screen is available as a list of elements, there is a better fix for aim: act on an element by its id and let the harness find its center. That removes the pixel error altogether, and it is the reason browser agents prefer the page's structure to its picture.

Quick check
Two Book buttons sit one above the other. A click is aimed at the center of the lower one, and lands a few pixels too high. What does this print: where it lands, for each error?
upper = {"id": "book-1", "x": 320, "y": 44, "w": 44, "h": 22}
lower = {"id": "book-2", "x": 320, "y": 84, "w": 44, "h": 22}
x, y = lower["x"] + lower["w"] // 2, lower["y"] + lower["h"] // 2

for error in (0, 10, 15, 40):
    lands = [e["id"] for e in (upper, lower) if e["y"] <= y - error < e["y"] + e["h"]]
    print(error, lands[0] if lands else "nothing", end="  ")
  1. 0 book-2 10 book-2 15 book-1 40 book-1
  2. 0 book-2 10 book-2 15 book-2 40 nothing
  3. 0 book-2 10 book-2 15 nothing 40 book-1
  4. 0 book-2 10 nothing 15 nothing 40 nothing
Show the answer

0 book-2 10 book-2 15 nothing 40 book-1

The lower button covers y from 84 to 105, and its center is at 95. Ten pixels high is 85, still inside it. Fifteen high is 80, in the gap between the buttons: a harmless miss. Forty high is 55, inside the upper button, which covers 44 to 65. The same kind of error gives a hit, a miss or the wrong flight, depending only on its size.

Quick check
In the run above, why is a click that lands on the wrong button worse than a click that misses everything?
  1. A missed click makes the whole application crash
  2. A wrong button always takes far longer to respond
  3. The application carries out the wrong action for real
  4. The agent can't tell that a missed click has missed
Show the answer

The application carries out the wrong action for real

A click on empty space changes nothing. The agent looks again, sees the same screen and retries. A click on the neighboring button is a valid action: the application moves on, with the wrong flight, and shows a perfectly normal next page. Unless the agent checks that page against its goal, it carries the error through to the end.

The screen can change under you

Looking before every action is slow, and it is tempting to save observations: read the form once, then send the whole sequence of clicks and keystrokes. Here is that shortcut, on a day when the application shows a marketing dialog partway through.

Uses the setup above, 140 lines.

print("A plan made from one screenshot, carried out without looking again:")
# a dialog will appear after the second click
app = App(popup_after=2)
first_look = {e["id"]: center(e) for e in app.elements()}
plan = [
    ("click", first_look["from"]), ("type", "Lisbon"), ("click", first_look["to"]),
    ("type", "Oslo"), ("click", first_look["search"])]
for kind, value in plan:
    print(
        "  ", kind, str(value).ljust(10), "->",
        app.click(*value) if kind == "click" else app.type(value))
print("   page:", app.page, "| subscribed to marketing:", app.subscribed)

print()
print("The same task, looking before every action:")
app, actions = run(App(popup_after=2), GOAL, log=True)
print("   result:", outcome(app), "| subscribed to marketing:", app.subscribed)

The plan was correct when it was made. Its fifth action was a click at (190, 115), where the Search button was. By then a dialog covered that spot, and its "Subscribe" button sat exactly there. The agent subscribed its user to marketing email and never searched.

The loop that looks before every action meets the same dialog, sees it, closes it and carries on: eight actions and the right flight.

An observation has a short life. A dialog, a slow page, a list that reorders, a notification sliding in: any of them makes the last screenshot a picture of a screen that no longer exists. Batching a few actions is reasonable only when they can't be affected, such as typing into a box that already has the focus. Anything that clicks should be preceded by a fresh look.

Drill

Is it still safe to click?

Write safe_to_click(seen, now, target). seen is the list of elements from the observation a plan was made from, and now is the list on screen at this moment. Both use the format of the last drill, with later elements on top. Return True only if all of these hold: an element with the id target is in both lists; its box (x, y, w, h) is the same in both; and a click at the center of that box, (x + w // 2, y + h // 2), would land on that element now, with nothing drawn over it.

Uses the setup above, 140 lines.

def safe_to_click(seen, now, target):
    return True


form = [{"id": "search", "x": 150, "y": 100, "w": 80, "h": 30}]
dialog = form + [{"id": "popup", "x": 100, "y": 30, "w": 220, "h": 140}]
print(safe_to_click(form, form, "search"), safe_to_click(form, dialog, "search"))
In the lesson, your code is checked as you write it.
Quick check
An agent has just typed a city into a text box and wants to save observations. Which of these is the safest thing to do without looking at the screen again?
  1. Type more text into the same box, which already has the focus
  2. Press Confirm, since the form has been filled in
  3. Click the first result, where the list will appear
  4. Click the Search button at the position it had earlier
Show the answer

Type more text into the same box, which already has the focus

More typing into a focused box depends on nothing that is likely to have moved. Every other option is a click at a remembered or expected position, on a screen that may have changed, and two of them act on things the agent has never seen at all.

In your own words
What makes an observation stale, and what should an agent do about it?

Write a sentence or two, then compare it with a good answer in the lesson.

In a production harness: the real computer tool, and cheaper routes

What it costs, and when to use it

Operating a screen is the most general way for an agent to act and nearly always the most expensive. Every step sends an observation, and a screenshot is large: on the order of a thousand tokens or more, where a tool result is often a few dozen. The history grows with every step, so the total grows faster than the number of steps.

That gives a clear order of preference.

  1. An API or a tool, when one exists. One call, a structured result, nothing to misclick.
  2. The page's structure: the elements and their ids, from the browser or the accessibility tree. Cheaper to read than an image, and actions can name elements instead of coordinates.
  3. Pixels: screenshots and coordinates. Works on anything with a screen, including applications that expose nothing else.

Real agents mix them: a tool where there is one, the element list where the application provides it, and a screenshot for the canvas, the chart or the old program that only a person could read before.

Worked example

Worked example: what the screenshots cost

Count the input tokens of a whole run under three ways of handling observations. The sizes are assumptions chosen to compare plans, not measurements: 1,500 tokens for a screenshot, 300 for the same screen as a list of elements, and 150 of text per step.

Uses the setup above, 140 lines.

def input_tokens(steps, screen_tokens, keep, text_tokens=150):
    """Input tokens for a run: each step resends the history. keep=0 keeps all."""
    total = 0
    for step in range(1, steps + 1):
        screens = min(step, keep) if keep else step
        total += screens * screen_tokens + step * text_tokens
    return total

# Assumed sizes: 1,500 tokens for a screenshot, 300 for the same screen as a list of
# elements, and 150 tokens of text for each step.
print("steps | all screenshots kept | last 3 screenshots | last 3, as elements")
for steps in (10, 30, 100):
    print(
        str(steps).rjust(5), "|", f"{input_tokens(steps, 1500, 0):,}".ljust(21), "|",
        f"{input_tokens(steps, 1500, 3):,}".ljust(23), "|",
        f"{input_tokens(steps, 300, 3):,}")
It runs right here in your browser, with nothing to install.

Every step re-sends the history, so keeping every screenshot makes the cost grow with the square of the number of steps: 90,750 tokens for 10 steps, and 8,332,500 for 100. Ten times the steps cost over ninety times as much. Keeping only the last three screenshots, and dropping older ones from the history, brings 100 steps down to 1,203,000: the agent rarely needs to see what the screen looked like twenty actions ago, and a line of text saying what it did is enough. Reading screens as element lists cuts it again, to 846,600. The numbers are assumed, and the shape isn't: long computer-use runs are affordable only if old observations are dropped.

In your own words
A colleague proposes a computer-use agent to move data between two internal systems. What do you ask first?

Write a sentence or two, then compare it with a good answer in the lesson.

Practice what you just read

  • 7 code examples you can edit and run
  • a practice exercise that checks your code
  • a 5-question quiz
  • 11 review cards for spaced repetition

Try a quiz question

What does a computer-use agent receive as its observation, and what does it send back as an action?

  1. The screen, as an image or element list; a click, keystroke or scroll
  2. A function's return value; a call to the next function
  3. The application's source code; an edit to that code
  4. A database row; a query for the next row

Answer it at the end of the lesson to see if you're right, and why.

Before this lesson

Keep learning

Where this lesson leads.

575 lessons in thirteen phases. 176 are free without an account, and a free account opens the rest. See every lesson.

Phase 10 · about 20 hours of study

Agent harnesses

Build the software around a model that turns it into a capable, safe agent 24 lessons 1 free
  1. What an agent harness is FreeThe software around a model that turns it into an agent.35 min
  2. The core loop, built properlyStop reasons, parallel tool calls, errors and turn budgets.40 min
  3. A tool registry for agentsSchemas from functions, validation, output limits and helpful errors.40 min
  4. File and search toolsRead, write, edit by exact replacement, glob and grep inside one folder.45 min
  5. Running code safelyExecution tools, step limits, output caps and real sandboxes.45 min
  6. Giving an agent a databaseSchema as context, a query tool with caps and readable errors, guards in layers, and execution accuracy.45 min
  7. Permissions and approvalsRules, modes and approval prompts that keep a person in control.40 min
  8. Hooks and skillsExtend a harness at lifecycle events, and load know-how on demand.40 min
  9. Multi-agent systemsCoordinating many agents on one large job.35 min
  10. Managing the context windowBudgets, clearing old tool results, compaction and a cacheable prefix.45 min
  11. Memory, plans and long tasksTo-do lists, progress notes, memory files and checkpoints.45 min
  12. SubagentsHand a focused task to a fresh context with fewer tools.40 min
  13. Streaming, interrupts and sessionsLive output, stopping safely, and saving and resuming transcripts.40 min
  14. Agents as graphsState, reducers and routing, a checkpoint after every step, and pausing for a person's approval.45 min
  15. Evaluation harnessesRun agents on tasks in fresh environments and grade what they leave behind.45 min
  16. Tracing, budgets and guardrailsSee what an agent did, cap what it spends, and stop it when it loops.40 min
  17. Attack your own harnessRun fourteen attacks against a harness with switches, read which layer stops each one, score attacks stopped against ordinary tasks done, and keep every finding as a release test.40 min
  18. Capstone: build a coding agentA harness that edits a project, runs its tests and stops when they pass.45 min
  19. The same agent on the real SDKsPut the parts you built beside the tool runner and the Agent SDK, choose a surface for a job, and take an MCP server's tools into your registry with prefixed names, one rule per server and a capped result.35 min
  20. Agents that use a computerAn agent with a screen, a mouse and a keyboard: the observe-decide-act loop, what a click lands on, why a wrong hit is worse than a miss, what happens when the screen changes under a plan, and what screenshots cost.45 min
  21. Sandboxes for agents that browseTreat every page as untrusted input: check hosts properly, gate actions by their effect, and layer the defenses.45 min
  22. Protocols between agentsHand a task to an agent someone else built: agent cards, the life of a task, the messages on the wire, safe retries, and whom to trust.45 min
  23. Tracing a run end to endFollow one run across an orchestrator, subagents and tool servers: trace ids, the traceparent header, the critical path, tokens by branch, and which traces to keep.45 min
  24. Retrieval over a graphAnswer questions that hop between documents: facts as triples, a walk out from the entities a question names, and the hubs and missed facts that limit it.45 min

Never trained an AI? Teach a small one to write, right in your browser. It takes about 20 seconds and needs no account. Teach it to write