Agents that use a computer
An agent with a screen, a mouse and a keyboard: the observe-decide-act loop, what a click lands on, why a wrong hit is worse than a miss, what happens when the screen changes under a plan, and what screenshots cost.
What it covers7
Why it matters
Most of the software people use has no API for an agent to call. It has a screen, a mouse and a keyboard. A computer-use agent works that way: it is shown the screen, decides on a click or some typing, and is shown the screen again. That lets a model operate almost anything a person can, and it brings problems that tool calls never had. A click can miss. It can land, convincingly, on the wrong thing. The screen can change between looking and acting. This lesson builds the loop on a small simulated application and measures each of those problems.
What you'll be able to do
- Describe the loop of an agent that operates a screen: observe, decide, act, observe again
- Turn a target on screen into a click, and work out what a click at a point lands on
- Explain why an agent must verify the result of its actions, and measure what happens when it doesn't
- Show why a plan made from one observation goes wrong when the screen changes
- Choose between an API, a structured view of the page, and raw pixels, and estimate what each costs
Show the answer
The screen changes partway, and later clicks land elsewhere
The plan is a list of screen positions that were right when the screenshot was taken. If a dialog appears, a page loads late or a list reorders, the same positions now hold different things, and the agent presses them anyway. Looking again before each action is the basic defense, and you will see below what happens without it.
The idea: look, decide, act, look again
Recall: The core loop, built properly ran a loop with tool results. A computer-use agent runs the same loop with the screen in place of the result, and Sandboxes for agents that browse shows what such an agent can reach.
A model with a mouse
An agent with tools calls functions: search_flights(from, to). A computer-use agent has one very general tool instead: the computer. Its actions are the ones a person has.
- Click at a position on the screen, given as x and y in pixels from the top left corner.
- Type some text, into whatever has the keyboard's focus.
- Press a key, scroll, or wait.
And its observation is the screen. Usually that is a screenshot, an image the model reads. Sometimes the harness can also supply the screen as a list of its elements: each button, text box and label, with its position. Browsers and operating systems keep such a list for screen readers, where it is called the accessibility tree.
The application below is a small flight-booking screen, simulated in Python. elements() returns what is on the screen, and hit(x, y) says what a click at a point would land on.
Setup (runs before every cell) · 140 lines
The lesson runs this once, before its first cell, and every cell, drill and example starts with what it defines. A cell that changes a setup object changes it for the cells after it, until the page reloads.
import random FLIGHTS = {"book-1": "09:10 direct $240", "book-2": "13:45 one stop $180"} class App: """A tiny flight-booking screen, 400 by 200: a stand-in for a real application.""" def __init__(self, popup_after=None): self.page, self.focus, self.clicks, self.booked = "search", None, 0, None self.values = {"from": "", "to": ""} self.popup_after, self.popup, self.subscribed = popup_after, False, False def box(self, id, role, label, x, y, w, h): return {"id": id, "role": role, "label": label, "x": x, "y": y, "w": w, "h": h} def elements(self): """The boxes on screen now. Later ones are drawn on top of earlier ones.""" if self.page == "search": out = [ self.box( "from", "textbox", "From: " + self.values["from"], 20, 40, 160, 30), self.box( "to", "textbox", "To: " + self.values["to"], 200, 40, 160, 30), self.box("search", "button", "Search", 150, 100, 80, 30)] elif self.page == "results": out = [ self.box("flight-1", "text", FLIGHTS["book-1"], 20, 40, 280, 30), self.box("book-1", "button", "Book", 320, 44, 44, 22), self.box( "flight-2", "text", FLIGHTS["book-2"], 20, 80, 280, 30 ), self.box("book-2", "button", "Book", 320, 84, 44, 22)] elif self.page == "confirm": out = [ self.box( "summary", "text", "Book " + self.booked + "?", 60, 60, 280, 30), self.box( "cancel", "button", "Cancel", 110, 120, 70, 30 ), self.box("confirm", "button", "Confirm", 210, 120, 80, 30)] else: out = [self.box("done", "text", "Booked: " + self.booked, 60, 60, 280, 30)] if self.popup: out += [ self.box("popup", "dialog", "Sign up for deals!", 100, 30, 220, 140), self.box( "popup-yes", "button", "Subscribe", 130, 110, 90, 30 ), self.box("popup-no", "button", "No thanks", 230, 110, 80, 30)] return out def hit(self, x, y): """The topmost element under a point, or None.""" found = None for e in self.elements(): if e["x"] <= x < e["x"] + e["w"] and e["y"] <= y < e["y"] + e["h"]: found = e return found def click(self, x, y): self.clicks += 1 target = self.hit(x, y) if self.clicks == self.popup_after: self.popup = True # a dialog slides in just after this click if target is None: self.focus = None return "clicked empty space" id = target["id"] if id == "popup": return "clicked the dialog, nothing happened" if id in ("popup-yes", "popup-no"): self.popup, self.subscribed = False, self.subscribed or id == "popup-yes" if id == "popup-yes": return "SUBSCRIBED to marketing email" return "closed the dialog" self.focus = id if target["role"] == "textbox" else None if id == "search" and self.values["from"] and self.values["to"]: self.page = "results" elif id in FLIGHTS: self.page, self.booked = "confirm", FLIGHTS[id] elif id == "confirm": self.page = "done" elif id == "cancel": self.page, self.booked = "results", None return "clicked " + id def type(self, text): if self.focus is None: return "typed, with nothing focused" self.values[self.focus] += text return "typed into " + self.focus def center(e): return e["x"] + e["w"] // 2, e["y"] + e["h"] // 2 def decide(elements, goal, check=True): """A stand-in for the model: rules where a real model would reason.""" on = {e["id"]: e for e in elements} if "popup-no" in on: return ("click", "popup-no") if "from" in on: for field in ("from", "to"): if on[field]["label"] != field.capitalize() + ": " + goal[field]: return ("fill", field) return ("click", "search") if "book-1" in on: return ("click", "book-2") # the goal asks for the cheapest flight if "confirm" in on: wrong = check and goal["flight"] not in on["summary"]["label"] return ("click", "cancel" if wrong else "confirm") return ("stop", None) def run(app, goal, noise=0.0, seed=0, check=True, max_actions=40, log=False): """Look, decide, act, and look again, until the model says stop.""" rng, actions = random.Random(seed), 0 while actions < max_actions: elements = app.elements() # 1. observe kind, target = decide(elements, goal, check) # 2. decide if kind == "stop": break x, y = center(next(e for e in elements if e["id"] == target)) x = round(x + rng.gauss(0, noise)) # the aim is a little off y = round(y + rng.gauss(0, noise)) result = app.click(x, y) # 3. act actions += 1 if log: print(str(actions).rjust(2), "click", str((x, y)).ljust(10), "aiming at", target.ljust(8), "->", result) if kind == "fill" and app.focus == target: result = app.type(goal[target]) actions += 1 if log: print(str(actions).rjust(2), "type ", repr(goal[target]).ljust(10), " " * 18, "->", result) return app, actions GOAL = {"from": "Lisbon", "to": "Oslo", "flight": "$180"} def outcome(app): if app.page != "done": return "unfinished" right = GOAL["flight"] in app.booked right = right and app.values == {"from": "Lisbon", "to": "Oslo"} return "right" if right else "wrong booking"
app = App() print("the screen, as a list of elements:") for e in app.elements(): print( " ", e["id"].ljust(7), e["role"].ljust(8), repr(e["label"]).ljust(10), "| left", str(e["x"]).rjust(3), "top", str(e["y"]).rjust(3), "| width", str(e["w"]).rjust(3), "height", e["h"]) print() for point in ((100, 55), (190, 115), (190, 90), (179, 69), (180, 70)): target = app.hit(*point) print( "a click at", str(point).ljust(10), "lands on:", target["id"] if target else "nothing")
Three elements, each a box. A click at (100, 55) lands inside the "From" box, and one at (190, 115) on the Search button. A click at (190, 90) is between them and lands on nothing.
The last two lines are a pixel apart. (179, 69) is the bottom right corner of the "From" box, and (180, 70) is just outside it. To the application, a click is a point, and a point is either inside a box or it isn't.
elements = [
{"id": "row", "x": 0, "y": 0, "w": 300, "h": 40},
# drawn later, so it is on top
{"id": "delete", "x": 250, "y": 5, "w": 40, "h": 30},
]
def hit(x, y):
found = None
for e in elements:
if e["x"] <= x < e["x"] + e["w"] and e["y"] <= y < e["y"] + e["h"]:
found = e["id"]
return found
print(hit(100, 20), hit(260, 20), hit(260, 37), hit(310, 20))Show the answer
row delete row None
At (100, 20) only the row is under the point. At (260, 20) both are, and the button, drawn later, is on top. At (260, 37) the point is below the button's bottom edge at 35, so it falls through to the row. (310, 20) is past the row's right edge. A slip of a few pixels near a Delete button changes what gets pressed.
What does a click land on?
Write hit(elements, x, y). Each element is a dictionary with id, x, y, w and h: its left edge, top edge, width and height. Elements later in the list are drawn on top of earlier ones. Return the id of the topmost element that contains the point, or None if there is none. A box contains its left and top edges and not its right and bottom ones.
Uses the setup above, 140 lines.
def hit(elements, x, y): return None screen = [ {"id": "row", "x": 0, "y": 0, "w": 300, "h": 40}, {"id": "delete", "x": 250, "y": 5, "w": 40, "h": 30}, ] print(hit(screen, 100, 20), hit(screen, 260, 20), hit(screen, 310, 20))
Look, decide, act, look again, until the model says it is finished or a limit is reached.
Write a sentence or two, then compare it with a good answer in the lesson.
In code: aim, verification and stale screens
Look, decide, act, look again
The harness for such an agent is the loop you know, with the screen in place of tool results.
- Observe. Capture the screen.
- Decide. The model reads it, with the goal and the history, and picks one action.
- Act. The harness performs that action.
- Go back to step 1, until the model says it is finished or a limit is reached.
In the code below, decide is a stand-in for the model: a few rules where a real model would reason about the screenshot. Everything else is the real loop. The goal is to book the cheapest flight from Lisbon to Oslo.
The loop is run, and decide is the stand-in model. Both sit in the lesson's setup above, so the cell below only starts the run. This is the loop in full:
def run(app, goal, noise=0.0, seed=0, check=True, max_actions=40, log=False): """Look, decide, act, and look again, until the model says stop.""" rng, actions = random.Random(seed), 0 while actions < max_actions: elements = app.elements() # 1. observe kind, target = decide(elements, goal, check) # 2. decide if kind == "stop": break x, y = center(next(e for e in elements if e["id"] == target)) # the model's aim is a little off x, y = round(x + rng.gauss(0, noise)), round(y + rng.gauss(0, noise)) result = app.click(x, y) # 3. act actions += 1 if log: print( str(actions).rjust(2), "click", str((x, y)).ljust(10), "aiming at", target.ljust(8), "->", result) if kind == "fill" and app.focus == target: result = app.type(goal[target]) actions += 1 if log: print( str(actions).rjust(2), "type ", repr(goal[target]).ljust(10), " " * 18, "->", result) return app, actions
Uses the setup above, 140 lines.
app, actions = run(App(), GOAL, log=True) print("result:", outcome(app), "|", app.elements()[0]["label"], "| actions:", actions)
Seven actions. Each click is aimed at the center of the element the "model" chose, and each line shows what the application did in response. Typing goes to whichever box was clicked last, since that box has the focus.
Notice what the loop does not do: it never assumes an action worked. After the click on "From", it looks at the screen again before typing. That costs an observation per action, and the next two sections show why it is worth it.
Show the answer
To stop a run that isn't making progress from going on forever
A turn limit is the last line of defense against a run that loops: an agent retrying a click that keeps missing, say, will otherwise spend tokens until someone notices. The computer-use loop above has the same guard, as a cap on actions.
500 clicks on the screen, colored by what they hit
The lower Book button is the target. Each click has an error of 20 pixels, and the dot's color says what it landed on.
Uses the setup above, 140 lines.
import matplotlib.pyplot as plt from matplotlib.patches import Rectangle app = App() app.page = "results" boxes = app.elements() cx, cy = center(next(e for e in boxes if e["id"] == "book-2")) rng = random.Random(1) colors = {"book-2": "tab:green", "book-1": "tab:red", None: "tab:gray"} counts = {"book-2": 0, "book-1": 0, None: 0} fig, ax = plt.subplots(figsize=(6, 3.4)) for e in boxes: ax.add_patch(Rectangle((e["x"], e["y"]), e["w"], e["h"], fill=False)) for _ in range(500): x, y = cx + rng.gauss(0, 20), cy + rng.gauss(0, 20) hit = app.hit(x, y) key = hit["id"] if hit and hit["id"] in counts else None counts[key] += 1 ax.plot(x, y, ".", color=colors[key], markersize=3) ax.set_xlim(250, 420) ax.set_ylim(130, 20) ax.set_title("500 clicks aimed at the lower Book button, error 20 pixels") plt.show() print(counts)
Aim is imperfect
A model reading a screenshot has to turn "the Book button" into two numbers. It is good at that and not exact: its coordinates can be off by some pixels, more on small or crowded targets.
The cell adds random error to every click, and runs the task 500 times at each level. It also compares two versions of the agent. One confirms a booking as soon as the confirmation page appears. The other reads the summary on that page first, and cancels if it isn't the flight it meant to book.
Uses the setup above, 140 lines.
print("aim error | no check: right, wrong, unfinished |") print(" | check: right, wrong, unfinished | actions") import matplotlib.pyplot as plt wrong = {False: [], True: []} for noise in (0, 5, 10, 20, 30): cells, mean_actions = [], 0 for check in (False, True): tally = {"right": 0, "wrong booking": 0, "unfinished": 0} total = 0 for seed in range(500): app, actions = run(App(), GOAL, noise=noise, seed=seed, check=check) tally[outcome(app)] += 1 total += actions wrong[check].append(tally["wrong booking"]) cells.append( ( str( tally["right"] ) + ", " + str(tally["wrong booking"]) + ", " + str(tally["unfinished"]) ).ljust(13)) mean_actions = round(total / 500, 1) print( (str(noise) + " pixels").ljust(9), "|", cells[0].ljust(59), "|", cells[1].ljust(44), "|", mean_actions) plt.plot([0, 5, 10, 20, 30], wrong[False], marker="o", label="confirms at once") plt.plot([0, 5, 10, 20, 30], wrong[True], marker="s", label="reads the summary first") plt.xlabel("aim error (pixels)") plt.ylabel("wrong bookings in 500 runs") plt.title("Wrong bookings against aim error") plt.legend() plt.show()
With an error of 5 or 10 pixels the task nearly always succeeds: the targets are big enough, and a click that misses does nothing, so the agent simply looks and tries again.
At 20 pixels something worse than failure appears. Without reading the summary, 69 runs in 500 end with the wrong flight booked. The two Book buttons are 22 pixels tall and 40 apart, so a click aimed at one sometimes lands on the other. The application responds exactly as if that had been intended, and the agent confirms.
Reading the summary removes those: 500 right, none wrong, for about 13 actions instead of 7. A miss costs a retry. A hit on the wrong thing costs whatever that thing does, so the rule for these agents is to check the state of the world before any step that can't be undone.
When the screen is available as a list of elements, there is a better fix for aim: act on an element by its id and let the harness find its center. That removes the pixel error altogether, and it is the reason browser agents prefer the page's structure to its picture.
upper = {"id": "book-1", "x": 320, "y": 44, "w": 44, "h": 22}
lower = {"id": "book-2", "x": 320, "y": 84, "w": 44, "h": 22}
x, y = lower["x"] + lower["w"] // 2, lower["y"] + lower["h"] // 2
for error in (0, 10, 15, 40):
lands = [e["id"] for e in (upper, lower) if e["y"] <= y - error < e["y"] + e["h"]]
print(error, lands[0] if lands else "nothing", end=" ")Show the answer
0 book-2 10 book-2 15 nothing 40 book-1
The lower button covers y from 84 to 105, and its center is at 95. Ten pixels high is 85, still inside it. Fifteen high is 80, in the gap between the buttons: a harmless miss. Forty high is 55, inside the upper button, which covers 44 to 65. The same kind of error gives a hit, a miss or the wrong flight, depending only on its size.
Show the answer
The application carries out the wrong action for real
A click on empty space changes nothing. The agent looks again, sees the same screen and retries. A click on the neighboring button is a valid action: the application moves on, with the wrong flight, and shows a perfectly normal next page. Unless the agent checks that page against its goal, it carries the error through to the end.
The screen can change under you
Looking before every action is slow, and it is tempting to save observations: read the form once, then send the whole sequence of clicks and keystrokes. Here is that shortcut, on a day when the application shows a marketing dialog partway through.
Uses the setup above, 140 lines.
print("A plan made from one screenshot, carried out without looking again:") # a dialog will appear after the second click app = App(popup_after=2) first_look = {e["id"]: center(e) for e in app.elements()} plan = [ ("click", first_look["from"]), ("type", "Lisbon"), ("click", first_look["to"]), ("type", "Oslo"), ("click", first_look["search"])] for kind, value in plan: print( " ", kind, str(value).ljust(10), "->", app.click(*value) if kind == "click" else app.type(value)) print(" page:", app.page, "| subscribed to marketing:", app.subscribed) print() print("The same task, looking before every action:") app, actions = run(App(popup_after=2), GOAL, log=True) print(" result:", outcome(app), "| subscribed to marketing:", app.subscribed)
The plan was correct when it was made. Its fifth action was a click at (190, 115), where the Search button was. By then a dialog covered that spot, and its "Subscribe" button sat exactly there. The agent subscribed its user to marketing email and never searched.
The loop that looks before every action meets the same dialog, sees it, closes it and carries on: eight actions and the right flight.
An observation has a short life. A dialog, a slow page, a list that reorders, a notification sliding in: any of them makes the last screenshot a picture of a screen that no longer exists. Batching a few actions is reasonable only when they can't be affected, such as typing into a box that already has the focus. Anything that clicks should be preceded by a fresh look.
Is it still safe to click?
Write safe_to_click(seen, now, target). seen is the list of elements from the observation a plan was made from, and now is the list on screen at this moment. Both use the format of the last drill, with later elements on top. Return True only if all of these hold: an element with the id target is in both lists; its box (x, y, w, h) is the same in both; and a click at the center of that box, (x + w // 2, y + h // 2), would land on that element now, with nothing drawn over it.
Uses the setup above, 140 lines.
def safe_to_click(seen, now, target): return True form = [{"id": "search", "x": 150, "y": 100, "w": 80, "h": 30}] dialog = form + [{"id": "popup", "x": 100, "y": 30, "w": 220, "h": 140}] print(safe_to_click(form, form, "search"), safe_to_click(form, dialog, "search"))
Show the answer
Type more text into the same box, which already has the focus
More typing into a focused box depends on nothing that is likely to have moved. Every other option is a click at a remembered or expected position, on a screen that may have changed, and two of them act on things the agent has never seen at all.
Write a sentence or two, then compare it with a good answer in the lesson.
In a production harness: the real computer tool, and cheaper routes
What it costs, and when to use it
Operating a screen is the most general way for an agent to act and nearly always the most expensive. Every step sends an observation, and a screenshot is large: on the order of a thousand tokens or more, where a tool result is often a few dozen. The history grows with every step, so the total grows faster than the number of steps.
That gives a clear order of preference.
- An API or a tool, when one exists. One call, a structured result, nothing to misclick.
- The page's structure: the elements and their ids, from the browser or the accessibility tree. Cheaper to read than an image, and actions can name elements instead of coordinates.
- Pixels: screenshots and coordinates. Works on anything with a screen, including applications that expose nothing else.
Real agents mix them: a tool where there is one, the element list where the application provides it, and a screenshot for the canvas, the chart or the old program that only a person could read before.
Worked example: what the screenshots cost
Count the input tokens of a whole run under three ways of handling observations. The sizes are assumptions chosen to compare plans, not measurements: 1,500 tokens for a screenshot, 300 for the same screen as a list of elements, and 150 of text per step.
Uses the setup above, 140 lines.
def input_tokens(steps, screen_tokens, keep, text_tokens=150): """Input tokens for a run: each step resends the history. keep=0 keeps all.""" total = 0 for step in range(1, steps + 1): screens = min(step, keep) if keep else step total += screens * screen_tokens + step * text_tokens return total # Assumed sizes: 1,500 tokens for a screenshot, 300 for the same screen as a list of # elements, and 150 tokens of text for each step. print("steps | all screenshots kept | last 3 screenshots | last 3, as elements") for steps in (10, 30, 100): print( str(steps).rjust(5), "|", f"{input_tokens(steps, 1500, 0):,}".ljust(21), "|", f"{input_tokens(steps, 1500, 3):,}".ljust(23), "|", f"{input_tokens(steps, 300, 3):,}")
Every step re-sends the history, so keeping every screenshot makes the cost grow with the square of the number of steps: 90,750 tokens for 10 steps, and 8,332,500 for 100. Ten times the steps cost over ninety times as much. Keeping only the last three screenshots, and dropping older ones from the history, brings 100 steps down to 1,203,000: the agent rarely needs to see what the screen looked like twenty actions ago, and a line of text saying what it did is enough. Reading screens as element lists cuts it again, to 846,600. The numbers are assumed, and the shape isn't: long computer-use runs are affordable only if old observations are dropped.
Write a sentence or two, then compare it with a good answer in the lesson.
Practice what you just read
- 7 code examples you can edit and run
- a practice exercise that checks your code
- a 5-question quiz
- 11 review cards for spaced repetition
Try a quiz question
What does a computer-use agent receive as its observation, and what does it send back as an action?
Before this lesson
Where this lesson leads.
575 lessons in thirteen phases. 176 are free without an account, and a free account opens the rest. See every lesson.
Phase 10 · about 20 hours of study
Agent harnesses
Build the software around a model that turns it into a capable, safe agent
24 lessons 1 free
- What an agent harness is FreeThe software around a model that turns it into an agent.35 min
- The core loop, built properlyStop reasons, parallel tool calls, errors and turn budgets.40 min
- A tool registry for agentsSchemas from functions, validation, output limits and helpful errors.40 min
- File and search toolsRead, write, edit by exact replacement, glob and grep inside one folder.45 min
- Running code safelyExecution tools, step limits, output caps and real sandboxes.45 min
- Giving an agent a databaseSchema as context, a query tool with caps and readable errors, guards in layers, and execution accuracy.45 min
- Permissions and approvalsRules, modes and approval prompts that keep a person in control.40 min
- Hooks and skillsExtend a harness at lifecycle events, and load know-how on demand.40 min
- Multi-agent systemsCoordinating many agents on one large job.35 min
- Managing the context windowBudgets, clearing old tool results, compaction and a cacheable prefix.45 min
- Memory, plans and long tasksTo-do lists, progress notes, memory files and checkpoints.45 min
- SubagentsHand a focused task to a fresh context with fewer tools.40 min
- Streaming, interrupts and sessionsLive output, stopping safely, and saving and resuming transcripts.40 min
- Agents as graphsState, reducers and routing, a checkpoint after every step, and pausing for a person's approval.45 min
- Evaluation harnessesRun agents on tasks in fresh environments and grade what they leave behind.45 min
- Tracing, budgets and guardrailsSee what an agent did, cap what it spends, and stop it when it loops.40 min
- Attack your own harnessRun fourteen attacks against a harness with switches, read which layer stops each one, score attacks stopped against ordinary tasks done, and keep every finding as a release test.40 min
- Capstone: build a coding agentA harness that edits a project, runs its tests and stops when they pass.45 min
- The same agent on the real SDKsPut the parts you built beside the tool runner and the Agent SDK, choose a surface for a job, and take an MCP server's tools into your registry with prefixed names, one rule per server and a capped result.35 min
- Agents that use a computerAn agent with a screen, a mouse and a keyboard: the observe-decide-act loop, what a click lands on, why a wrong hit is worse than a miss, what happens when the screen changes under a plan, and what screenshots cost.45 min
- Sandboxes for agents that browseTreat every page as untrusted input: check hosts properly, gate actions by their effect, and layer the defenses.45 min
- Protocols between agentsHand a task to an agent someone else built: agent cards, the life of a task, the messages on the wire, safe retries, and whom to trust.45 min
- Tracing a run end to endFollow one run across an orchestrator, subagents and tool servers: trace ids, the traceparent header, the critical path, tokens by branch, and which traces to keep.45 min
- Retrieval over a graphAnswer questions that hop between documents: facts as triples, a walk out from the entities a question names, and the hubs and missed facts that limit it.45 min
Never trained an AI? Teach a small one to write, right in your browser. It takes about 20 seconds and needs no account. Teach it to write