There's a figure in WebArena, a widely used benchmark for web-browsing agents, that shows the same web page at the same moment three different ways: as a screenshot, as the HTML DOM, and as an accessibility tree. Hand an agent one of those and it's working from a different page than an agent handed either of the others.
Monitoring tools that keep only summaries (averages, mostly) and throw the raw events away have a familiar weakness. You can answer whatever question somebody thought of when they built the dashboard. Anything else, tough luck. Picking how an agent sees a page works the same way. Before the agent takes a single step, somebody has already decided which parts of the page it's allowed to ask about.
The screenshot shows what rendered
Pixels are the only format that actually knows what an image shows. They're also cheaper than you'd probably guess. Models read input in chunks called tokens, and tokens set both the bill and how much fits in one request. One provider's vision docs put a 1920×1080 image at 2,691 tokens.
The catch is that a screenshot only holds what's on screen. One VisualWebArena agent gave up on a purchase while the Add to Cart button sat just below the visible area. Pixels get misread, too. WebVoyager's agent mixed up calendar dates with the numbered tags drawn over the page to help it click, and those tags are generated from the page's code. So the "pixel" agent was quietly leaning on another format the whole time.
The accessibility tree repeats what the page claims
Browsers build a stripped-down outline of every page for screen readers, listing each control by its role and name. WebArena feeds that outline to agents like this:
[164] textbox 'Search' focused: True required: False
[171] button 'Go'It's small (in the test below, the model's entire input with the tree came to about 6,700 tokens), and it's numbered, so the agent can just say "click 171."
The problem is that every line is the page making a claim about itself. The W3C's own guidance says so plainly:
"A role is a promise."
Nobody enforces that promise. Something labeled a button doesn't have to behave like one. And some content never makes it into the tree at all, like graphics painted onto a canvas element.
The DOM has nearly everything, at a price
The DOM is the browser's full, live model of the page, and it is enormous. WorkArena measured business-software pages at 40,000 to 500,000 tokens, and that's after stripping out scripts and styles. Half a million tokens to read one damn page.
It still misses things. A text box's value attribute holds its starting value, which may not be what the user has typed since. An element can sit right there in the DOM while something else covers it, so finding it doesn't mean a click will reach it.
Sometimes all that bulk earns its cost, though. An April preprint, not yet peer-reviewed, ran WorkArena tasks with one model reading either HTML (around 56,700 tokens of total input) or the tree. HTML succeeded 73.3% of the time; the tree, 55.8%. The authors tie part of that gap to layout information the markup keeps and the tree throws away.
No raw page to fall back on
Which format wins also depends on which model is reading it. In OSWorld, a desktop benchmark that includes browser tasks, adding the tree to screenshots more than doubled GPT-4V's success rate, from 5.26% to 12.17%. For Gemini-ProV it went the other way, from 5.80% down to 3.48%.
This is where the monitoring analogy stops holding up. With telemetry, you keep the raw events and figure out your questions later. A page doesn't give an agent anything raw to keep, not even the DOM. Every format is already a summary, so what matters is knowing which questions yours can't answer.
When an agent screws up, start with what it was shown. Did it trust a label the code never honored? Miss a button it would have had to scroll to? Read a field value that went stale once the user typed?
-
Pixels versus markup tokens: SeeAct's Table 1 puts average HTML at 128,827 tokens against 4,240 visual tokens on the same Multimodal-Mind2Web pages, though the two columns use different tokenizers and aren't equivalent units of information.
-
How browsers guess clickability: A W3C HTML-AAM implementation discussion shows browsers inferring clickable elements from event handlers, and the ambiguity this creates when a handler sits on an ancestor element, so an unlabeled clickable div doesn't predictably vanish from the tree.
-
Hidden metadata as attack: An unreviewed August preprint on Android accessibility and prompt injection planted malicious instructions in an accessibility node that never rendered on screen, and reports that the agents were compromised.
-
Pruning HTML before reading: Mind2Web's preprocessing cut pages from an average of 1,135 elements to 580 while keeping the target element 94.7% of the time on training data, which shows the shrinking happening before the model sees anything.