A vision model looking at a screenshot of a web page, trying to work out which parts of it are interactive, is doing the job the screen reader heuristics were built for: recovering meaning that an interface displays but never declares. The machinery is unrecognizably different, but the gap it works around has not changed.
SeeClick, a model trained specifically to match a plain-language instruction to the right spot on a screen, reached 53.4% accuracy across mobile, desktop and web screenshots. The split underneath that number is more informative. On mobile interfaces the model located text targets 78% of the time and icons or widgets 52% of the time. Text can be read. An icon can only be interpreted, and only if you have seen enough gear-shaped things open enough settings panels to generalize from them. Like the rules before it, the model did best where meaning sat closest to the surface.
A benchmark called OSWorld asked something harder: not whether a model could find one element, but whether it could finish real tasks in desktop and web environments. File operations, multi-step workflows, sequences spanning several applications. Given only screenshots, one vision model completed 5.26% of the tasks. The researchers then added a single input: the accessibility tree, the structured description a browser assembles from semantic markup and ARIA roles, in effect the list of what a page says its own parts are. Same model, same tasks. Completion rose to 12.17%. Low by any standard, but the declared structure, patchy as it is in the wild, more than doubled the model's success rate.
Pixels show what an interface looks like and say nothing about what its parts are for. The tree helps because it supplies exactly that, which is also what the hand-written rules were guessing at.
The failure modes have changed character, though, and that part matters for anyone building on top of interface interpretation. A rule fails in a known direction. WebKit's engineers could describe exactly when their clickability inference would produce false positives and why. Firefox's table classifier has conditions you can enumerate. You can open the rule and read it.
When a vision model misidentifies an element there is no rule to open. A 2026 diagnostic study made this concrete: researchers generated multiple valid descriptions of the same interface element and found that models could locate it under one phrasing and miss it entirely under another. The model had not failed to learn the element. It had learned something about the wording that did not generalize. If you are deploying one of these systems, that has a specific operational consequence. A model can pass every case in your test suite and then miss the same element under a differently worded instruction, and its sensitivity to phrasing is not recoverable from its accuracy on any particular prompt.
A broken rule can be read and patched. An opaque inference has to be retrained, which means first noticing the failure, then assembling enough examples to move the model's behavior. The older heuristics failed where someone could see it happen. Learned inference tends to fail plausibly, producing results that look fine until something downstream breaks.
Through two generations of compensation the underlying absence has not moved. If it keeps not moving, the reason is probably not technical: interpretation that works most of the time is interpretation nobody has to account for, and a failure that cannot be located tends not to get assigned to anyone.
