Take a store photo of a sofa in a staged living room. A typical AI description narrates the whole room. The browser extension that Ananya Gubbi Mohanbabu, a UC Berkeley PhD student, built with her adviser Amy Pavel reads the surrounding page first. It sees that the sofa is what's for sale and identifies the fabric as chenille. For a blind shopper using a screen reader, the software that speaks the screen aloud, that one detail can decide the purchase. In her first-author 2024 study, 10 of 12 blind and low-vision participants preferred its description to one written without the page. Two wanted the room back, and those details could inform a purchase too.
The extension had already made room for those two. It kept the site's own image description and offered the version written without the page for comparison. The paper notes that errors drawn from context can sound especially believable. Deciding what matters on a page is a form of filtering, and here the unfiltered version stayed one step away.
Task Mode, another first-author paper, written with Yotam Sechayk and Pavel, applies that filter to an entire page. The user states a goal, and GPT-4o scores every element for relevance. At the strictest setting, low-scoring elements disappear from the screen and from the screen reader, and they stop responding to clicks. In a 12-person study on pages prepared in advance, the six participants who used screen readers finished tasks in about half the time they needed without the filter.
The paper is direct about the cost:
"This can lead to errors where useful elements are not scored as relevant."
To reach a button that was wrongly hidden, the person has to lower the threshold or go back to the full page. Participants did, trying several thresholds. Misses rarely stopped them, though some ran into text stranded without its heading. The paper reports choosing GPT-4o over a slightly more accurate model for speed and cost. That is an ordinary engineering trade, but accuracy decides how often someone has to go looking.
Relevant elements also stay where they sit in the page's structure instead of moving into a ranked list. This doesn't bring a hidden button back, since only the threshold does that. Once the button returns, though, the person finds it where the page always kept it, in an arrangement they already know.
A11y-CUA, where she shares first authorship with Rosiana Natalie, turns to agents, AI systems that click and type on your behalf. On 60 Windows tasks, Claude Sonnet 4.5 succeeded 78.3% of the time in a standard setup. Restricted to the keyboard, while still seeing screenshots, it succeeded 41.7% of the time, and at 150% zoom, 28.3%. Both harder settings came with twice the step budget, 100 steps against 50.
Screen reader users in the study waited for the speech and checked where they were before acting. The agent fired off long sequences of actions and, after pop-ups, sometimes typed into the wrong application. It was working at a pace the people it might hand control back to do not share. The authors conclude that blind and low-vision users "cannot easily collaborate" with such agents. Participants asked for agents that "narrate what they are doing in clear and screen-reader-friendly language," and for easy undo.
Read together, the systems suggest a condition: hiding things on someone's behalf is defensible when that person can get them back with tools and layouts they already know. Her own work meets that condition only partway. At Task Mode's strictest setting, the way back runs through Task Mode's own controls rather than the person's screen reader. A task success rate cannot register that difference, so a team optimizing against that rate has little reason to notice it. The A11y-CUA authors propose measuring recovery and checking that agents leave screen reader and zoom settings intact, though the paper did not test either proposal.
That suggests a cheap test, a version of asking what the next person can still do. Stop the agent halfway, then see whether the person, using only their own tools, can find where it stopped and finish the job.
-
A duty of honesty: The W3C Technical Architecture Group's new User Agents Group Note calls for meaningful visibility into sources, actions, and outcomes when users cannot readily observe them, though as a Group Note it carries no W3C endorsement.
-
Pairing agent and human needs: A W3C Strategy Team paper, W3C and the State of the Web in the AI Era, proposes that agents' access to the accessibility tree come paired with protection of screen-reader users' needs, offered for discussion rather than as implemented safeguards.
-
What the agents actually did: The A11y-CUA project page describes agents replaying long navigation sequences, cycling panes, and sometimes stopping after locating a file or skipping the final confirmation, alongside the human participants who checked focus before acting.
-
Correct steps, unusable results: The KNOWS benchmark paper found that failures in visual steps could leave a finished document or presentation unusable even when agents completed more than half of the other evaluation steps, on a deliberately demanding set of 110 tasks.
