In a recent benchmark called DRBench, browser-based agents produced enterprise research reports that read fluently and looked well-structured. They achieved 1.11% insight recall and 6.67% factuality. The reports were polished. They contained almost nothing.
That result sits at the end of a research trajectory led by Alexandre Drouin at ServiceNow's Frontier AI Research group, and it captures something the broader agent field has been slow to confront. Agent research generally treats measurement as a settled problem: did the agent get the right answer? For enterprise tasks, defining "right answer" turns out to be the hard problem, and the one that blocks real deployment.
Correctness Is a Different Object Every Time
WorkArena, published at ICML 2024, placed browser agents inside a live ServiceNow instance and asked them to do things knowledge workers actually do: fill out forms, configure service catalog orders, filter lists, navigate dashboards. What matters methodologically is how the benchmark checks answers. Form tasks query the database to verify a record was created with specific field values. Catalog tasks check that an order contains the right items and quantities. Knowledge-base tasks compare returned answers against acceptable responses. There's no single correctness metric because there's no single kind of task.
That design exposed something a simpler benchmark would miss. List-filtering tasks, described in the paper as "fairly simple for a human to complete," produced a 0% success rate across every model tested. The tasks required interacting with a non-standard HTML widget. An agent could describe the desired filter perfectly and still fail, because correctness lived in the UI state, not in the agent's narration.
WorkArena also revealed a subtler problem. Agents with thought-history enabled sometimes performed worse, committing to early mistakes instead of self-correcting. Memory became a liability when the benchmark measured end-state correctness, reinforcing early missteps rather than enabling recovery.
From Tasks to Workflows
WorkArena++, published at NeurIPS 2024, composed those atomic tasks into multi-step workflows requiring planning, reasoning, and retrieval. If correctness is already hard to define for a single enterprise action, composition makes it worse. Errors propagate through dependent steps, recovery paths matter, and the evaluation has to account for sequences rather than individual outcomes.
The BrowserGym ecosystem paper (TMLR 2025) turned the measurement lens on the measurement tools themselves. Web-agent benchmarks had proliferated with fragmented evaluation methods and reproducibility gaps from model version changes and live website drift. If the instruments aren't calibrated, the readings don't mean much. Without comparable measurement infrastructure, the field can't even track whether it's making progress on the problems WorkArena exposed.
Measuring Judgment
DRBench made the sharpest turn. Instead of measuring completed browser actions, it measures whether an agent produced a useful research report. One hundred tasks across ten domains ask agents to synthesize information from emails, chat logs, spreadsheets, slide decks, and the open web. The evaluation is insight-centered: ground-truth insights embedded alongside plausible distractors, with scoring based on source attribution, evidence recovery, and temporal precision. An agent that captured a numeric value but missed the qualifier "as of Q2 2024" scored as incorrect despite surface-level overlap.
Hence those fluent, empty reports. DRBench's evaluation demands evidence recovery, contextual binding, and resistance to plausible noise. Fluency alone produces polished documents that score near zero. The distance between a polished report and a useful one, measured precisely, turns out to be enormous.
The Specification Bottleneck
This research program lives inside an enterprise software company, which shapes the work in ways worth noting. The benchmarks use real enterprise environments with real UI complexity, real data models, and real organizational artifacts. Each expansion in scope made the definition of "correct" harder, and agent performance dropped accordingly.
The specification problem sits upstream of everything else. Agents can act. Whether the work itself has been defined precisely enough to know when it's been done right is a separate, harder question.
That specification doesn't exist for most enterprise tasks today. And until someone builds it, capability benchmarks will keep telling a more optimistic story than deployment will.
- Repeated-trial reliability gaps: tau-bench introduced pass@k as a measure of whether agents succeed consistently across attempts, finding that GPT-4o's retail performance dropped from ~61% on a single try to under 25% across eight, a distinction most capability reports don't surface.
- Security evaluation in BrowserGym: DoomArena extends the measurement program into adversarial territory, integrating with BrowserGym to test how agents handle malicious users and poisoned environments rather than just benign task completion.
- Verification as deployment blocker: A qualitative study of 16 practitioners across 12 companies found that several demonstrated higher-level agent capabilities experimentally but could not move them to production because adequate output verification was absent.
- Standardizing how agents get judged: AgentBeats proposes replacing fixed evaluation harnesses with judge agents that interact with subject agents through standardized protocols like A2A and MCP, treating assessment itself as an agent coordination problem.