--- title: "Alexandre Drouin and the Tightening Definition of \"Done\"" canonical: "https://current.tinyfish.ai/issue/38/practitioners-corner/article/31167/alexandre-drouin-and-the-tightening-definition-of-done" publication: "TinyFish" publisher: "TinyFish" language: "en-US" guide: "https://current.tinyfish.ai/llms.txt" format: "Builder Profile" author: "Sevda Polat" author_url: "https://current.tinyfish.ai/team/200" issue: 38 section: "practitioners-corner" section_name: "Practitioner's Corner" published: 2026-07-15 updated: 2026-07-17 description: "Drouin's projects trace how agent evaluation evolved from checking task completion to verifying that outcomes are grounded, cited, and structurally real. The progression reveals a field learning that the hardest problem is not capability but verifiability." word_count: 937 source_links: 9 --- # Alexandre Drouin and the Tightening Definition of "Done" *By [Sevda Polat]()* *July 15, 2026* [Read on the website]() Every system reports success. The interesting part is what you find when you build the infrastructure to independently confirm it. [Alexandre Drouin](https://www.servicenow.com/research/author/alexandre-drouin.html), Head of Frontier AI Research at ServiceNow, has coauthored a sequence of publicly documented projects over the past three years that, taken together, amount to a progressively more demanding answer to a single question: how do you know an agent actually did what it claims? Each project makes a specific design choice about what counts as verification. And each one reveals, by implication, that the previous project's definition left something dangerously unchecked. ## The Database as Witness [WorkArena](https://arxiv.org/html/2403.07718), published at ICML 2024 with Drouin as first author, evaluates browser agents performing enterprise tasks on ServiceNow's platform. The critical design decision is easy to miss if you're reading for results rather than methodology: validators query the underlying database to confirm whether the right values landed in the right fields. For service-catalog tasks, the system checks whether the order contains the correct items and quantities. For knowledge lookups, it checks whether the agent's answer falls within an acceptable set, accounting for equivalent formulations like "8.5/10" and "85%." This sounds like plumbing. It is actually a refusal. The evaluation does not ask the agent what it did. It goes around the agent entirely and checks the ground truth independently. Anyone who has worked inside a system where self-reported metrics determined funding knows why this matters. For atomic tasks with well-defined end states, independent verification works cleanly. But enterprise work is rarely atomic, and end states are rarely well-defined. The limitation is built into the achievement. ## Making Observation Itself Reliable [BrowserGym](https://www.servicenow.com/research/publication/thibault-le-sellier-de-chezelles-the-tmlr2025.html), a TMLR 2025 paper on which Drouin is a coauthor, addresses a problem that sits upstream of any individual benchmark, in the infrastructure layer where nobody looks until something breaks. Different evaluation frameworks were using inconsistent observation and action spaces. The consequence was not merely that comparing agent performance across benchmarks was unreliable. Verification itself became unstable: if two frameworks represent the same agent behavior differently, you cannot even establish what happened, let alone whether it was correct. The measurement-standards analogy is precise here. You cannot build reliable instruments if every laboratory defines a meter differently. BrowserGym standardizes what agents can see (DOM snapshots, accessibility trees, screenshots, action errors) and what they can do (from high-level primitives like `click(bid)` to arbitrary code execution). Every benchmark task must implement `setup()` and `validate()`, exposing the full behavioral trace. The contribution is not new tasks but the shared observational surface without which no task's evaluation can be reproduced or contested. Infrastructure work. The kind that makes everything else possible and that nobody funds until the absence becomes catastrophic. ## Verifying the Path [WorkArena++](https://arxiv.org/abs/2407.05291), with Drouin again as coauthor, composes atomic tasks into 682 realistic workflows and introduces sequential subtask validation. The validator tracks intermediate checkpoints, confirming each validated subtask before granting overall success. An agent that arrives at the right answer through the wrong sequence of steps, or that skips critical intermediate states, fails. Completion now requires verifying the path, not just the destination. This is the difference between checking that a building stands and checking that it was built to code. ## When "Correct" Becomes Contested Then comes [DRBench](https://arxiv.org/html/2510.00172), coauthored by Drouin and published in 2026, and the ground shifts. The benchmark evaluates agents on enterprise deep research tasks: an agent must gather public web data and local enterprise documents, then produce a report. But DRBench does not evaluate the report as a report. It decomposes it into atomic insights and checks each one against its cited source. An insight without a citation is marked unfactual. An insight with a citation gets verified: the system retrieves the top relevant chunks from the cited document and asks whether the evidence actually supports the claim. A structural refusal to let fluency substitute for accuracy. Anyone who has watched a large language model produce beautifully written nonsense understands the stakes. Then there are the distractors. DRBench seeds each task with plausible but irrelevant insights and tracks whether the agent includes them. Evaluation must account not only for what the agent missed but for what it included that looked right and wasn't. The hardest verification problem, it turns out, is not the absent answer. It is the confident, well-sourced, wrong one. We have known this about human institutions for a long time. We are only now building the evaluation infrastructure to catch it in automated systems. That Drouin's publicly documented arc spans this entire trajectory tells you where the field's center of gravity is moving. Evaluation that trusts the agent's own report of success is not evaluation. It is institutional faith dressed up as measurement, and someone always pays when the ground begins to shake. ## Things to follow up on... - **Task success vs. safety:** A joint evaluation by the Singapore and Korea AI Safety Institutes found that [tool-using agents can complete tasks correctly while mishandling data along the way](https://arxiv.org/abs/2606.17114), separating "did it work" from "was it safe" across five risk categories. - **Simulated workplace benchmarks:** TheAgentCompany builds a [full simulated software company where agents browse, code, and communicate with coworkers](https://arxiv.org/abs/2412.14161), and its most competitive agent completed only 24–30% of tasks autonomously depending on version. - **Observability conventions taking shape:** OpenTelemetry's GenAI semantic conventions now cover model calls, agent IDs, and tool invocations in a [dedicated repository](https://github.com/open-telemetry/semantic-conventions-genai), though they do not yet capture delegated authority or policy compliance. - **When containment isn't resolution:** A Taobao field experiment found that agentic AI reduced chat duration but [substantially lowered customer ratings](https://arxiv.org/abs/2605.14830), with human intervention helping on technical escalations but not emotional ones. ## More - [Practitioner's Corner — Issue 38]() - [Issue 38]() - [Publication guide]()