Alexandre Drouin and the Benchmark as a Theory of Work

Alexandre Drouin leads frontier AI research at ServiceNow and holds an adjunct appointment at Université Laval. WorkArena, the benchmark he first-authored for ICML 2024, tests AI agents on enterprise tasks inside a live, cloud-hosted ServiceNow instance — the software that IT departments actually use to track incidents, manage assets, and fulfill service requests. The tabs, the autocomplete fields, the reference lookups are all present, because it is the product rather than a model of the product. In a Berkeley lecture, Drouin described the trade plainly: he wanted agents measured against software people work in, and accepted the latency and flakiness of a live cloud environment as the price of that.

The choice also narrowed what could go in. The benchmark contains 33 task types across six categories: lists, forms, knowledge bases, service catalogs, menus, and dashboards. An agent might filter an incident list by priority and assignment group, create a hardware asset record, order a configured laptop, or read a value off a chart.

Every one of those tasks ends somewhere a program can look. A form task writes a database record, and the validator retrieves it and compares the stored values against the specification. A list task changes a query, and the validator reads the query's logical structure. Order something from the catalog and the validator inspects the resulting request field by field: items, quantities, configuration options. Even the thinnest tasks, navigating a menu or switching user sessions, resolve into a URL or a session state that can be read programmatically.

In the same lecture, Drouin distinguished his approach from benchmarks that grade an agent by comparing its clicks against a recorded human sequence. WorkArena checks the end state instead: did the system wind up reflecting the right outcome? That permits alternative solution paths, which is the point. It also requires that the right outcome be expressible as a condition a program can verify.

The compositional extension, WorkArena++, chains those atomic tasks into 341 multi-step workflows in which a value produced at one step becomes the input to the next. The paper is candid about coverage: thousands more tasks would be needed, and work involving software outside ServiceNow is absent. But the binding constraint is narrower than either of those. If a task doesn't terminate in a programmatically verifiable state, there is no mechanism by which it could be included at all.

Nothing in the papers, the repository, or the public talks documents what fell outside that line. There is no worker study, no list of candidate tasks considered and set aside. But the examples are not hard to supply. A senior IT analyst works the same incident queue the benchmark tests agents on, and her most valuable move in a given week may be noticing that three tickets filed by three different departments describe one underlying infrastructure problem. That recognition doesn't land in a field anything can check. A service desk worker takes a request that maps cleanly to a catalog item and belongs to a user who has misdiagnosed her own need; the work is the conversation that sorts it out, and the order that follows is the residue.

Drouin does not claim to have captured knowledge work, and it would be unfair to fault a benchmark for boundaries its authors state openly. Still, when an agent scores well on WorkArena, what the score reports is competence at tasks whose success can be established by inspecting system state. His design choices make that boundary unusually legible, which is more than most evaluation work offers. What sits on the other side of it — interpretive, relational, dependent on judgment that never resolves into a database field — goes unmeasured, and unmeasured work has a way of dropping out of the conversation about what these systems can do.