Variance Is the Constraint

For decades, U.S. states reported average highway travel times. Two routes across New Jersey might average 58.6 and 59.5 minutes, a difference no driver would ever notice. But the second route's travel time swung nearly twice as much from one day to the next. A commuter reading the averages would conclude the routes were interchangeable. A dispatcher running a delivery fleet, who has to promise customers a window, would read the variance and conclude the opposite. Anyone who drove home from the Columbus Day weekend on Monday experienced one draw from that distribution, and it was rarely the average. In 2017, federal rules began requiring states to report travel-time reliability alongside the mean.

Web agents have the same measurement problem, and nobody has written the equivalent rule for them yet.

A recent study ran the same agents on the same shopping tasks three times each. Of those tasks, 46.5% succeeded in at least one run, and only 27.8% succeeded in all three. That gap of nearly nineteen percentage points is the distance between what an agent can do on a good pass and what it will do on the day you need it. On one model's Reddit tasks, the same agent scored 47% in one run and 42% in the next. Both numbers are accurate, and neither tells you much about tomorrow.

Nobody should expect a patch to close that gap. These are probabilistic systems working inside web environments that shift underneath them: the model makes slightly different choices each run, pages load at different speeds, server responses vary. Small differences compound across steps. That compounding shapes how production systems actually get built.

A study of 86 deployed agent systems found that 68% executed ten steps or fewer before a human intervened, and that developers named reliability, rather than capability, as their leading challenge. As Sable Whitford observed in an earlier piece here, those short runs may be sized to what a human reviewer can catch, not to what the agent could theoretically handle. The variance data explains why. Each additional step is another place where this run can drift away from the last one. Keeping tasks short keeps the range of possible outcomes narrow enough for a person to check the result.

Seen this way, the cautious habits of production teams make a good deal of sense. Short runs narrow the spread, human checkpoints catch what falls outside it, and modest task scope keeps the riskiest work away from the agent altogether. An outsider might read these choices as signs of an immature field waiting for better models. I read them as teams responding rationally to a fundamental property of the system they are working with.

The field's benchmarks, meanwhile, report one success rate per agent. The public leaderboards for WebArena and VisualWebArena show a single number, with nothing about how much it moved across runs or how many runs produced it. A steady agent and an erratic one with the same average sit side by side looking identical, much as those two New Jersey routes did until someone measured the spread.

The variance study had one more finding that deserves attention. Architectural add-ons such as memory layers or saved skills sometimes raised the average score without reliably reducing the spread, and some made it wider. Structurally, this is not surprising. Each new module gives the agent more possible paths through a task, and more paths can diverge as easily as they converge. The best runs get better while the worst runs may stay where they were. A higher-scoring agent is not necessarily a more predictable one, and for any system where someone downstream depends on the result, predictability may matter more than peak performance.

Highway engineers eventually learned that the average is not what people experience, and that reporting it alone had steered investment toward the wrong problems. Agent benchmarks that report capability without reliability risk repeating that mistake, widening the distance between what gets announced and what gets deployed. That distance tends to be discovered later, in production, by the people who built on the announced number.