The Number You Didn't Ask About

An agent succeeds on a customer service task 70% of the time. Report the chance it gets at least one success in three tries, and you can write 97%. Report the chance it handles three consecutive requests without failing, and you write 34%. Same agent. Same math. A 63-point gap created entirely by which question you decided to answer.

The intuition is simple: each additional attempt is another chance to fail, and failures don't average out when you need every attempt to land. At a 90% per-trial success rate over eight consecutive tasks, you're down to 43%. The exponent is merciless. What looks production-ready in a single trial becomes a coin flip across a shift's worth of work.

The optimistic version, pass@k, originated in code generation research, where it made perfect sense. Generate several candidate solutions, run them against a test suite, ship the one that passes. You're selecting the best attempt from a batch. But a customer calling about a refund doesn't get three agents in parallel. Neither does a fraud check, a booking flow, or a compliance verification. For sequential work, Sierra Research's tau-bench introduced pass^k: the probability that all k attempts succeed. The metric that captures what production actually feels like.

The results were clarifying. GPT-4o scored roughly 61% on tau-bench's retail tasks in a single trial. Respectable. But pass^8 in the same domain fell below 25%. A 40-point collapse, from the best model tested at the time.

And tau-bench is essentially alone in reporting this number. SWE-bench, GAIA, WebArena, TheAgentCompany: the benchmarks generating headlines and shaping procurement decisions all report single-trial accuracy or pass@k. Researchers at KTH ran 60,000 trajectories on SWE-bench and found that single-run estimates vary by up to 6 percentage points depending on which run you happen to pick. Reported improvements of 2-3 points may be noise.

So the benchmarks measure capability. Practitioners experience consistency. The gap between those two properties of the same system is where production plans go wrong. In a study of 306 practitioners, reliability was the top development challenge, and teams responded by deliberately constraining agent autonomy: 68% of production agents execute at most ten steps before requiring human intervention. The LangChain developer survey tells the same story from a different angle, with 32% of respondents naming quality as their top barrier to production deployment. These teams watched the demo work. Then they watched Monday happen.

The industry's most-cited numbers are built on a metric designed for batch selection, applied to sequential execution. That gap is the distance between a compelling demo and a deployment that holds up across thousands of runs.

The next time an agent benchmark crosses your desk reporting 85% accuracy, you have the question worth asking: 85% of what? One success in several attempts, or reliable performance every time? In a field where almost no major benchmark reports the sequential number, the absence is itself a finding. Pass^k wasn't deliberately suppressed; it just never became the convention. And that silence has been shaping procurement conversations, deployment plans, and staffing estimates built on numbers that describe a system's best day, not its typical one.