--- title: "The Number You Didn't Ask About" canonical: "https://current.tinyfish.ai/issue/33/foundations/article/22078/the-number-you-didnt-ask-about" publication: "TinyFish" publisher: "TinyFish" language: "en-US" guide: "https://current.tinyfish.ai/llms.txt" format: "Reliability Lens" author: "Nora Kaplan" author_url: "https://current.tinyfish.ai/team/26" issue: 33 section: "foundations" section_name: "Foundations" published: 2026-06-11 updated: 2026-06-12 description: "How the gap between pass@k and pass^k quietly distorts every agent benchmark, procurement decision, and deployment plan built on headline numbers." word_count: 665 source_links: 11 --- # The Number You Didn't Ask About *By [Nora Kaplan]()* *June 11, 2026* [Read on the website]() An agent succeeds on a customer service task 70% of the time. Report the chance it gets at least one success in three tries, and you can write 97%. Report the chance it handles three consecutive requests without failing, and you write 34%. Same agent. Same math. A 63-point gap created entirely by which question you decided to answer. The intuition is simple: each additional attempt is another chance to fail, and failures don't average out when you need every attempt to land. At a 90% per-trial success rate over eight consecutive tasks, you're down to 43%. The exponent is merciless. What looks production-ready in a single trial becomes a coin flip across a shift's worth of work. The optimistic version, [pass@k](https://arxiv.org/pdf/2602.07150), originated in code generation research, where it made perfect sense. Generate several candidate solutions, run them against a test suite, ship the one that passes. You're selecting the best attempt from a batch. But a customer calling about a refund doesn't get three agents in parallel. Neither does a fraud check, a booking flow, or a compliance verification. For sequential work, Sierra Research's [tau-bench](https://arxiv.org/pdf/2406.12045) introduced pass^k: the probability that *all* k attempts succeed. The metric that captures what production actually feels like. The results were clarifying. GPT-4o scored roughly 61% on tau-bench's retail tasks in a single trial. Respectable. But pass^8 in the same domain [fell below 25%](https://sierra.ai/blog/benchmarking-ai-agents). A 40-point collapse, from the best model tested at the time. And tau-bench is [essentially alone](https://simmering.dev/blog/agent-benchmarks/) in reporting this number. SWE-bench, GAIA, WebArena, TheAgentCompany: the benchmarks generating headlines and shaping procurement decisions all report single-trial accuracy or pass@k. Researchers at KTH ran [60,000 trajectories](https://arxiv.org/abs/2602.07150) on SWE-bench and found that single-run estimates vary by up to 6 percentage points depending on which run you happen to pick. Reported improvements of 2-3 points may be noise. So the benchmarks measure capability. Practitioners experience consistency. The gap between those two properties of the same system is where production plans go wrong. In a [study of 306 practitioners](https://arxiv.org/pdf/2601.09032), reliability was the top development challenge, and teams responded by deliberately constraining agent autonomy: 68% of production agents execute at most ten steps before requiring human intervention. The [LangChain developer survey](https://www.langchain.com/state-of-agent-engineering) tells the same story from a different angle, with 32% of respondents naming quality as their top barrier to production deployment. These teams watched the demo work. Then they watched Monday happen. The industry's most-cited numbers are built on a metric designed for batch selection, applied to sequential execution. That gap is the distance between a compelling demo and a deployment that holds up across thousands of runs. The next time an agent benchmark crosses your desk reporting 85% accuracy, you have the question worth asking: 85% of what? One success in several attempts, or reliable performance every time? In a field where almost no major benchmark reports the sequential number, the absence is itself a finding. Pass^k wasn't deliberately suppressed; it just never became the convention. And that silence has been shaping procurement conversations, deployment plans, and staffing estimates built on numbers that describe a system's best day, not its typical one. ## Things to follow up on... - **Failures aren't independent:** A [reliability science framework paper](https://arxiv.org/html/2603.29231v1) tested whether agent failures behave like independent coin flips and found systematic violations, meaning the pass^k = p^k formula actually *understates* the consistency problem in practice. - **Autonomy keeps growing:** Anthropic's research on agent autonomy found that [99.9th-percentile turn duration nearly doubled](https://www.digitalapplied.com/blog/ai-agent-evaluation-pipeline-2026-testing-methodology) between October 2025 and January 2026, meaning more steps per run and more surface area for the exponential to bite. - **Scaling is an allocation problem:** A March 2026 survey of 650 enterprise technology leaders found that organizations reaching production scale [weren't spending more on AI overall](https://www.digitalapplied.com/blog/ai-agent-scaling-gap-march-2026-pilot-to-production) but were allocating proportionally more to evaluation infrastructure and monitoring than to model selection. - **Phil Schmid's walkthrough:** Google DeepMind's Phil Schmid published a [practitioner-oriented explainer](https://www.philschmid.de/agents-pass-at-k-pass-power-k) using a flight rebooking agent to show how pass^k changes staffing estimates and architectural decisions. ## More - [Foundations — Issue 33]() - [Issue 33]() - [Publication guide]()