Share

Evaluation Thesis

The Score Is Not the Agent

By Sable Whitford— September 17, 2026

Feature image for article: The Score Is Not the Agent

Six models, a hundred enterprise problems, one evaluation harness — and the top-ranked agent changed depending on whether it faced scripted opponents or other agents. The rank correlation between the two conditions was 0.182, barely better than ranking them by coin flip. Three recent studies converge on the same finding: an agent's benchmark score is not a property of the agent. It's a property of the agent and everything around it while the test ran. That distinction changes what any benchmark result can tell you.

Evaluation Thesis

The Score Is Not the Agent

By Sable Whitford— September 17, 2026

Six models, a hundred enterprise problems, one evaluation harness — and the top-ranked agent changed depending on whether it faced scripted opponents or other agents. The rank correlation between the two conditions was 0.182, barely better than ranking them by coin flip. Three recent studies converge on the same finding: an agent's benchmark score is not a property of the agent. It's a property of the agent and everything around it while the test ran. That distinction changes what any benchmark result can tell you.

Postcondition Walkthrough

The Agent Clicked Save. Check the Database Anyway.

By Sable Whitford— September 17, 2026

Feature image for article: The Agent Clicked Save. Check the Database Anyway.

An agent navigates to the right page, types into the right-looking field, clicks save, and gets a confirmation. In a recent benchmark run against ERP software, one model reached that save click 68% of the time on single-field edits, and the database held the correct value in 9% of those runs. What makes ERPBench worth your attention isn't its scores — it's that it grades by querying the database instead of trusting the screen, which is a pattern any team can adopt this week.

Postcondition Walkthrough

The Agent Clicked Save. Check the Database Anyway.

By Sable Whitford— September 17, 2026

An agent navigates to the right page, types into the right-looking field, clicks save, and gets a confirmation. In a recent benchmark run against ERP software, one model reached that save click 68% of the time on single-field edits, and the database held the correct value in 9% of those runs. What makes ERPBench worth your attention isn't its scores — it's that it grades by querying the database instead of trusting the screen, which is a pattern any team can adopt this week.

Evaluation Reading

Towards a Science of AI Agent Reliability

Eighteen months of capability gains barely moved the needle on reliability. A four-dimensional framework tries to explain why.

AlphaEval: Evaluating Agents in Production

Across 94 real-company tasks, swapping the scaffold shifted performance more than swapping the model.

From Benchmarks to Deployment: A Comprehensive Review of Agentic AI Evaluation

AgentHijack: Benchmarking Computer Use Agent Robustness to Common Environment Corruptions

Investigating the Impact of Dark Patterns on LLM-Based Web Agents

ERPBench: State-Grounded Evaluation for Computer-Use Agents in Enterprise Software

Evaluation Reading

Towards a Science of AI Agent ReliabilityEighteen months of capability gains barely moved the needle on reliability. A four-dimensional framework tries to explain why.

AlphaEval: Evaluating Agents in ProductionAcross 94 real-company tasks, swapping the scaffold shifted performance more than swapping the model.

Quick links

From Benchmarks to Deployment: A Comprehensive Review of Agentic AI Evaluation

AgentHijack: Benchmarking Computer Use Agent Robustness to Common Environment Corruptions

Investigating the Impact of Dark Patterns on LLM-Based Web Agents

ERPBench: State-Grounded Evaluation for Computer-Use Agents in Enterprise Software

Benchmark Diagnostics

A benchmark result is a number attached to specific conditions. When the result gets cited, the conditions usually stay behind.

Each card below takes a published finding and asks: what would need to be true about the evaluation setup for this number to mean what it seems to mean? The answer tends to involve something the headline leaves out, whether that's the grading method, the infrastructure wrapping the model, or whether the evaluation measured outcomes beyond the stated task.

These are mostly careful, well-designed studies. But a score separated from its operating conditions becomes a different kind of claim than the one the researchers published.

Benchmark Diagnostics

A benchmark result is a number attached to specific conditions. When the result gets cited, the conditions usually stay behind.

Each card below takes a published finding and asks: what would need to be true about the evaluation setup for this number to mean what it seems to mean? The answer tends to involve something the headline leaves out, whether that's the grading method, the infrastructure wrapping the model, or whether the evaluation measured outcomes beyond the stated task.

These are mostly careful, well-designed studies. But a score separated from its operating conditions becomes a different kind of claim than the one the researchers published.

State Grading

Agents Save the Form but Write Wrong Data

ERPBench tested screenshot-only agents on a live ERP system and graded results against the database, not the screen. Agents saved forms in up to 85% of runs but entered the correct value only 3% of the time. The difference comes down to whether you check the screen or the database.

Scaffold Sensitivity

The Scaffold Ships Quietly Inside the Score

SWE-bench Verified scores swing 10-20 points depending on which code framework wraps the model during testing. Vendors self-report using their best harness. When Epoch AI upgraded its scaffold in February 2026, scores jumped without any model changing. The scaffold is part of the measurement.

Run Consistency

One Successful Run Means Less Than Eight

tau-bench measures pass^k: whether an agent succeeds consistently across repeated identical runs. On retail tasks, pass^8 fell below 25% for tested agents. But the benchmark's four built-in scaffolds shift the same model's score by up to 0.267 and rearrange the rankings.

Collateral Outcomes

Task Complete and Warranty Purchased Are Both True

TrickyArena graded task success and susceptibility to deceptive UI patterns independently on the same execution trace. GPT-4o agents completed tasks at 38.9% while triggering unwanted side outcomes 41.1% of the time. A success metric that only checks task completion would miss the second number.