Something about production agent systems keeps nagging at me: the people running them could let the loop go longer. They keep deciding not to.
A survey of deployed systems found most agents taking ten steps or fewer before a human gets pulled in, mostly on models anyone can rent, mostly graded by somebody's judgment rather than any benchmark suite. Read that as a ceiling and the story writes itself. The models can't be trusted past ten, so nobody lets them go past ten, and the day the models get better the loops get longer.
There's another reading, and the model doesn't appear in it anywhere. We throttle a producer when the consumer downstream can't keep up. That's backpressure, one of the oldest moves in operations, and the throttle is always sized to the consumer's appetite. Nothing about it describes what the producer could have done.
It's the same reason good teams ship small diffs. Not because the build pipeline gags on fifty commits (it'll happily eat them) but because when something breaks at 3 AM you want the search space to be one change wide. The pipeline was never the constraint. Finding the problem was.
So ten steps may be sized to the reviewer.
I've been on call since I was seventeen, and the nights that wrecked me were never the ones where I broke it myself. Those were almost pleasant, in a grim way. You know the shape of your own mistake before the alert finishes loading. The bad ones went like this. Paged for a system I'd never touched, making changes with no feel for the blast radius. Then sitting across a table the next morning from someone whose customers had felt it, trying to explain what happened in language that would hold up. That morning is what a long agent run manufactures at the far end. Forty steps deep, what lands is a stack of actions somebody now has to accept, defend, and unwind if they were wrong. That somebody usually doesn't work on the team that shipped the agent.
The best evidence for what happens next comes out of code review, because there you can size both the artifact and the reviewer. In a controlled experiment, 50 mostly professional developers caught 59% of planted defects in a 31-line change and 35% when the change was an order of magnitude bigger, with time spent per defect found climbing about two thirds. A companion result held the content constant and split it into two tidy pull requests. Reviewers made fewer false accusations. They found no more defects and worked no faster. Packaging bought manners, not capacity. The bigger patches were also harder in themselves, so take all of it as a direction rather than a law. The direction is still shitty news for anyone hoping review scales with output.
The quantity nobody counts is how much disputable action is in flight right now, waiting on a human who may or may not exist. Security guidance sensibly demands explicit approval for anything irreversible or externally visible, which governs individual moves. Step caps govern whole runs. I went looking for a spec that connects them and can't find one.
The step budget should shrink as the blast radius grows. Absent that, the loop cap becomes the proxy.
Careful teams behave like they already know this. A deployment paper on support agents at a bank serving over 100 million users prescribes narrow scope and deep specialization: five to fifteen tools, a routine lifted straight out of an existing human procedure, handoff the second a case wanders outside it. The reason they give points downstream. The person who inherits the conversation has to be able to finish it.
Skip the proxy and you get what Sevda Polat traced in "A Queue Is Not a Shelf": one organization roughly doubled output per developer while review coverage slid from 89% to 68%, and substantive review from 39% to 21%. The work sped up. The verifying didn't.
Which makes human evaluation look less like a placeholder waiting on a real metric. Ask what the missing benchmark would have to certify. Not whether the model picked a defensible action. Whether your support organization will stand behind this refund tomorrow, in writing, to a furious customer.
Nobody benchmarks that. Capability is a property of the model. Deployability is a property of whoever has to catch what comes out.
- What deployed agents disclose: A survey of public documentation for 30 prominent systems in the 2025 AI Agent Index found detailed action traces for ten, agent-specific system cards for four, and no published internal safety results for 25 — a reminder that tightly controlled internally and legible from outside are separate properties.
- Correct and unsafe at once: In a joint Singapore and Korea AI Safety Institute evaluation of ordinary, non-adversarial tool-using tasks, individual correctness scores reached 90% while fully correct-and-safe runs ranged from 15% to 35%, mostly through agents putting useful information in front of the wrong audience.
- Risk labels nobody sees: The protocol maintainers' own explanation of tool annotations concedes that flags marking a tool as destructive were not broadly surfaced in approval interfaces and cannot be trusted when the server supplying them is untrusted, which is worth reading before treating them as a gate.
- When the handoff happens matters: A randomized field experiment on human intervention in agent-handled support chats found that people preserved quality after technical breakdowns but much less so after emotional ones, suggesting the escalation point, not its existence, determines what repair remains possible.