When Damage Outruns Detection

A human mistake is an incident. Someone notices, investigates, corrects. The error and the attention it draws share a time window, and that shared window is what makes correction possible. An agent mistake at agent speed is structurally different: the error has already been relied upon, forwarded, embedded in downstream decisions before anyone registers it occurred. You are not investigating an incident. You are doing archaeology.

Most adoption conversations have not absorbed this. They treat speed as a quantity. Faster means more of the same. But speed, past a certain threshold, changes the nature of the problem it produces.

On August 1, 2012, Knight Capital's automated trading system sent more than 4 million orders into the market in 45 minutes while trying to fill 212 customer orders. The firm lost over $460 million. The detail that matters most: the SEC found that 97 automated emails flagging the error had reached personnel before the market opened. No one acted on them. The alerts existed. The humans existed. The institutional capacity to absorb information at the speed the system generated consequences did not. The organization had built detection without building the organizational muscle to act on what was detected. Everything functioned identically, right until the moment it didn't.

That was one automated system, in one function, with a known blast radius. Today's agents are general-purpose actors running concurrently across organizational boundaries, compounding errors across domains simultaneously. Knight Capital was a warning about tempo mismatch in a contained system. Agents generalize that mismatch across the organization.

McKinsey reports that 88% of organizations use AI regularly but no more than 10% have scaled agents in any single function. The usual explanations are capability gaps and governance hesitation, and both are real. But underneath them sits something organizations feel before they can name it: they cannot metabolize action at the speed agents produce it. The gap between experimentation and scaled value is not primarily a technology problem. It is a tempo problem wearing a technology costume.

Take the domain where absorption should be easiest. Code review has the thickest quality infrastructure of any knowledge-work process: diffs, tests, CI pipelines, established review norms. If any domain can handle increased agent output, this one can. Microsoft's recent study found that coding-agent adopters merged 24% more pull requests per engineer per day, with heavy users reaching 50%. The authors themselves warn that merged PRs are an imperfect proxy and that the field lacks agreed measures of quality. Meanwhile, OpenAI's Codex usage data shows 99th-percentile internal users accumulating 71 hours of agent runtime in a single day, because concurrent agents let cumulative work exceed 24 hours. The paper describes humans overseeing "a team of agents." It does not measure whether that oversight is genuine supervision or dispatch-and-forget.

We are carefully measuring the speed of production while leaving the capacity for absorption unmeasured. That asymmetry tells you where risk is accumulating.

Every organization carries tacit repair mechanisms. Someone catches the odd output, flags the drifting context, notices the number that doesn't feel right. These mechanisms evolved at human tempo. They depend on proximity between action and attention, on the fact that someone is close enough to the work to sense when something has gone wrong. At agent speed, that proximity collapses. There is no one watching at the speed watching would need to happen. As I've argued elsewhere, when agents act faster than humans can track, tacit correction fails silently. The gap does not announce itself. It simply widens.

This is what makes the persistent distance between broad AI experimentation and scaled organizational value so structurally predictable. Organizations are framing agent adoption as a production problem. How do we generate more output, faster? But production was never the constraint. Reception is. Until we measure absorption with the same seriousness we bring to throughput, speed will keep producing a pattern that should look familiar to anyone who has watched institutions defer hard problems: technically successful actions that no one in the organization can confidently call their own. Someone always pays for that gap. It is never the people who chose to defer.