Reading an Architecture as an Argument About Failure

A web agent navigating an unfamiliar site has to solve two problems on every step: what to do next, and how to do it in this particular interface. Each step is a fresh reasoning call. Each call is a chance to drift. An SOP collapses one of those problems. The document carries institutional knowledge about what the workflow looks like when a person does it correctly. The agent still has to figure out how to click the right button on a page it's never seen, but it's no longer also guessing at the sequence. The reasoning burden per step drops. The surface area where things go wrong gets smaller.

That only matters as a design priority if you believe per-step reasoning cost is a primary failure driver. Skyvern's broader architecture suggests they do.

Their original system was a single prompt in a loop, making decisions and taking actions together. It scored around 45% on the WebVoyager benchmark. Simple tasks worked. Compound objectives broke it. Ask it to add three items to a cart and it might add the first, then confidently report success.

The response came in two layers. A Planner phase decomposes complex objectives into smaller goals and tracks what's done and what remains. That alone pushed accuracy to roughly 68.7%. Then a Validator phase, checking visual state after every action. Did a popup block the click? Did the URL actually change? If not, it reports back for replanning. With the Validator, accuracy reached 85.85%.

Look at what each jump actually fixed. The jump from 45% to 68.7% came from reducing what the agent had to hold in its head at any moment. The jump from 68.7% to 85.85% came from catching actions that looked successful but weren't. That second problem is the harder one. A reasoning failure is at least legible. An action that executed cleanly, returned no error, and still didn't work looks identical to success from inside the pipeline. The only way to catch it is to look at the screen afterward, the way a person would. The Validator does exactly that: it interrogates the visual state of the page, checking whether the world actually changed. Two different failure modes. Two different architectural responses.

Then there's where it was tested. Skyvern ran its benchmark on cloud infrastructure with async browsers, proxy networks, and CAPTCHA solvers. Not local machines with safe IPs. The 85.85% was achieved against whatever the live web threw at it.

A score earned behind bot detection answers a different question than a score earned in a cooperative testing environment. One tells you something about what the system does on a Tuesday morning when a site updates its defenses. The other tells you what it can do when nothing is fighting back.

The three choices share a logic. Reduce per-step reasoning burden so compound errors don't accumulate. Validate every action visually so silent failures surface immediately. Test in hostile conditions so the numbers reflect something close to production reality. Each addresses a different place where the distance between demo and deployment opens up. And the model stays the same across all three jumps. The Planner, the Validator, the cloud testing: every improvement is structural. They constrain where failures can hide. The design is the argument, and the argument is that reliability lives in the scaffolding around the model.