You Can't Stage Someone Else's Website

A housing association in Wales built an automation to handle food-bank referrals. It depended on a third-party website. The week it went live, the food bank changed both its approach and its website, and the integration was dead. The rule the team wrote down afterward: "only automate things that are completely within our control."

You can read that as timidity. I read it as a team correctly locating the edge of what they were able to test.

What breaks the assumption

Every pre-deployment practice we have rests on one idea: that you can reproduce the conditions your software will meet. Stand up a staging environment, run the suite, ship with some confidence that the real thing behaves like the rehearsal. I've spent years arguing that staging is a comforting fiction even when you own both ends of the wire. When one end belongs to another company, it isn't even that.

They change their interface on their schedule, for their reasons, and nobody sends you a note. A button moves, a form field gets renamed, a workflow grows a step. None of it is aimed at you. The person who shipped that change doesn't know your agent exists and wouldn't care if you told them.

Some changes are aimed at you. Organizations that notice automated traffic add CAPTCHAs, throttle requests, restructure pages to break scrapers, or block you outright. That's what makes this structural rather than merely expensive: the environment moves in response to being used, so measuring it changes what you're measuring. A benchmark built from a snapshot of an interface starts going stale the moment it's captured, and running the agent against the live site to find out whether the snapshot still holds is exactly the traffic that gets you blocked. You cannot stage a phone call to an insurance company's claims line. You cannot rehearse a purchase on a live retail site without buying something.

Why bounded agents are good engineering

A study of 86 agent systems found that 68% executed fewer than ten steps before a human had to intervene. Almost half needed one inside five. The systems touching the outside world ran tightest of all: fixed action sequences, mandatory approval at decision points, read-only modes, sandboxed verification before anything hit the real interface.

Against an assumption of full autonomy, those numbers look like a technology that isn't finished. But look at what the constraints actually do: they're the ordinary response to an untestable environment. Short execution horizons cap how far a system can drift before someone checks its work, and narrow scopes limit what it can reach while drifting. The human checkpoint is sitting exactly where pre-deployment testing would sit, if pre-deployment testing were on offer.

A water utility in Portugal lost a working automation to a routine laboratory website redesign. It moved the follow-on projects onto back-end database integrations and started budgeting ongoing maintenance for anything still touching a front end. Same lesson as the Welsh team, bought at a higher price.

Where the confidence has to come from

Distributed systems engineering went through a duller version of this. Teams spent years trying to make staging a faithful replica of production, gave up, and got observability and resilience engineering out of the surrender: instrument everything, deploy in small increments, detect early, recover quickly. Production was too complicated to copy, but it was ours.

Here it isn't. The environment runs on someone else's calendar and may change specifically because you showed up. That leaves monitoring, containment and recovery doing the job that pre-deployment testing does everywhere else, not as a fallback but as the primary control. The Welsh rule is the clean way out, and most organizations won't take it, because the work worth automating usually sits on the other side of the boundary. If you're crossing it anyway, be honest about which controls are holding the weight.