I have never inherited a system whose configuration I fully understood on day one. Nobody has. You show up, you read whatever documentation exists, you trace the wiring, and you start building a mental model that will always be incomplete. That's fine. That's ops. What I actually care about is whether the gap between what the system does and what anyone understands about why it does it is holding steady or getting wider.
XCON is the best-documented case I know of where that gap kept widening until it became the main engineering problem.
XCON was an expert system that Digital Equipment Corporation built in the early 1980s to configure VAX computer orders. A VAX order wasn't a shopping cart. It was a constraint-satisfaction problem: processors, memory boards, backplanes, cabinets, power supplies, storage devices, controllers, cables, bus addresses, interrupt vectors, all with compatibility rules that shifted every time DEC shipped new hardware. The average order involved around 78 parts. By 1988 the system covered 42 CPU families and roughly 25,000 components. Human configurators were struggling to keep up. XCON wasn't.
And it worked. Fewer than one order in a thousand had a misconfiguration traceable to the system. After more than 80,000 orders, its maintainers wrote that those orders still represented only a fraction of the possible situations. Eighty thousand real orders in, and they're telling you they haven't seen most of the space yet.
The system's knowledge lived in if-then rules, about 6,200 of them by 1987, with roughly half changing every year. The maintainability assessment published that year describes what was going on inside those rules. A single rule often carried several jobs at once: it might mark a disk drive as configured, record which cabinet it went in, set up the cabling, and generate an output label. When developers needed to support a new device, they'd copy the rules for a similar one and edit the obvious differences, which meant carrying forward conditions whose rationale nobody remembered. Developers could recognize each other's coding styles the way I used to recognize which sysadmin had written a particular cron job. By the quirks. Certainly not by the documentation.
So from the outside, the system was doing great. On the inside, the people maintaining it were less and less sure what would break if they touched anything. If you've ever left a config line alone because you couldn't figure out why it was there and didn't want to find out the hard way, you know that feeling.
They started rebuilding. A methodology called RIME was designed to reorganize the knowledge into something more modular and easier to see into. Eight experienced developers rotated onto the effort, each for a week or two before going back to production work. (A week or two at a time, squeezed between production duties, is how nearly every rewrite I've ever watched gets staffed.) The researchers who assessed the results called them anecdotal and asked for longer-term measures they hadn't collected yet.
Meanwhile the production system kept growing. By 1989, XCON had over 10,000 rules. RIME was still the stated approach. The original system was still in production, still the thing that actually configured the orders. I wrote about XCON's maintenance paradox before, but I didn't get into this part: the rebuild was part-time and preliminary, and the growth was continuous and compounding. The redesign was never going to catch up with the system it was supposed to replace.
A 1980s rule engine and a modern AI agent aren't the same kind of thing architecturally. But the organizational pattern doesn't need matching architecture. All it needs is a successful system whose accumulated knowledge grows faster than the team's ability to understand it.
Agent deployments that work are going to pile up the same inventory: prompts, tool configurations, exception lists, workarounds. They'll pile it up for the same reason, too, because every addition solves a real problem for a real user. The harder question from XCON is about timing. By the time you recognize you need to redesign, the thing is in production, the business depends on it, and it's growing faster than you can rebuild it. XCON's maintainers saw the problem clearly and started on the fix. Nothing in the available record shows them finishing before the ground moved under them.
