XCON and the Maintenance Paradox

Ordering a VAX from Digital Equipment Corporation got you a pile of parts. CPUs, memory boards, backplanes, power supplies, disk drives, cables. Somebody had to work out how all of it fit together before assembly could start: what the order was missing, which board went in which slot, whether the cabinet could actually power what you'd stuffed into it.

Technical editors did this by hand, one order at a time, the day before assembly. XCON replaced them, then surpassed them, producing assembly diagrams that went further into physical layout than the editors ever had. By 1989 it configured hundreds of systems a day. It worked. The former editors stayed on as reviewers, checking its output and flagging anything wrong.

Those flags went back to development. Reviewer spots a bad diagram, a knowledge engineer finds the gap, writes the missing rule, tests it. Additive work. You find out what the system doesn't know and you teach it.

For a while, that was the whole job.

Two backplane-selection tasks took 36 rules in December 1980. Over three years, developers added 40, removed three, changed 11, arriving at 73. Twenty-seven of the additions were special cases, situations the original conditions hadn't told apart: a few general rules in the middle, exceptions thickening around them. Then somebody reworked the whole thing. Thirty-one of the 73 rules replaced, most of the rest modified, total still 73. The task hadn't grown. Three years of patching had buried what the team understood about backplanes, and the rewrite dug it back out.

That rewrite was a small instance of something happening across the whole system. Rules did several jobs at once: one documented example marked a drive as configured, recorded the cabinet space it took, created cabling information, tracked containment, and generated output labels, all in the same rule. When developers needed to support a new device, they copied a similar device's rules and edited them, often without understanding every action in the original. If you don't know why a line is there, you have two options and both are bad. Change it and maybe break something a hundred rules away, or leave it and carry your ignorance into the copy. Anyone who has inherited a configuration file knows that fork.

Developers could tell who had written a rule from its style. They were also uncertain what some of those rules did, and reluctant to touch them.

And it kept working. Digital started a major redesign not because XCON was failing in production but because it was getting harder to change every time a new product shipped. Maintenance had begun as teaching. By then it was closer to surgery, where nothing gets cut without first understanding tissue you didn't put there.

This is not a story about expert systems being a bad idea. XCON ran for a decade. The transition probably has less to do with the technology than with what happens when encoded knowledge becomes essential to operations: enough people contribute enough additions for enough different reasons that nobody holds the whole picture, and the work shifts from adding knowledge to managing the consequences of knowledge already there.

Nobody builds rule systems like XCON anymore. The artifacts changed shape: prompts, the tools a model is allowed to call, the test cases you judge it against, the documents you feed it, the points where it has to stop and ask permission. They accumulate the same way, and they get edited by people who didn't write the originals and can't always say why a line is there.

Nobody goes looking for this problem while the system is doing its job, which is why it gets found late. Usually by whoever has been asked to change something the company can no longer turn off.