We've seen pilots where someone spent the first hour of every day manually checking that overnight data feeds completed successfully. The automation couldn't start until this check finished. The pilot demo never showed this hour of manual work—it just showed the automation running smoothly on clean data.
This happens everywhere. The pilot succeeds. The technology works. But when we examine what actually happened, there's always someone who kept it running—catching edge cases, fixing data issues, manually handling exceptions. The pilot succeeded partly because this person existed. Production can't depend on them the same way.
When One Percent Becomes a Hundred Daily
The pilot processed a hundred test transactions over six weeks. Everything worked. Then production started, processing ten thousand transactions daily. On day forty-seven, someone discovered a customer with three addresses on file. The automation didn't know which one to use. It failed.
This wasn't a bug. The pilot never encountered this case because it represents roughly 1% of transactions. One percent of a hundred test cases is one transaction—easy to miss. One percent of ten thousand daily transactions is a hundred exceptions every day.
Pilots succeed because someone catches edge cases, fixes data issues, and manually handles exceptions. Production can't depend on this person the same way.
We've observed this pattern repeatedly: pilots operate on the subset of cases that are well-understood and predictable. Production forces you to handle everything else. The customer with the partial refund. The product that exists in the catalog but isn't actually available. The order that was placed, cancelled, then reinstated. None of these appeared in pilot testing because they're individually rare. Collectively, they're constant.
Humans handle these variations without conscious thought. Someone sees three addresses and picks the most recent one. The automation requires explicitly handling each case—and pilots rarely run long enough to reveal the full complexity.
The Data Someone Cleaned
Pilot data is almost always cleaner than production data. Not because anyone is hiding problems, but because pilots naturally work with recent, accessible data that someone selected to represent normal conditions.
Production data is messier:
- The customer record that was migrated from three different legacy systems and has conflicting information in each
- The product description that contains special characters that break parsing
- The timestamp that's in the wrong timezone because it came from a regional database
We've seen pilots succeed because someone spent three hours every morning cleaning data before the automation ran. They removed duplicates, standardized formats, fixed encoding issues. The pilot metrics showed the automation working perfectly. What they didn't show was the manual data preparation that made it possible.
At production scale, that manual preparation becomes impossible. You can't have someone clean ten thousand records daily. The automation has to handle data as it actually exists—inconsistent, incomplete, occasionally contradictory.
The Oversight That Doesn't Scale
Pilots depend on constant human oversight. Someone watches the pilot run, catches anomalies, manually intervenes when something looks wrong. At pilot scale—ten concurrent sessions, a handful of test scenarios—this oversight is feasible. At production scale, it becomes impossible.
We've watched teams discover this gap too late. The pilot ran for three months with someone checking results daily, manually correcting issues, noting patterns. The automation appeared reliable. Then production started, and that same person couldn't keep up. They were catching maybe 5% of issues instead of 95%. The automation that seemed production-ready was actually dependent on intensive manual oversight that couldn't scale.
Pilots succeed by operating in conditions that production can't maintain. The manual work, the curated data, the limited edge cases, the constant oversight—all of it scales poorly or not at all.
What Gets Discovered Too Late
Pilots don't reveal ongoing maintenance requirements. The automation works during the pilot because conditions are stable. Someone updates it when the website changes. Adjusts it when data formats shift. Handles new edge cases as they appear.
Then production starts, and those changes happen constantly. The website updates its layout twice a month. Data formats vary by region. New edge cases appear daily. The automation that seemed stable during the pilot requires continuous adjustment to stay functional.
We've seen organizations successfully move from pilot to production. They understand what their pilots were hiding and build systems that work without those hidden supports. They design for the exceptions that will appear. They build data pipelines that handle inconsistency. They create monitoring that doesn't require constant human oversight.
The pilot proved the concept. Production requires proving you can maintain it when the manual supports disappear.