--- title: "A Saga Pattern Survivor on the Third Path Nobody Drew on the Architecture Diagram" canonical: "https://current.tinyfish.ai/issue/50/echoes/article/41585/a-saga-pattern-survivor-on-the-third-path" publication: "TinyFish" publisher: "TinyFish" language: "en-US" guide: "https://current.tinyfish.ai/llms.txt" issue: 50 section: "echoes" section_name: "Echoes" published: 2026-10-08 updated: 2026-10-09 word_count: 1369 source_links: 5 --- # A Saga Pattern Survivor on the Third Path Nobody Drew on the Architecture Diagram *October 8, 2026* [Read on the website]() *Dex Rollback spent three years as a distributed systems architect at a major e-commerce platform during the great monolith decomposition of the mid-2010s. He now runs an independent consulting practice called Compensating Actions LLC, which helps organizations manage the operational fallout of distributed transaction patterns. We spoke over video call from what he described as "the room where I keep the whiteboards I should have burned." Neither Dex nor his impeccably named consultancy exist outside this conversation, but the problems he describes are painfully real.* --- ## You were part of the wave of teams that adopted the saga pattern after hitting the limits of two-phase commit. What pushed you over the edge? **Dex:** We had a checkout flow that crossed five services. Payments, inventory, shipping, loyalty points, notifications. We were running XA transactions across three of those, and during a flash sale — Black Friday 2015, I think — the transaction coordinator became such a severe bottleneck that we were holding locks across services for *seconds* at a time. Throughput dropped roughly 35%.[^1] The coordinator itself was a single point of failure, and when it hiccupped, participants just sat there. Blocked. Waiting for a commit or abort that wasn't coming. So someone on the team found Chris Richardson's microservices.io page on sagas, and we went back to the original 1987 Garcia-Molina and Salem paper.[^2] We thought, okay, *this* is the answer. Decompose the transaction into steps, write compensating transactions for each one, and we're free. Beautiful. Elegant. We were very excited for about six weeks. ## I take it the excitement wore off. **Dex:** Violently. ## Where did it break down first? **Dex:** The sheer volume of failure-path design work. Every step in the saga needed a compensation. And compensation is not rollback. The original paper is explicit about this: it "does not necessarily return the database to the state that existed when the execution began."[^2] You're creating a *new* consistent state from whatever reality exists now, because other transactions have been running concurrently. So for every step, we had to sit down and answer a surprisingly philosophical question: what does "undo" even mean here, given that the world has moved on since this step executed? We had this taxonomy — compensatable transactions, pivot transactions, retriable transactions.[^3] The pivot is your point of no return. Everything before it needs compensation defined. Everything after it should be retriable. Sounds clean on a whiteboard. In practice, deciding which step is the pivot for a five-service checkout flow involved three teams arguing for two weeks about whether charging the credit card or reserving inventory was the truly irreversible act. We nearly lost a senior engineer over it. He didn't quit. He just stopped coming to the meetings, which in retrospect was the healthier choice. ## And then compensation itself started failing? **Dex:** That's the thing nobody warned us about. The tutorials all show this beautiful backward cascade: step 3 fails, so you compensate step 2, then step 1. Arrows going left. Very satisfying. But compensation is just another distributed call. It fails for the same reasons the forward step fails — network partitions, database deadlocks, the target service being mid-deployment. We started seeing what I call **zombie records**. An order marked FAILED in our order service, but inventory still showing RESERVED because the compensation call to release stock timed out.[^4] The system couldn't go forward, couldn't go backward. Garcia-Molina and Salem actually named this state in 1987 — they said the system "is stuck" and proposed, essentially, giving it to a human.[^2] Which is exactly what we ended up doing, thirty years later, with fancier monitors. ## What did that look like operationally? **Dex:** We built a reconciliation queue. Every stuck saga got an entry with the full event history, the current state of each participating service as best we could determine, and a set of resolution actions an operator could take. We had an oncall rotation for it. SLAs. Dashboards. A Slack channel that nobody muted because they were afraid to. > And here's the part that still gets me: we had moved to microservices partly to eliminate the operational overhead of our monolith. Now we had a queue of ambiguous transaction states that required a named human owner, domain knowledge of the business flow, and the authority to force-complete or force-compensate individual steps. We'd rebuilt the governance structures we thought we were escaping, except now they were distributed across five teams and a Kafka dead-letter queue. ## You mentioned the dead-letter queue. That became its own problem? **Dex:** The DLQ graveyard. You retry compensation three times, it fails, you push it to the DLQ. Now you have a growing backlog of entries that represent real money. Reserved inventory. Pending charges. Loyalty points in limbo. And the DLQ doesn't have context. It has a message payload. Someone has to reconstruct what happened, what the customer saw, and what the right resolution is. That someone has a pager and a Jira board and a very specific facial expression. You've seen the expression. It's the one where the mouth is smiling because they're on a video call but the eyes have gone somewhere else entirely. ## How did service evolution affect saga definitions over time? **Dex:** This was the slow-burning problem, the one that didn't show up in any proof of concept. A saga definition is a contract across multiple teams. When the payments team changed their refund API — added a required field, shortened their refund window — every saga that included a payment compensation step was potentially broken. But nobody knew which sagas were in flight. We had sagas that ran for *days* during holiday periods. Long-running order fulfillment flows where a customer orders on Monday and the compensation might not fire until Thursday. The durable execution engine could replay the orchestration logic, sure.[^5] But it couldn't guarantee that the external world still accepted the compensation. A refund window that was 30 days when the saga started might be 14 days by the time the compensation fires. The payment processor's API version might have changed underneath you. We ended up needing inter-team versioning contracts — basically, you can't ship a breaking change to your compensation endpoint without checking a registry of active saga definitions. Which, if you squint at it long enough, looks remarkably like the coordination overhead of the monolith we decomposed. Except now it's voluntary, which means it's worse, because voluntary coordination is the kind people skip when they're behind on a sprint. ## If you could go back and tell your 2015 self one thing about sagas, what would it be? **Dex:** Budget for the reconciliation queue from day one. Not the technical implementation — the *people*. The oncall rotation, the runbooks, the authority to make financial decisions about stuck transactions. The pattern works. It's sound. Garcia-Molina and Salem got the theory right in 1987. But every architecture diagram I've ever seen of a saga shows two paths: the happy path and the compensation path. Nobody draws the third path — the one where compensation fails and a human has to figure out what "consistent" means at 2 AM with incomplete information and a customer on hold. > The 1987 paper knew this. They wrote that manual intervention "is definitely not an elegant solution, but it is a practical one."[^2] Thirty years of tooling later, that sentence is still the most honest thing written about distributed compensation. I keep it on a sticky note on one of those whiteboards I should have burned. --- [^1]: XA/2PC benchmarks from the mid-2010s typically reported 30-40% throughput reduction compared to local transactions. See https://singhajit.com/distributed-systems/two-phase-commit/ [^2]: Garcia-Molina, H. and Salem, K. "Sagas." *SIGMOD '87: Proceedings of the 1987 ACM SIGMOD International Conference on Management of Data*, pp. 249-259. Full text: https://www.cs.cornell.edu/andru/cs711/2002fa/reading/sagas.pdf [^3]: The compensatable/pivot/retriable taxonomy is detailed in saga pattern implementations. See https://medium.com/@abhi.strike/microservices-patterns-saga-pattern-2843752f5b4d [^4]: The "zombie records" problem — orders marked FAILED with inventory still RESERVED — is documented in practitioner accounts of compensation failures. See https://coldfusion-example.blogspot.com/2026/01/the-saga-pattern-trap-handling.html [^5]: Durable execution platforms replay orchestration code against recorded history but cannot guarantee external compatibility over time. AWS prescriptive guidance notes that compensatory transactions and retries add complexity resulting in maintenance overhead. See https://docs.aws.amazon.com/prescriptive-guidance/latest/cloud-design-patterns/saga-orchestration.html ## More - [Echoes — Issue 50]() - [Issue 50]() - [Publication guide]()