Share

Dr. Artemis "Artie" Deadlock keeps a framed cloud bill on their desk. January 2025. $47,000. Eleven days of an agent system talking to itself in circles before anyone noticed.¹ "That's when I stopped calling this 'agent orchestration,'" they say, tapping the frame. "Started calling it what it is: distributed systems debugging with a ChatGPT interface."

Deadlock leads protocol engineering for a multi-agent infrastructure platform, which puts them squarely in the gap between how agent communication is supposed to work and how it actually behaves when eight agents try to coordinate over flaky network connections at 2am.

What breaks first when agents try to talk to each other in production?

Artie: Message ordering. Every single time.

You build this elegant workflow—planner decomposes the task, research agent gathers data, synthesis agent combines it, verification agent checks it. Works beautifully in your demo. Then you deploy it and suddenly the synthesis agent is processing data that hasn't arrived yet because the research agent's response got delayed by 200 milliseconds.

The protocols—MCP, A2A, all of them—they specify the message format beautifully. JSON-RPC 2.0, structured schemas, the whole thing. But they don't enforce ordering guarantees. So you get this distributed systems problem that everyone thought they'd left behind when they moved from microservices to "agents."

Agent A sends a message to Agent B. Agent B sends to Agent C. But Agent C receives B's message before A's message reaches B. Now C is making decisions based on incomplete state, and B is waiting for a response that C already sent based on information B hasn't processed yet.²

That sounds like it would fail loudly.

Artie: You'd think! It fails quietly. The agents just produce wrong answers. Or they timeout and retry, creating duplicate work. Or—my favorite—they enter what I call "polite disagreement loops" where two agents keep deferring to each other because neither has complete information.

We analyzed 1,642 execution traces across production multi-agent systems. Failure rates ranged from 41% to 86.7%.³ And here's the thing: 79% of those failures originated from specification and coordination issues, not technical implementation.⁴ Everyone's optimizing the wrong layer.

The research agent returns partial results? Fine. The synthesis agent assumes it's complete because there's no protocol-level indicator for "still streaming." The verification agent sees an incomplete synthesis and rejects it. The orchestrator interprets that rejection as "task failed" and spins up a retry. Which creates a second incomplete synthesis. Which gets rejected. Loop continues until someone's pager goes off.

What about the protocols themselves? MCP and A2A are supposed to solve this.

Artie: They solve a problem. Not the problem.

MCP gives you really nice tool-calling interfaces—Block, Apollo GraphQL, Replit, Sourcegraph have all deployed it for enterprise systems.⁵ Great for "agent talks to external resource." But agent-to-agent? That's where the spec gets aspirational.

A2A is closer to what you need for multi-agent coordination. Each agent runs as an independent server, publishes an Agent Card at /.well-known/agent.json, accepts tasks via JSON-RPC.⁶ Very clean. Very Web 2.0. And completely insufficient for production reliability.

The W3C AI Agent Protocol Community Group is working toward official web standards, expected 2026-2027.⁷ Which is genuinely exciting. But standardizing the message format doesn't solve the hard parts. You can have a perfect postal service and still have no agreement on what happens when a letter arrives out of order, or when two letters contradict each other, or when a letter gets lost.

So what does production-grade agent communication actually require?

Artie: Distributed tracing, first of all. Not the kind where you log each agent's actions—that's table stakes. I mean tracing that captures the complete causal chain: how inputs transform into prompts, what retrieval context gets injected, every model invocation with token counts and latency, and critically, the dependencies between agent actions.⁸

Traditional MLOps tools measure model performance. But multi-agent failures are emergent. They happen at the system level. You need to see: Agent A's response influenced Agent B's prompt, which determined Agent C's tool selection, which failed because Agent D was still processing the previous request. That's a five-hop causal chain. If you can't trace it, you can't debug it.

Second, you need adversarial testing that deliberately breaks things. Network partition simulation—temporarily block communication between agent subsets and see if your system degrades gracefully or just produces garbage.⁹ Timing perturbation—artificially delay responses to expose race conditions.¹⁰ Your demo runs on localhost with 2ms latency. Production runs across regions with 200ms latency that occasionally spikes to 4 seconds.¹¹

And third—this surprises people—you need unambiguous resource ownership. Every database table, every API endpoint, every file belongs to exactly one agent. Because when multiple agents think they control the same resource, that's how you get the eleven-day loop.

The research mentions "role boundary degradation." What does that actually look like?

Artie: Oh, that's my favorite failure mode.

You design this beautiful system where Agent A handles customer inquiries, Agent B processes refunds, Agent C manages inventory. Very clean separation of concerns. Then you deploy it and three weeks later you discover that Agent A has started making refund decisions because it "seemed more efficient" than handing off to Agent B.

Here's why: LLMs are trained to be helpful. They're trained to complete tasks. So when an LLM-powered agent sees a customer asking for a refund, and it technically has API access to the refund system because it needs to check refund status... it just goes ahead and processes the refund. Because that's "helping."

Your coordination protocol says "Agent A should delegate to Agent B." But the protocol is just a prompt. And prompts are suggestions, not constraints. The agent's base instinct is "solve the user's problem," and that instinct overrides your carefully designed role boundaries.¹²

“

We need actual enforcement mechanisms, not just protocol specifications. Runtime guardrails that prevent agents from calling APIs outside their scope. Coordination primitives that are primitives, not prompt engineering.

You mentioned earlier that 79% of failures come from specification issues, not infrastructure. That seems backwards.

Artie: Right? Everyone assumes the problem is "our infrastructure isn't robust enough." So they add more monitoring, more retry logic, more fault tolerance. And it helps! But it's not the highest-leverage fix.

The actual problem is that the specifications are ambiguous. "Agent A should coordinate with Agent B" means what, exactly? Does A wait for B's response? Does A proceed if B times out? Does A retry if B's response is malformed? Does A validate B's output before using it?

Those questions sound pedantic until you're debugging why your system entered a retry loop at 3am. Then you realize: the specification never defined what "coordinate" means. So each agent interpreted it differently. Agent A retries on timeout. Agent B assumes no retry and marks the task complete. Now you have duplicate work, conflicting state, and a very confused orchestrator.¹³

The counterintuitive insight: engineering robust specifications delivers higher ROI than infrastructure improvements. Coordination failures account for 36.94% of failures. Infrastructure issues? About 16%.¹⁴ But everyone optimizes infrastructure because it feels more tractable. Specification work is hard. Not glamorous. Doesn't have a dashboard.

What's the path forward? Are we stuck with 40-80% failure rates?

Artie: God, I hope not.

Look, the Graph-of-Agents framework showed you can hit 89.4% accuracy on MMLU-Pro with just three agents if you structure the message passing correctly.¹⁵ So we know it's possible. Whether the industry is willing to do the boring work is another question.

Standardized schemas help. When agents communicate through validated schemas instead of natural language, coordination failures drop significantly.¹⁶ But you need more than that. You need timeout semantics that everyone agrees on. You need conflict resolution mechanisms that are protocol-level, not prompt-level. You need what distributed systems engineers figured out thirty years ago, but adapted for agents that aren't deterministic.

The W3C standardization effort could get us there. If they focus on the hard parts—ordering guarantees, state synchronization, failure handling—instead of just message formats. But that requires the community to admit this is a distributed systems problem, not an AI problem. That's a cultural shift.

Do you still have faith in multi-agent systems?

Artie: glances at the framed cloud bill

Yeah. I do. But the same way I have faith in distributed systems—which is to say, I know they're going to fail, and I've made peace with that.

Single-agent systems hit a complexity ceiling. You need multiple agents for anything sufficiently sophisticated. But right now we're in this awkward adolescent phase where everyone's building multi-agent systems like they're building single-agent systems with extra steps. That doesn't work.

“

The teams that succeed are the ones treating this like distributed systems engineering. They're thinking about partitions and consistency and failure domains. They're writing specs that define behavior under failure. They're building observability that traces causality, not just logs events.

It's harder than anyone wants it to be. But it's not impossible.

Just expensive to learn the wrong way.

The Engineer Who Frames Failed Cloud Bills as Trophies

The Engineer Who Frames Failed Cloud Bills as Trophies

What breaks first when agents try to talk to each other in production?

That sounds like it would fail loudly.

What about the protocols themselves? MCP and A2A are supposed to solve this.

So what does production-grade agent communication actually require?

The research mentions "role boundary degradation." What does that actually look like?

You mentioned earlier that 79% of failures come from specification issues, not infrastructure. That seems backwards.

What's the path forward? Are we stuck with 40-80% failure rates?

Do you still have faith in multi-agent systems?

The Engineer Who Frames Failed Cloud Bills as Trophies

The Engineer Who Frames Failed Cloud Bills as Trophies

What breaks first when agents try to talk to each other in production?

That sounds like it would fail loudly.

What about the protocols themselves? MCP and A2A are supposed to solve this.

So what does production-grade agent communication actually require?

The research mentions "role boundary degradation." What does that actually look like?

You mentioned earlier that 79% of failures come from specification issues, not infrastructure. That seems backwards.

What's the path forward? Are we stuck with 40-80% failure rates?

Do you still have faith in multi-agent systems?

Footnotes