Share

The Signal

When Organizations Commit to Agents They Don't Yet Have

By Nora Kaplan— December 10, 2025

Feature image for article: When Organizations Commit to Agents They Don't Yet Have

The questions arrive after the commitment. How do we handle authentication across fifty regional variations? What happens when site structures change mid-workflow? How do we monitor behavior we can't see inside? Teams reach out saying they've adopted agents, then reveal through their requirements that they've restructured work around capabilities they're still building. The gap between organizational change and infrastructure reality is widening—and the teams caught in it can't go back.

The Signal

When Organizations Commit to Agents They Don't Yet Have

By Nora Kaplan— December 10, 2025

The questions arrive after the commitment. How do we handle authentication across fifty regional variations? What happens when site structures change mid-workflow? How do we monitor behavior we can't see inside? Teams reach out saying they've adopted agents, then reveal through their requirements that they've restructured work around capabilities they're still building. The gap between organizational change and infrastructure reality is widening—and the teams caught in it can't go back.

Where This Goes

Framework Wars End Through Consolidation, Not Competition

The agent framework explosion of 2023-2024 is collapsing faster than anyone expected. Microsoft merged AutoGen with Semantic Kernel in October. LangChain pivoted from chains to LangGraph. CrewAI went from interesting to $18M Series A with 60% Fortune 500 penetration.

Foundation models are absorbing what frameworks used to provide. O1's native reasoning, Gemini 2.0's built-in tool use. We think this reveals something deeper than market consolidation. Agent infrastructure is maturing by disappearing into fewer layers.

Teams building at scale won't choose between twenty frameworks. They'll pick integrated stacks where orchestration fades into models and deployment infrastructure.

Where This Goes

Framework Wars End Through Consolidation, Not Competition

The agent framework explosion of 2023-2024 is collapsing faster than anyone expected. Microsoft merged AutoGen with Semantic Kernel in October. LangChain pivoted from chains to LangGraph. CrewAI went from interesting to $18M Series A with 60% Fortune 500 penetration.

Foundation models are absorbing what frameworks used to provide. O1's native reasoning, Gemini 2.0's built-in tool use. We think this reveals something deeper than market consolidation. Agent infrastructure is maturing by disappearing into fewer layers.

Teams building at scale won't choose between twenty frameworks. They'll pick integrated stacks where orchestration fades into models and deployment infrastructure.

TAKE NOTE

Observable signal: Enterprise adoption hits 79% with insurance sector showing 325% increase in full AI deployment, while foundation models absorb orchestration capabilities that frameworks used to provide separately.

Trajectory ahead: Expect vertical integration between models and deployment infrastructure within six months, reducing explicit orchestration layers as capabilities move into foundation models themselves.

Underlying dynamic: Running web agents at scale reveals that composition flexibility matters less than integrated reliability when systems operate continuously against adversarial environments.

Building implications: Teams will shift from framework selection to stack integration decisions, optimizing for fewer moving parts rather than maximum flexibility in agent architecture.

TinyFish's read: Consolidation changes what building agents means—less composition, more integration. The path to quiet tech runs through fewer layers, not better orchestration.

From the Labs

Hybrid Workflows Beat Full Autonomy

Stanford and CMU researchers measured autonomous agents at 88% faster and 90% cheaper than humans, but success rates dropped 32-50% on complex tasks. Hybrid systems routing structured work to agents while keeping judgment calls with humans improved performance 68.7%, revealing where autonomy actually delivers value.

Where do agents actually win?

Speed and cost on structured execution, not reliability on ambiguous, high-stakes decisions.

What does architecture look like?

Explicit routing logic that sends deterministic tasks to agents, judgment calls to humans.

Modular Agents Enable Independent Optimization

Plan-and-Act splits agents into separate Planner and Executor components, achieving 57.58% success on WebArena-Lite and 81.36% on WebVoyager. Using synthetic data for training, the framework lets you optimize planning and execution independently without rebuilding the entire system when requirements change.

Why split planning from execution?

Swap strategies without touching the planner, focusing optimization where it yields actual returns.

How do you train specialized agents?

Synthetic data generation eliminates expensive human annotation, making domain-specific tuning practical.

Direct Agent Communication Beats Centralized Routing

Anemoi's semi-centralized architecture lets agents communicate directly rather than routing everything through a single planning LLM. Using GPT-4.1-mini as planner, it hit 52.73% accuracy on GAIA benchmark, beating the strongest open-source baseline by 9.09%. Better coordination protocols outperform throwing larger models at orchestration.

What drives performance gains?

Coordination architecture, not model scale—smaller models with better protocols beat larger centralized systems.

Where does this lead?

Standardized protocols enabling heterogeneous agents to collaborate without tight coupling between components.

Execution Constraints Belong In Planning

REMAP integrates execution time estimation into multi-agent path finding, accounting for kinodynamic constraints, communication latency, and controller variability. Testing with 300 agents on benchmark maps showed 20% improvement in solution quality over baselines that treat execution as deterministic, making plans that survive contact with reality.

Why model execution during planning?

Real-world constraints directly impact whether agents hit deadlines, not just theoretical optimality.

Which deployments benefit most?

Time-sensitive operations in logistics and manufacturing gain reliability by incorporating actual execution behavior.

From the Labs

Hybrid Workflows Beat Full Autonomy

Stanford and CMU researchers measured autonomous agents at 88% faster and 90% cheaper than humans, but success rates dropped 32-50% on complex tasks. Hybrid systems routing structured work to agents while keeping judgment calls with humans improved performance 68.7%, revealing where autonomy actually delivers value.

Where do agents actually win?

Speed and cost on structured execution, not reliability on ambiguous, high-stakes decisions.

What does architecture look like?

Explicit routing logic that sends deterministic tasks to agents, judgment calls to humans.

From the Labs

Modular Agents Enable Independent Optimization

Plan-and-Act splits agents into separate Planner and Executor components, achieving 57.58% success on WebArena-Lite and 81.36% on WebVoyager. Using synthetic data for training, the framework lets you optimize planning and execution independently without rebuilding the entire system when requirements change.

Why split planning from execution?

Swap strategies without touching the planner, focusing optimization where it yields actual returns.

How do you train specialized agents?

Synthetic data generation eliminates expensive human annotation, making domain-specific tuning practical.

From the Labs

Direct Agent Communication Beats Centralized Routing

Anemoi's semi-centralized architecture lets agents communicate directly rather than routing everything through a single planning LLM. Using GPT-4.1-mini as planner, it hit 52.73% accuracy on GAIA benchmark, beating the strongest open-source baseline by 9.09%. Better coordination protocols outperform throwing larger models at orchestration.

What drives performance gains?

Coordination architecture, not model scale—smaller models with better protocols beat larger centralized systems.

Where does this lead?

Standardized protocols enabling heterogeneous agents to collaborate without tight coupling between components.

From the Labs

Execution Constraints Belong In Planning

REMAP integrates execution time estimation into multi-agent path finding, accounting for kinodynamic constraints, communication latency, and controller variability. Testing with 300 agents on benchmark maps showed 20% improvement in solution quality over baselines that treat execution as deterministic, making plans that survive contact with reality.

Why model execution during planning?

Real-world constraints directly impact whether agents hit deadlines, not just theoretical optimality.

Which deployments benefit most?

Time-sensitive operations in logistics and manufacturing gain reliability by incorporating actual execution behavior.

Quiet Tech That Compounds

The agent ecosystem runs on infrastructure nobody photographs for launch announcements. Foundation model releases generate headlines. Flashy demos get retweeted. Production systems depend on decidedly unglamorous capabilities.

Durable execution engines that recover from failures. Cost attribution tooling that tracks multi-step workflow economics. Evaluation frameworks that measure stability alongside accuracy. Semantic layers preventing hallucinations. Memory systems making agents feel adaptive instead of amnesiac. Guardrails catching failures before users see them.

None of this generates headlines. All of it determines whether agents actually work when they leave the demo environment. Here's what serious builders are investing in.

Quiet Tech That Compounds

The agent ecosystem runs on infrastructure nobody photographs for launch announcements. Foundation model releases generate headlines. Flashy demos get retweeted. Production systems depend on decidedly unglamorous capabilities.

Durable execution engines that recover from failures. Cost attribution tooling that tracks multi-step workflow economics. Evaluation frameworks that measure stability alongside accuracy. Semantic layers preventing hallucinations. Memory systems making agents feel adaptive instead of amnesiac. Guardrails catching failures before users see them.

None of this generates headlines. All of it determines whether agents actually work when they leave the demo environment. Here's what serious builders are investing in.

Workflow Infrastructure

Durable Execution Recovers From Stochastic Failures

Traditional orchestrators expect deterministic steps. Agents break that assumption constantly. Purpose-built engines like Zipline, Inngest, and Hatchet combine queue, state store, and retry logic. They recover from failure points without expensive re-execution. Production viability depends on this infrastructure.

Cost Attribution

Workflow-Level Economics Replace Billing Guesswork

A single agent action fans out. Multiple model calls, tool invocations, RAG lookups. Each has different cost curves. Specialized tooling like Paid abstracts usage into workflow-level metrics. You get actual margin tracking instead of spreadsheet estimates. Financially essential, photographically boring.

Evaluation Standards

Holistic Benchmarks Test Beyond Accuracy Theater

CLASSic evaluates cost, latency, stability, and security alongside accuracy. Enterprise deployment requires knowing if agents are prohibitively expensive, slow to respond, or easily exploited. Comprehensive frameworks test 500-1000 scenarios. They include failure modes, not just happy paths.

Data Foundation

Semantic Layers Prevent Hallucination Chaos

LLMs accessing unorganized data lakes generate unreliable outputs. Structured catalogs and semantic layers define metrics, hierarchies, relationships. They give models the context needed for accurate queries. RAG shines when paired with this foundation, not raw data dumps.