--- title: "Foundations" canonical: "https://current.tinyfish.ai/issue/24/foundations" publication: "TinyFish" publisher: "TinyFish" language: "en-US" guide: "https://current.tinyfish.ai/llms.txt" permalink: "https://current.tinyfish.ai/issue/24/foundations" issue: 24 section: "foundations" description: "Conceptual clarity earned from building at scale" authors: ["Nora Kaplan","Rina Takahashi"] published: 2026-04-13 updated: 2026-04-09 articles: 18 --- # TinyFish > CURRENT is a journal about AI agents in production, for the engineering and product leaders who have to make them work. Drawing on experience building enterprise web agents, it explains why the demo succeeded and the deployment didn't, and what that means for choosing vendors, briefing the board, and avoiding mistakes others have already made. [Read on the website]() ## Issue 24 — Foundations Conceptual clarity earned from building at scale ### The Practical Piece - [The Classification Decision That Makes Agent Cost Models Fragile]() — By [Nora Kaplan]() · Published 2026-04-08 · Format: The Practical Piece Somewhere in a config file, a practitioner is deciding whether "extract pricing data from competitor pages" counts as reasoning or execution. A frontier model or a cheap model gets the job, and the cost architecture of the entire system rides on this single classification. No framework draws a formal boundary. ### The Argument - [Where Agentic AI's 40% Failure Rate Actually Lives]() — By [Rina Takahashi]() · Published 2026-04-08 · Format: The Argument AI captured 80% of global venture funding last quarter. Quarter-trillion dollars, one category. Meanwhile, Gartner projects that four in ten agentic AI projects won't survive to 2027. Both numbers are credible. Both are rational. And the gap between them is where actual deployments go quiet. ### Further Reading - **Multi-Agent Patterns That Survive Production (and Why Most Don't)** — [Source]() — Published 2026-04-08 Real failure data from multi-agent deployments: cascading hallucinations, latency traps, and which patterns actually hold up. - **Vendor Lock-in as Economic Liability: The Trust × Flexibility Framework** — [Source]() — Published 2026-04-08 Treats agentic AI vendor selection as a long-term cost structure commitment, not a technical preference you revisit later. - **What Agent Orchestration Frameworks Actually Commit You To** — [Source]() — Published 2026-04-08 Framework choice locks in observability, state management, and debugging costs that compound well past the initial build. - **Agent Cost Optimization as Architecture, Not Afterthought** — [Source]() — Published 2026-04-08 Names a 90% cost reduction from Plan-and-Execute patterns and argues model tiering belongs in design, not ops. - **The $4.63M Price Tag on Unsecured Agents** — [Source]() — Published 2026-04-08 Puts dollar figures on the downside: breach costs, red-team timelines, and blast-radius math for agents with real permissions. ### The Cost Stack The cost structure of agentic systems is genuinely strange. Per-token prices keep falling. Total bills keep climbing. The most expensive line item on paper often isn't the one that blows up your budget. And some of the biggest cost drivers have nothing to do with the models themselves. We mapped six categories that enterprises actually hit when deploying agents at scale. A few are obvious. Others are the kind of thing you discover three months into production, staring at an invoice that doesn't match your projections. Together, they form a picture of where agent budgets go that looks very different from where most teams expect. - **The Model You Pick Sets the Ceiling** — [Source]() — Published 2026-04-08 Tag: Model Inference Inference accounts for roughly 85% of enterprise AI spend. The price spread is staggering: Claude Opus 4.5 runs $25 per million output tokens; GPT-5 nano costs $0.05 per million input. Same model, different provider? Expect 10x variance. Choosing where to run is a budget decision disguised as a technical one. - **Falling Token Prices Hide a Rising Total Bill** — [Source]() — Published 2026-04-08 Tag: Token Volume Gartner puts the agentic multiplier at 5–30x more tokens per task compared to standard chat. Complex multi-agent workflows regularly blow past a million tokens on a single job. Per-token costs keep dropping. Volume is rising faster. The bill grows anyway. - **Context Windows Are Quietly Burning Your Tokens** — [Source]() — Published 2026-04-08 Tag: Context Bloat Every token in the context window gets billed on every LLM call. One agent resending a 25,000-token document across five reasoning steps burns 125,000 input tokens on a single task. MCP tool metadata alone can eat 40–50% of available context. RAG and prompt caching cut this waste by 60–90%. - **Coordination Tax Rivals the Agents' Own Costs** — [Source]() — Published 2026-04-08 Tag: Orchestration Overhead When agents hand off to other agents, the coordination layer becomes its own cost surface. A single edge case can trigger retry chains costing 50x the normal path. Framework choice compounds this: in benchmarks, some orchestration frameworks consumed nearly double the tokens of others running identical tasks. - **Below the API Layer, a Whole Other Bill** — [Source]() — Published 2026-04-08 Tag: Infrastructure Costs Kubernetes provisioning, vector databases, memory backends, retry queues, load balancers. All separate line items, all easy to forget. Self-hosting above 100K requests per day saves 60–80% versus API pricing. Below that threshold, fixed GPU costs make self-hosting the more expensive option. - **Observability Bills Punish You for Actually Looking** — [Source]() — Published 2026-04-08 Tag: Observability Spend Agent observability runs $25–$500/month at startup scale, $2K–$10K+ for enterprise volumes. The billing math gets adversarial fast: a 10-span trace with one evaluation score costs 12 billable units. Some teams have resorted to sampling 0.1% of traffic just to keep observability affordable. ### Patterns That Pay A single user request to an agent can trigger a cascade of LLM calls for planning, tool selection, execution, and verification. Five or more inference round-trips where a chat completion needs one. At scale, that multiplier becomes a budget line worth staring at. Documented patterns for cutting these costs exist, and several carry production data or peer-reviewed evidence behind them. But each one trades savings for a specific kind of brittleness. The five cards below name what works, what it actually delivers, and exactly where it starts to crack. - **Let a Frontier Model Think, Not Do** — [Source]() — Published 2026-04-08 Tag: Cost Pattern Plan-and-Execute splits the expensive reasoning from the cheap labor. A frontier model architects the strategy; distilled or smaller models carry out each step. Production deployments report 80% cost cuts, stretching toward 90% with full distillation. The catch: distilled executors fail on rare edge cases outside their training data. Precisely the cases where failure costs the most. - **Scope Isolation Starves Each Agent of Bloat** — [Source]() — Published 2026-04-08 Tag: Token Reduction Monolithic agent architectures stuff every subagent with the full conversation history. Scope isolation walls off each agent's context to only what it needs. LangChain's multi-agent documentation shows a 67% token reduction per request through this boundary enforcement. The cost: each handoff between isolated agents adds 500ms–1.5s of summarization latency and risks dropping context that mattered. - **Match the Model Tier to the Task** — [Source]() — Published 2026-04-08 Tag: Model Routing A frontier reasoning call can cost 190x more than a small model handling equivalent work. Routing each agent step by complexity, sending simple extraction to cheap models and reserving heavy inference for hard decisions, yields 30–70% savings across practitioner reports. The fragility: you now depend on a routing classifier that adds its own latency, and a single provider outage can break a multi-model pipeline in ways a single-model system never would. - **Stop Resending the Same Tokens Every Call** — [Source]() — Published 2026-04-08 Tag: Prompt Caching Agents hammer the same system prompts, tool schemas, and few-shot examples on every invocation. Provider-level prompt caching (Anthropic discounts cached tokens by 90%) eliminates that redundancy. Practitioners report 45–80% cost reductions on cache-eligible workloads. The caveat worth remembering: when tool schemas update, stale cache hits silently degrade output quality unless you manage invalidation explicitly. - **Cache the Plan, Not Just the Prompt** — [Source]() — Published 2026-04-08 Tag: Plan Caching The strongest evidence in this set comes from a peer-reviewed preprint: caching and adapting prior execution plans across five agent workloads cut costs by 50% and latency by 27%, retaining 96.6% accuracy. This goes beyond prompt-prefix caching to reuse the planner's actual reasoning. Requires a Plan-Act architecture to work, and effectiveness drops sharply on genuinely novel tasks where no prior plan applies. ## Other sections in Issue 24 - [Daily Brief]() - [Market Pulse]() - [Practitioner's Corner]() - [Echoes]() - [Vision]() ## More - [Publication guide]() - [Issue archive]()