--- title: "Foundations" canonical: "https://current.tinyfish.ai/issue/17/foundations" publication: "TinyFish" publisher: "TinyFish" language: "en-US" guide: "https://current.tinyfish.ai/llms.txt" permalink: "https://current.tinyfish.ai/issue/17/foundations" issue: 17 section: "foundations" description: "Conceptual clarity earned from building at scale" authors: ["Nora Kaplan","Rina Takahashi"] published: 2026-02-23 updated: 2026-02-16 articles: 18 --- # TinyFish > CURRENT is a journal about AI agents in production, for the engineering and product leaders who have to make them work. Drawing on experience building enterprise web agents, it explains why the demo succeeded and the deployment didn't, and what that means for choosing vendors, briefing the board, and avoiding mistakes others have already made. [Read on the website]() ## Issue 17 — Foundations Conceptual clarity earned from building at scale ### Context at Scale - [The Salience Problem at Million-Token Scale]() — By [Nora Kaplan]() · Published 2026-02-16 · Format: Context at Scale A pricing agent holds three years of product data in context—entire catalog, seasonal patterns, competitor moves. It recommends a discount strategy that seems reasonable until someone checks the logic: it's applying clearance patterns from eighteen months ago, when supply chain constraints created different market dynamics. The information was still there, 400,000 tokens back. Never forgotten. Just no longer relevant. ### Building with Memory - [What Happens When Context Windows Run Out of Room]() — By [Rina Takahashi]() · Published 2026-02-16 · Format: Building with Memory An agent conversation runs long enough and you hit the wall—the model can't hold everything anymore. Compression discards details that might matter later. External storage creates retrieval gaps. Structured relationships require infrastructure complexity. Context windows fill up predictably. The decisions that follow depend on what you're prepared to give up. ### Four Perspectives - **Hot Path, Cold Path: Agent Memory's Dual-Layer Standard** — [Source]() — Published 2026-02-16 Working memory stays fast and immediate. Retrieval happens separately. The architecture pattern that actually ships in production. - **Cache Misses Cost 10× More Than Hits in Agent Systems** — [Source]() — Published 2026-02-16 When agents generate 100× more tokens than humans, KV cache hit rates determine whether your infrastructure budget scales linearly or exponentially. - **AgentLeak: 68.8% of Multi-Agent Systems Leak Through Internal Channels** — [Source]() — Published 2026-02-16 Benchmark study reveals inter-agent messages leak at 57.8% while final outputs leak at 27.2%. Coordination between agents creates the vulnerability. - **What 306 Production Deployments Say About Agent Complexity** — [Source]() — Published 2026-02-16 MAP study finds 68% of agents execute ten steps or fewer, 70% rely on prompting over fine-tuning. Simplicity wins when systems ship. ### Memory Patterns Memory architecture for agents involves choosing between competing priorities: what you remember, how you organize it, when you retrieve it. Some patterns optimize for semantic similarity. Others compress aggressively or reason about relationships. None of these approaches are mutually exclusive. Production systems typically layer multiple patterns. Vector stores handle semantic recall, graphs manage relationship reasoning, summarization controls costs. The patterns below work as composable building blocks rather than competing options. Understanding what each optimizes for helps you decide which combinations actually serve your agent's work. - **Semantic Similarity as Memory** — [Source]() — Published 2026-02-16 Tag: Vector Stores Store conversations as embeddings, retrieve by meaning rather than exact text. Works beautifully for finding similar past interactions even when phrasing changes. Scales well, responds fast. But semantic similarity alone misses context: two conversations about "the Paris project" might mean completely different things. - **Condensing Transcripts Over Time** — [Source]() — Published 2026-02-16 Tag: Summarization Layers Use LLMs to periodically compress conversation history into rolling summaries. Achieves 89-95% compression while keeping context bounded. The risk: details that seem trivial now might matter later. Works when storage costs outweigh the occasional missed nuance in long-running interactions. - **Relationships Between Facts** — [Source]() — Published 2026-02-16 Tag: Graph Memory Organize memories as nodes and edges: people, places, events, temporal connections. Enables reasoning about how information relates across sessions. Zep's temporal graphs show 18.5% accuracy gains with 90% lower latency. Choose this when agents need to understand how facts connect and change. - **Dynamic Movement Between Layers** — [Source]() — Published 2026-02-16 Tag: Hybrid Storage Mem0 moves information between short-term and long-term storage based on usage patterns, recency, significance. Mimics how human memory actually forms: selective retention instead of compressing everything. Shows 26% quality improvements while cutting token costs by 90%. Balances sophistication with practical simplicity. - **Facts That Evolve** — [Source]() — Published 2026-02-16 Tag: Temporal Encoding Zep encodes entities and relationships in temporal knowledge graphs that update with each interaction. Tracks when facts change, invalidates outdated information while preserving history. Combines semantic embeddings, keyword search, and graph traversal for near-constant retrieval time regardless of memory size. - **Holding Everything in Prompt** — [Source]() — Published 2026-02-16 Tag: Context Windows Pass entire conversation history directly to the model without external storage. Eliminates retrieval complexity for short interactions. Models claiming 200,000 token windows become unreliable around 130,000. Context resets with each request, so no learning across sessions or persistent identity emerges. ### System Trade-offs Memory systems for AI agents force you to choose between competing goods. Every optimization creates a corresponding sacrifice somewhere else in the system. Faster retrieval means accepting lower accuracy. Stronger security means higher computational overhead. Lower token costs mean less comprehensive context. These tensions don't resolve—they define the boundaries of what's possible. What follows are six specific decision points where the trade-off becomes concrete. Each one shows what you gain and what you lose. Understanding them means knowing which sacrifices your application can tolerate and which ones break it entirely. - **Comprehensive Context Costs 26,000 Tokens Per Request** — [Source]() — Published 2026-02-16 Tag: Token Economics Full-context approaches hit 72.9% accuracy while burning through 26,000 tokens per request. Memory-optimized systems like Mem0 reach 66.9% accuracy at a fraction of that. Output tokens cost 3-8× more than input. Pick comprehensive understanding or sustainable economics. - **Voice AI Demands Sub-Second, Loses Six Points** — [Source]() — Published 2026-02-16 Tag: Response Latency Mem0 delivers 0.71s median latency versus full-context's 9.87s. That speed costs 6 percentage points of accuracy. Voice applications can't wait nine seconds for context assembly. You optimize for speed because the alternative makes your product unusable. - **Trust Decay Filters Legitimate Memories Too** — [Source]() — Published 2026-02-16 Tag: Decay Parameters Trust decay multiplies scores by 0.95 raised to age in days, filtering entries below 0.6. Set it aggressive and you block legitimate context. Set it permissive and subtle attacks slip through. Both choices degrade the system differently. - **HNSW Recall Consumes Three Times RAM** — [Source]() — Published 2026-02-16 Tag: Vector Storage HNSW achieves higher recall than IVF but consumes several times more memory. At 100 million vectors, 1536 dimensions requires 600GB. Drop to 768 dimensions and you fit 400 million vectors in the same RAM. Accuracy and density move opposite directions. - **Exact Search Breaks Past Small Datasets** — [Source]() — Published 2026-02-16 Tag: Search Algorithms FLAT provides brute-force exact search but can't scale beyond small datasets. Approximate k-NN returns results in milliseconds instead of seconds. Production systems configure HNSW or IVF to exceed 95% recall. Never 100%. That last five percent costs too much. - **MicroVM Security Adds 15-20% Computational Overhead** — [Source]() — Published 2026-02-16 Tag: Isolation Architecture MicroVMs eliminate shared kernel vulnerabilities through hardware virtualization. They also add 15-20% overhead. Containers are lightweight but fail when agents execute untrusted code. TEE infrastructure reduces breach liability by 85% while slowing execution and raising infrastructure spend. ## Other sections in Issue 17 - [Daily Brief]() - [Market Pulse]() - [Practitioner's Corner]() - [Echoes]() - [Vision]() ## More - [Publication guide]() - [Issue archive]()