Share

Context at Scale

The Salience Problem at Million-Token Scale

By Nora Kaplan— February 16, 2026

Feature image for article: The Salience Problem at Million-Token Scale

A pricing agent holds three years of product data in context—entire catalog, seasonal patterns, competitor moves. It recommends a discount strategy that seems reasonable until someone checks the logic: it's applying clearance patterns from eighteen months ago, when supply chain constraints created different market dynamics. The information was still there, 400,000 tokens back. Never forgotten. Just no longer relevant.

Context windows expanding from thousands to millions of tokens create memory management problems that didn't exist at smaller scale. Complete memory creates salience problems. And information that should have been forgotten persists, contaminating future inferences.

Context at Scale

The Salience Problem at Million-Token Scale

By Nora Kaplan— February 16, 2026

A pricing agent holds three years of product data in context—entire catalog, seasonal patterns, competitor moves. It recommends a discount strategy that seems reasonable until someone checks the logic: it's applying clearance patterns from eighteen months ago, when supply chain constraints created different market dynamics. The information was still there, 400,000 tokens back. Never forgotten. Just no longer relevant.

Context windows expanding from thousands to millions of tokens create memory management problems that didn't exist at smaller scale. Complete memory creates salience problems. And information that should have been forgotten persists, contaminating future inferences.

Building with Memory

What Happens When Context Windows Run Out of Room

By Rina Takahashi— February 16, 2026

Feature image for article: What Happens When Context Windows Run Out of Room

An agent conversation runs long enough and you hit the wall—the model can't hold everything anymore. Compression discards details that might matter later. External storage creates retrieval gaps. Structured relationships require infrastructure complexity. Context windows fill up predictably. The decisions that follow depend on what you're prepared to give up.

Building with Memory

What Happens When Context Windows Run Out of Room

By Rina Takahashi— February 16, 2026

An agent conversation runs long enough and you hit the wall—the model can't hold everything anymore. Compression discards details that might matter later. External storage creates retrieval gaps. Structured relationships require infrastructure complexity. Context windows fill up predictably. The decisions that follow depend on what you're prepared to give up.

Four Perspectives

Hot Path, Cold Path: Agent Memory's Dual-Layer Standard

Working memory stays fast and immediate. Retrieval happens separately. The architecture pattern that actually ships in production.

Cache Misses Cost 10× More Than Hits in Agent Systems

When agents generate 100× more tokens than humans, KV cache hit rates determine whether your infrastructure budget scales linearly or exponentially.

AgentLeak: 68.8% of Multi-Agent Systems Leak Through Internal Channels

What 306 Production Deployments Say About Agent Complexity

Four Perspectives

Hot Path, Cold Path: Agent Memory's Dual-Layer StandardWorking memory stays fast and immediate. Retrieval happens separately. The architecture pattern that actually ships in production.

Cache Misses Cost 10× More Than Hits in Agent SystemsWhen agents generate 100× more tokens than humans, KV cache hit rates determine whether your infrastructure budget scales linearly or exponentially.

Quick links

AgentLeak: 68.8% of Multi-Agent Systems Leak Through Internal Channels

What 306 Production Deployments Say About Agent Complexity

Memory Patterns

Memory architecture for agents involves choosing between competing priorities: what you remember, how you organize it, when you retrieve it. Some patterns optimize for semantic similarity. Others compress aggressively or reason about relationships. None of these approaches are mutually exclusive.

Production systems typically layer multiple patterns. Vector stores handle semantic recall, graphs manage relationship reasoning, summarization controls costs. The patterns below work as composable building blocks rather than competing options. Understanding what each optimizes for helps you decide which combinations actually serve your agent's work.

Memory Patterns

Memory architecture for agents involves choosing between competing priorities: what you remember, how you organize it, when you retrieve it. Some patterns optimize for semantic similarity. Others compress aggressively or reason about relationships. None of these approaches are mutually exclusive.

Production systems typically layer multiple patterns. Vector stores handle semantic recall, graphs manage relationship reasoning, summarization controls costs. The patterns below work as composable building blocks rather than competing options. Understanding what each optimizes for helps you decide which combinations actually serve your agent's work.

Vector Stores

Semantic Similarity as Memory

Store conversations as embeddings, retrieve by meaning rather than exact text. Works beautifully for finding similar past interactions even when phrasing changes. Scales well, responds fast. But semantic similarity alone misses context: two conversations about "the Paris project" might mean completely different things.

Summarization Layers

Condensing Transcripts Over Time

Use LLMs to periodically compress conversation history into rolling summaries. Achieves 89-95% compression while keeping context bounded. The risk: details that seem trivial now might matter later. Works when storage costs outweigh the occasional missed nuance in long-running interactions.

Graph Memory

Relationships Between Facts

Organize memories as nodes and edges: people, places, events, temporal connections. Enables reasoning about how information relates across sessions. Zep's temporal graphs show 18.5% accuracy gains with 90% lower latency. Choose this when agents need to understand how facts connect and change.

Hybrid Storage

Dynamic Movement Between Layers

Mem0 moves information between short-term and long-term storage based on usage patterns, recency, significance. Mimics how human memory actually forms: selective retention instead of compressing everything. Shows 26% quality improvements while cutting token costs by 90%. Balances sophistication with practical simplicity.

Temporal Encoding

Facts That Evolve

Zep encodes entities and relationships in temporal knowledge graphs that update with each interaction. Tracks when facts change, invalidates outdated information while preserving history. Combines semantic embeddings, keyword search, and graph traversal for near-constant retrieval time regardless of memory size.

Context Windows

Holding Everything in Prompt

Pass entire conversation history directly to the model without external storage. Eliminates retrieval complexity for short interactions. Models claiming 200,000 token windows become unreliable around 130,000. Context resets with each request, so no learning across sessions or persistent identity emerges.

System Trade-offs

Memory systems for AI agents force you to choose between competing goods. Every optimization creates a corresponding sacrifice somewhere else in the system.

Faster retrieval means accepting lower accuracy. Stronger security means higher computational overhead. Lower token costs mean less comprehensive context. These tensions don't resolve—they define the boundaries of what's possible.

What follows are six specific decision points where the trade-off becomes concrete. Each one shows what you gain and what you lose. Understanding them means knowing which sacrifices your application can tolerate and which ones break it entirely.

System Trade-offs

Memory systems for AI agents force you to choose between competing goods. Every optimization creates a corresponding sacrifice somewhere else in the system.

Faster retrieval means accepting lower accuracy. Stronger security means higher computational overhead. Lower token costs mean less comprehensive context. These tensions don't resolve—they define the boundaries of what's possible.

What follows are six specific decision points where the trade-off becomes concrete. Each one shows what you gain and what you lose. Understanding them means knowing which sacrifices your application can tolerate and which ones break it entirely.

Token Economics

Comprehensive Context Costs 26,000 Tokens Per Request

Full-context approaches hit 72.9% accuracy while burning through 26,000 tokens per request. Memory-optimized systems like Mem0 reach 66.9% accuracy at a fraction of that. Output tokens cost 3-8× more than input. Pick comprehensive understanding or sustainable economics.

Response Latency

Voice AI Demands Sub-Second, Loses Six Points

Mem0 delivers 0.71s median latency versus full-context's 9.87s. That speed costs 6 percentage points of accuracy. Voice applications can't wait nine seconds for context assembly. You optimize for speed because the alternative makes your product unusable.

Decay Parameters

Trust Decay Filters Legitimate Memories Too

Trust decay multiplies scores by 0.95 raised to age in days, filtering entries below 0.6. Set it aggressive and you block legitimate context. Set it permissive and subtle attacks slip through. Both choices degrade the system differently.

Vector Storage

HNSW Recall Consumes Three Times RAM

HNSW achieves higher recall than IVF but consumes several times more memory. At 100 million vectors, 1536 dimensions requires 600GB. Drop to 768 dimensions and you fit 400 million vectors in the same RAM. Accuracy and density move opposite directions.