How Persistent Memory Moves Through an Agent System

An agent system uses something from an earlier session to shape a later one. What actually moves between the two?

Three stages: a write, a retrieval, and an assembly into the model's prompt. All three are application code — decisions someone made and recorded in a configuration file, not capabilities the model arrives with. A paper presented at ACL 2026 makes each stage unusually legible, because the researchers put adversarial content through all three exactly as designed, and nothing malfunctioned.

Writing to memory

Frameworks like LlamaIndex and LangGraph supply the infrastructure for persistent memory. The policy — what's worth remembering, when to store it, in what form — lives in prompts and application configuration. A write can fire when the message buffer fills, when the model calls a memory-storage tool, or on a background schedule. What lands in storage might be extracted facts, raw message batches, JSON documents, images. The framework indexes it without judging it.

In the ACL experiment, the agent accepted an uploaded photograph of a living room and indexed it using CLIP embeddings: numerical representations that capture what an image is about in a form you can compare against later queries. Before uploading, the researchers had shifted the image's pixel values by at most eight units on a 0–255 scale, invisible to anyone looking at it, with the shifts chosen to pull the embedding toward a target concept, a specific furniture brand. The write stage did what it exists to do: store the item, compute its embedding, make it findable.

Retrieving by relevance

A new session opens and the agent needs context from earlier ones. Current implementations typically embed the user's query and search stored memories by cosine similarity, meaning how close two of those numerical representations sit in meaning-space, then return a configured number of top matches.

The perturbation was designed against that mechanism. The researchers optimized it across a distribution of plausible future queries in the target product category, so the poisoned image scored high when a relevant question came in and low otherwise. It sat in storage doing nothing until a conversation made it relevant.

Where retrieved content enters planning

Retrieved memories get placed into the messages sent to the model — under a heading, inside a labeled tag, interpolated into the system message — alongside the current instructions and the user's request. An earlier Foundations piece described the context window as working memory and selective retrieval as a way of keeping it relevant. This is the physical handoff underneath that description: application code picks stored objects, ranks them, and drops the winners into the prompt.

Nothing in that arrangement tells the model that recalled text carries less authority than fresh text. In the experiment, once the poisoned image entered the planning context, the agent's reasoning loop began referring to the target brand and category, which appeared nowhere in the user's request. Auditing the plan trace, the record of the agent's intermediate reasoning steps, found those references directly in 78.3% of successful memory-mediated attacks. Remove the poisoned memory, rerun the plan, and the recommendations change.

Sanitizing content at the write stage, on its own, cut attack success roughly in half. Getting it below 10% required checking again here, where stored text becomes something the model reads as input. The exposure concentrates where memory enters planning, not where it enters storage.

Store an observation, retrieve it when it looks relevant, feed it into planning. That sequence is what persistent memory is for, and unmodified, it is also the attack. Continuity means past content shaping present behavior; so does influence. What travels through the pipeline gets settled upstream, by defaults that most deployments accept once and never revisit.