--- title: "Compared With What? Checking Whether Agent Memory and Skill Libraries Earn Their Tokens" canonical: "https://current.tinyfish.ai/issue/52/foundations/article/52951/compared-with-what-checking-whether-agent-memory-and" publication: "TinyFish" publisher: "TinyFish" language: "en-US" guide: "https://current.tinyfish.ai/llms.txt" format: "Core Concepts" author: "Sable Whitford" author_url: "https://current.tinyfish.ai/team/201" issue: 52 section: "foundations" section_name: "Foundations" published: 2026-10-23 updated: 2026-10-24 description: "When agents get matched budgets, added memory and skill libraries often buy little; three questions test whether the machinery earns its tokens." word_count: 1006 source_links: 8 --- # Compared With What? Checking Whether Agent Memory and Skill Libraries Earn Their Tokens *By [Sable Whitford]()* *October 23, 2026* [Read on the website]() Every component you add to a production system is one more thing that can page you at 3 AM, and agents are no exception. So when a framework ships with memory, a reflection step (the agent reviews its own work), or a library of saved skills, and some benchmark says success went up, my first question is what the plain version was allowed to spend. Until somebody answers that, I'm not giving the add-on credit for anything. Two terms first. An agent works in **steps**, which are single actions like a click or a typed search. It also burns **tokens**, the units AI usage is measured and billed in. [Hajimiri et al.](https://arxiv.org/html/2606.15017v2) point out that earlier comparisons gave the plain and enhanced agents the same number of steps, while the add-on spent extra tokens off to the side, writing down routines or generating code. The enhanced agent got more compute, and then it got the credit for doing better. So the researchers [rebalanced](https://sinahmr.github.io/budget-constrained-web-agents/). On WebArena, a set of mock websites built for testing agents (a store, a forum, an admin panel, a code-hosting site), they ran three published add-ons: saved routines, self-written Python tools, and a bank of lessons the agent draws from past successes and failures. The plain agent got 15 steps to the add-ons' 10, plus some trimming of repeated page text, as its share of that overhead; the authors call the match approximate. The token column below counts what each version actually spent, side calls included. Both pairs ran on Google's Gemini 3 Flash, and they're two examples from a paper that spans three models and four sites. | Site | Version | Success | Tokens per task | |---|---|---|---| | Shopping | Plain agent | 47.8% | 46K | | Shopping | Skill Induction (Python tools) | 44.7% | 83K | | Forum | Plain agent | 47.5% | 72K | | Forum | Workflow Memory (saved routines) | 47.5% | 87K | The plain agent kept up, and even with the extra steps it came in cheaper. On shopping it used a little over half the add-on's tokens. The authors limit the claim to add-ons that learn during the run, on tasks that are largely independent of one another. A skill library you build once and reuse for months spreads its cost very differently. There's a second question underneath all this. When a skill library does help, is the gain coming from the knowledge itself, or from the agent machinery wrapped around it? A [myeloma study](https://www.nature.com/articles/s41698-026-01733-2) (myeloma is a blood cancer) tested that directly. Researchers wrote 42 clinical procedures for summarizing patient records and ran them on the same model two ways: as a full agent that planned, called tools, and ran clinical calculators, and as a simpler system that put the procedures in the prompt and repeatedly pulled relevant chunks of the record. Across 469 patient questions with expert-adjudicated answers, the agent matched the experts 80.1% of the time and the simpler system 79.0%, a gap within chance. Wrapped around the same procedures, the agent machinery added almost nothing overall. The agent did about 10 points better on the longest records, but that's ten patients in an exploratory analysis, nowhere near enough to sign a contract on. Neither the paper nor its code reports what each setup cost to run. Both had the same context limit (the cap on how much text the model can read at once), but a shared cap tells you little about the bill. That's the number I'd want next. Both studies used tasks that start fresh, which is why extra steps can stand in for memory there: whatever the agent needs is somewhere in the current session, and more clicking can find it. [MemoryArena](https://arxiv.org/html/2602.16313v2) chains tasks across sessions instead. Buy a camera body in one session, then later buy a lens that fits it. Which body you bought last week isn't on any page the agent can reach, so extra steps won't turn it up. On the shopping tasks, results were close to zero across the board. Keeping the full history in front of the model produced a fully valid bundle (a set of purchases that all fit together) 1% of the time. The memory systems Mem0 and ReasoningBank managed 0%; they finished pieces along the way but never the whole set. Budgets weren't matched in this study, so it can't settle memory versus more steps, and the 0% doesn't mean memory is pointless. This is the kind of task where memory ought to earn its cost, and current systems can't do it yet. So before you pay for the extra machinery (and the extra things that can break), put the first two questions to the vendor and the third to your own team: - What was the plain version allowed to spend, and could it have spent the add-on's overhead on more steps? - Does the skill library still help with the agent wrapper stripped off? - Do our tasks actually depend on earlier sessions, or does each one start clean? ## Things to follow up on... - **ReasoningBank's own accounting:** The original [ReasoningBank paper](https://arxiv.org/html/2509.25140) did publish token costs, roughly 4.3% more tokens for a 20.5% performance gain, but it never handed that overhead to a no-memory agent as extra steps, so the comparison is still worth running. - **Memory in a clinical sandbox:** In [FHIR-AgentEval](https://www.nature.com/articles/s41746-026-03348-0), memory distilled from earlier practice runs improved success on 43 healthcare-records tasks, though the memory configurations generally used more tokens than the bare baseline and budgets were never equalized. - **What a "skill" actually is:** The myeloma study's released [skill registry code](https://raw.githubusercontent.com/jomoll/onco-agent/master/src/skills/registry.py) shows its 42 procedures are written instruction packages rendered into prompt text, which is a very different thing from the executable Python skills tested on WebArena. - **Conversation recall versus later action:** [Mem0's original study](https://arxiv.org/html/2504.19413) reported LoCoMo scores of roughly 67–68% for answering questions about past conversations, a useful contrast with its 0% bundle completion on MemoryArena, even though the two setups weren't matched. ## More - [Foundations — Issue 52]() - [Issue 52]() - [Publication guide]()