Share

Core Concepts

Compared With What? Checking Whether Agent Memory and Skill Libraries Earn Their Tokens

By Sable Whitford— October 23, 2026

Feature image for article: Compared With What? Checking Whether Agent Memory and Skill Libraries Earn Their Tokens

Agent frameworks keep bolting on memory, self-reflection, and libraries of saved skills, and the benchmark numbers usually go up. Before you credit the add-on, ask what the plain agent was allowed to spend. When researchers gave a basic web agent a few extra steps instead, it kept pace with the enhanced versions and cost less to run. A cancer-records study found the same skill library worked about as well with the agent's planning and tool-calling stripped away. Then there are tasks spanning several sessions, where memory should finally earn its keep.

Core Concepts

Compared With What? Checking Whether Agent Memory and Skill Libraries Earn Their Tokens

By Sable Whitford— October 23, 2026

Agent frameworks keep bolting on memory, self-reflection, and libraries of saved skills, and the benchmark numbers usually go up. Before you credit the add-on, ask what the plain agent was allowed to spend. When researchers gave a basic web agent a few extra steps instead, it kept pace with the enhanced versions and cost less to run. A cancer-records study found the same skill library worked about as well with the agent's planning and tool-calling stripped away. Then there are tasks spanning several sessions, where memory should finally earn its keep.

Practical Walkthrough

The Other Agent Got the Message. Did It Change Anything?

By Sable Whitford— October 23, 2026

Feature image for article: The Other Agent Got the Message. Did It Change Anything?

Wire AI agents together and every hand-off lands in a transcript, which people tend to read as proof the messages mattered. It's a delivery receipt. To find out whether Agent B actually used what Agent A told it, change one fact in one message and see whether B's next move shifts. One study ran that test 793 times. Depending on where the noise bar sat, 39% to 75% of tests found no message driving the decision. And for the notes agents share, saving everything did worse than saving nothing.

Practical Walkthrough

The Other Agent Got the Message. Did It Change Anything?

By Sable Whitford— October 23, 2026

Wire AI agents together and every hand-off lands in a transcript, which people tend to read as proof the messages mattered. It's a delivery receipt. To find out whether Agent B actually used what Agent A told it, change one fact in one message and see whether B's next move shifts. One study ran that test 793 times. Depending on where the noise bar sat, 39% to 75% of tests found no message driving the decision. And for the notes agents share, saving everything did worse than saving nothing.

Further Reading

Ironies of Automation, Reread for AIBainbridge's 1983 plant-control paper, applied to AI: machines take routine work, people get the hard cases with less practice.

When the Job Shrinks to WatchingGroups Bainbridge's ironies into lost skills, monitoring strain and worker attitudes, including the status lost when work becomes watching.

Quick links

How Often a Person Steps In

Finishing the Task Without Learning the Library

What Students Kept Once the Chatbot Left

Cutting the Referral Out of the Journey

Tool Convention Mismatches

Fluent but unexecutable. That describes a tool call that's miserable to debug, because the transcript looks fine. The model understood the user, picked the right tool and chose a sensible value. It wrote that value in the user's language or the user's format. The API either rejects it or, worse, accepts it and does the wrong thing.

Standards bodies have spent decades documenting how people write place names, personal names, dates, numbers and money. Each card covers one mismatch and where you could catch it: schema, validator, parser, intake or model.

So far, researchers have measured only place-name failures in agents. The other cards draw on the standards themselves.

Tool Convention Mismatches

Fluent but unexecutable. That describes a tool call that's miserable to debug, because the transcript looks fine. The model understood the user, picked the right tool and chose a sensible value. It wrote that value in the user's language or the user's format. The API either rejects it or, worse, accepts it and does the wrong thing.

Standards bodies have spent decades documenting how people write place names, personal names, dates, numbers and money. Each card covers one mismatch and where you could catch it: schema, validator, parser, intake or model.

So far, researchers have measured only place-name failures in agents. The other cards draw on the standards themselves.

Anchor study

Right tool, right intent, wrong language

Lost in Execution (ACL 2026) translated benchmark queries into Chinese, Hindi and Igbo but left every tool schema in English. Models usually understood the request and picked the right function. Then they filled the city field with 纽约市 instead of "New York City." The authors put the fix at the interface between model and tool.

Place names

One city, several valid names, one accepted

UN naming guidance separates a place's local-script name, its official romanization and any traditional English name. For one city that gives 北京, Beijing, and the older Peking. A location parameter should state which form it expects. Catch this in the schema description, or normalize with a place-name lookup before the call.

Mitigation risk

Translating arguments back can quietly change them

The paper tested three fixes: tell the model to use English, translate the query first, or translate the values afterward. All three reduced mismatches, and none reached English-level accuracy. One round trip through Chinese turned a queen-size bed into roomType="suite with a king-size bed". An enum keeps values legal. Catching the swap means checking the value against what the user asked for.

Name order

Which name comes first is a locale setting