Can the state be put back
From 2020 to 2022 she worked on availability features inside IBM's Db2 engine, maintaining C++ backup, restore and recovery components across six releases. Her CV names two projects: handling table-space failure during backup and recovery in Db2 11.5.7, and optimizing storage for recovery objects in 12.1. Recovery objects are the things that have to still exist if going back is going to be an option at all — backup images, transaction logs, load-copy files.
Recovery work asks one question and asks it with more rigor than almost anything else in our field. Can the correct state be put back? Rebuild a database at 4 AM and you will find that no other question is admitted into the room.
Which is why it hears nothing at all when every write commits exactly as instructed and the outcome is still wrong. That failure doesn't live in the state. It lives in the order of decisions that produced the state, and a recovery object has no column for that.
Reading the tape
Pan left industry engineering for research, and in 2025 co-authored MAST at NeurIPS: fourteen ways multi-agent systems fail, pulled out of traces. A trace is the tape: every message, every tool call, in order.
One of the fourteen is Reasoning-Action Mismatch. In a trace from HyperAgent, a code-repair system, the navigator agent reasoned its way to a solution and then sent the planner something else. A pass/fail benchmark score will tell you the task failed and stop there. It can't carry that finding, because the finding is a comparison between what the agent concluded and what it transmitted, and the score keeps neither half.
The taxonomy comes with its own deafness, and the paper is straight about it. Six experts hand-labeled 150 traces to derive the categories; an OpenAI o1 model labeled the rest against those definitions. The authors call the result non-exhaustive, which is the polite way of saying that a failure with no category in the schema simply doesn't show up in the data.
The corpus itself came from seven open-source frameworks and eight research task sets. One of those sets, ProgramDev, is thirty programming exercises: Tic-Tac-Toe, Chess, Sudoku. Not one deployed production system appears in the table. Nobody cut a corner. That's what a trace corpus is: you collect from systems that emit traces in places researchers can actually reach, and the real deployments are proprietary and quiet.
Asking the people who ship
The production study begins from that quiet. Twenty interviews, thirty to ninety minutes each, then a 47-question survey with 306 valid responses. Pan describes the method without dressing it up:
"To understand production agents, we went to people actually shipping them to the real-world."
Self-report is the instrument I spent two decades discounting on principle, and it picked up two things no tape will ever hold. Why teams build their own benchmarks: public ones rarely fit bespoke production tasks, and that judgment is made in a meeting, not in a log. And how a team decides whether swapping the model underneath a working deployment is safe, which is a call made nowhere the system can see itself making it.
Precision and coverage are separate axes, and the population you have instrumented best is almost never the population you most need to be right about. So what is your current instrument physically incapable of recording, and how much resolution would you trade to find out?
-
Correctness and safety separate: A joint Singapore and Korea AI Safety Institute evaluation of tool-using agents found individual correctness scores as high as 90% while fully correct-and-safe trajectories ran between 15% and 35% — a gap only visible when you inspect the whole trajectory rather than the final answer.
-
Nobody publishes their traces: The FAccT 2026 AI Agent Index examined public documentation for 30 prominent deployed agents and found detailed action traces for ten, agent-specific system cards for four, and no public third-party testing information for 23, which is roughly why Pan had to go interview people instead.
-
When the baseline disappears: METR changed its developer productivity experiment design after developers increasingly declined AI-free work, an instrument failure of exactly the kind worth watching for in your own evaluation setup.
-
Tools shifted from reading to acting: A preprint monitoring 177,436 public MCP tools reports action tools rising from 27% to 65% of sampled usage over sixteen months, though its public-repository sample says nothing about private enterprise connectors.