Melissa Pan's Three Evidence Problems

A Db2 update record carries typed headers identifying the operation, the transaction, and the object it touched. Before-and-after images tell the recovery engine what to undo or replay. The system that writes the log also defines what every field means, which is why recovery works at all: the evidence arrives pre-interpreted. Its meaning is fixed before anything goes wrong.

Melissa Pan spent roughly three years maintaining C++ code in those Db2 recovery components at IBM. Then she moved to multi-agent systems, where the evidence works differently.

The MAST paper (NeurIPS 2025, Pan as co-author) set out to classify why systems built from several cooperating language models fail. Six experts read over 150 execution traces, the recorded sequence of everything the agents said and did, averaging more than 15,000 lines each. The categories had to be built out of the traces rather than brought to them, so the team worked inductively, coding examples, comparing them, revising the scheme, and coding again. A database engine hands you a schema for its own logs. Nothing here does.

The paper is specific about why this is hard. One surface symptom, missing information, could mean an agent withheld something it had, or ignored what another agent told it, or lost the detail through context management. In one trace, a phone agent never mentioned an API's username-format requirement to its supervisor, and the supervisor never asked. The result was a string of failed logins. The failed logins alone cannot separate those two breakdowns. Someone has to compare what one agent knew, what it passed along, and what the receiving agent did with the silence. The trace records behavior; a human supplies the frame that makes the behavior legible as a particular kind of failure. A July 2026 preprint from the same group automates part of the taxonomy-building for new systems. It still validates against independent annotator agreement, which moves the human from labeling to gatekeeping without changing what counts as evidence.

Pan's ICML 2026 paper studied how agents are actually deployed in production, and the method shifted again: 20 semi-structured interviews and 306 survey responses. Production agent systems are typically proprietary, and public documentation doesn't carry the implementation detail the questions required. The team could not read traces or logs. They could ask practitioners what they had seen and check the summaries against one another. What those accounts surfaced was reasoning that no trace contains. Sixty-eight percent of reported production agents run ten steps or fewer before a human is brought in, and the rationale for that number sits in teams' judgments about how much risk they can absorb and how much review capacity they have. The agent records the ten steps. It does not record why there are ten.

Pan has not publicly described any of this as a deliberate arc; the reading is mine. But the sequence is there in the work. Database recovery logs carry their own interpretation. Multi-agent traces need a vocabulary that someone builds before the traces can be classified at all. Production deployments generate evidence that is organizationally enclosed, reachable only through the accounts of people who work inside.

Each move changes what evidence is, who produces it, and who is allowed to see it. Better logging is an answer to the first problem, and it is a good answer. It does very little for the second, where the missing thing is a scheme of interpretation, and nothing at all for the third, where the missing thing is standing.