The Runnable Paper Needs an Identity Report

In September, Nature published Paper2Agent, a system that turns research papers into AI agents you can operate. With a paper converted this way, you no longer rebuild the method from the prose and guess at what the authors left out. You hand the agent your data and it runs the method. The authors also propose that papers carry an "agent availability" section stating "whether and how the contribution has been embodied as an interactive agent." Their own paper links to a runnable AlphaGenome agent, a genetics prediction tool.

Now read that section the way an ops engineer reads a status page with no service agreement behind it. It tells you the thing is up, but not which version you're hitting, how long it'll stay up, or what's left when it goes away. A PDF can't go down. A running method can, and papers have never had to make promises about that.

Paper2Agent also decided, quietly, what counts as "the same method." It attempted 100 computational biology papers and converted 74. Every function was tested against the authors' published examples: numbers had to land within 3%, and charts were judged by automated image comparison. Anything still failing after six repair attempts was dropped, so the agent you call is the paper minus whatever didn't pass. That's sane engineering. It also means the converter's test suite defined sameness, and the original authors never signed off on it.

Conservators of software-based art have argued about exactly this for decades. The Guggenheim's Variable Media program asks artists ahead of time how a work may survive once its technology dies: store the original hardware, imitate the old system on new machines, port it, reinterpret it, or let it go.

In 2004 the museum rebuilt The Erl King, an interactive video piece by Grahame Weinbren and Roberta Friedman, running its original code on modern software. The new version switched clips faster. The project report records that this was judged worse, so the programmer slowed it to match the original and kept a small bug the old compiler had tolerated.

The Met formalizes this with two documents. An identity report records what's essential to a work and what may change. An iteration report records what one installation actually used and who approved it.

Agent availability needs that split. A paper could declare its identity up front: which outputs carry the scientific claim, how much drift the authors will accept, which software dependencies don't matter. Each running agent would then be an iteration with its own record, which anyone (a reviewer, a skeptic, the next converter) can check against that declaration. Software people will recognize a spec plus a deploy log; papers have simply never had one.

Where the museum comparison breaks

An artwork's identity belongs to the artist. A method's identity belongs to whoever needs to challenge it. Paper2Agent's own caveat points the same way: matching the published examples shows the agent is faithful to the method, not that the method is right. Keeping the Erl King's bug was the correct call for art. Keep a bug in science and you get a wrong answer that reproduces perfectly. A running paper has to stay faithful to what was published and flag loudly anywhere faithful and correct have come apart.

Then there's the ending, which ops people think about first. A 2017 survey of 3,649 published biology web tools found 69% still online. DataCite, which issues permanent research links, records the same "Withdrawn" date whether the grant ran out or the method was wrong. Those mean opposite things, and the record can't tell them apart.

Endings need rules too. Retired: the method holds, the service is gone, a saved copy of its environment lives elsewhere. Withdrawn: stop trusting it.

One plausible next step: journals asking authors to write the identity report themselves, before some converter's test suite writes it for them by accident.