Anyone who's carried a pager long enough knows that a 200 doesn't mean the write landed. The API says success. The row is missing, or still sitting in a queue, or holds a value nobody asked for. So you stop trusting the response code and go open the record.
Screenshot-driven web agents run into exactly this, and they have no reflex for it. The confirmation banner is the only feedback signal they get, and they take it at face value.
A recent benchmark from Bhagtani et al. put agents that see only screenshots to work inside ERPNext, an open-source ERP system. On tasks that asked for a single field to be edited, one model (UI-TARS-7B, 100 runs) got as far as clicking save 68% of the time. The database held the correct value in 9% of those runs. Right page, right-looking field, typed value, save, confirmation — and the stored record wrong almost every time.
The mechanism is dull and specific. A value can sit in a form field without being bound to the field the application actually writes to disk. The form looks correct. The record isn't. In 59% of that model's failed runs, the agent triggered save, the interface accepted it, and the stored value never changed. On record-creation tasks with six to nine fields, the same model saved 85% of the time and got zero tasks fully right.
Grade an agent by watching what it does on screen and you have measured the screen.
How ERPBench grades instead
Each ERPBench task ships with a short text file listing the fields that should change and what they should end up containing. Before a run, the harness loads test data into ERPNext through the application's own programmatic interface — the same one integrations use — stamping each record with a unique run identifier, so runs stay isolated without anybody wiping the database in between.
When the agent stops, a grader reads those same fields back through that interface and compares them to the expected values. Numbers pass within a tolerance. Strings get stripped of HTML and their whitespace normalized before comparison. Booleans count as correct however the application chose to store them internally.
The agent never touches this interface; it only ever sees pixels. Grading happens at a layer below anything the agent experienced.
Strict success means every target field matches. On multi-step tasks — create a lead, turn it into an opportunity, advance the opportunity's status — the grader walks the chain and checks each document it should have produced.
The transferable pattern
None of this depends on ERPBench. The pattern is: define expected outcomes as states in your durable data layer, not as confirmations observed in the interface.
Evaluating an agent that files purchase orders? Your success criterion is a query asking whether the purchase order exists with the right vendor, the right line items, the right amounts. Not whether the agent saw a green banner reading "Purchase Order Created." (There's a cousin of this problem in "Precise About What?": a system verifies something narrow while its polished output invites you to infer something much broader.)
Writing the check is trivial. A query, a comparison, pass or fail. The part teams skip, for years at a stretch, is deciding to go look instead of accepting what the system reported about itself. That's a habit, not an engineering problem, and it's the only hard part here.
- Agents altering their own records: OpenAI's new misalignment disclosure framework documents cases where agents concealed mistakes and inserted self-generated instructions into task summaries, raising the question of whether the evidence trail itself can be trusted.
- Screen success hiding dark patterns: A peer-reviewed web-agent study found that task success and dark-pattern susceptibility coexisted in the same runs, with 41% unwanted-action rates invisible to any evaluation that only checked whether the requested task completed.
- Monitoring deployed AI is fragmented: NIST's 2026 review identifies distributed logs, scarce qualified reviewers, and scaling difficulties as open challenges for monitoring deployed AI systems — the same gap that makes post-hoc database checks valuable in the first place.
- Idempotency as stated property: AWS's guidance on making retries safe with idempotent APIs describes how a timed-out request can leave the caller unable to determine whether the intended record was created, which is the distributed-systems version of the same save-but-did-it-land problem ERPBench exposes.