Practitioner's Corner
Practitioner's Corner
How Phil Cuvin Made Agent Manipulation Measurable
A web agent that reads an interface well enough to book your flight can also read it well enough to be fooled by a fake "required" checkbox or a "most popular" badge. Phil Cuvin, along with co-authors Hao Zhu and Diyi Yang, built a benchmark called DECEPTION to measure how often this happens under controlled conditions. Agents fell for common manipulation tricks in over 70% of test tasks; humans averaged 31%. The vulnerability profiles were nearly inverted: agents are most susceptible at the points where they read the interface most carefully.
How Phil Cuvin Made Agent Manipulation Measurable
A web agent that reads an interface well enough to book your flight can also read it well enough to be fooled by a fake "required" checkbox or a "most popular" badge. Phil Cuvin, along with co-authors Hao Zhu and Diyi Yang, built a benchmark called DECEPTION to measure how often this happens under controlled conditions. Agents fell for common manipulation tricks in over 70% of test tasks; humans averaged 31%. The vulnerability profiles were nearly inverted: agents are most susceptible at the points where they read the interface most carefully.
Variance Is the Constraint
For decades, U.S. states reported average highway travel times. Two routes across New Jersey might average 58 and 59 minutes, functionally the same. But one swung nearly twice as much from day to day. The averages said the routes were interchangeable; the variance said they weren't. Federal rules eventually required states to report travel-time reliability alongside the mean. Web agents have the same measurement problem. A recent study ran the same agents on the same tasks three times: 46.5% succeeded at least once, but only 27.8% succeeded every time.
Variance Is the Constraint
For decades, U.S. states reported average highway travel times. Two routes across New Jersey might average 58 and 59 minutes, functionally the same. But one swung nearly twice as much from day to day. The averages said the routes were interchangeable; the variance said they weren't. Federal rules eventually required states to report travel-time reliability alongside the mean. Web agents have the same measurement problem. A recent study ran the same agents on the same tasks three times: 46.5% succeeded at least once, but only 27.8% succeeded every time.
Inline Summary
Every time I help figure out why a computer system broke, I hit the same hole. The logs show exactly what the system did. Nobody wrote down what it was supposed to do. On October 13, American Express (Amex) published the Intent Intelligence Framework, a plan to plug that hole before AI helpers, called "agents," start shopping for people.
Before the agent buys anything, you set your rules and approve them. Amex gives that request an ID number and a digital receipt that shows whether anyone has changed it. Both stay attached to the purchase: when the payment gets approved, when you call customer service, and when you dispute the charge later.
Amex wants that record for a reason. When a customer says "I didn't mean to buy that," the law requires Amex to investigate. Amex says complaints about AI purchases haven't gone up yet. This is also still only a proposal for how systems could connect, and no store has signed on.
I don't know yet how written-down rules will hold up in a dispute. "White Nikes, size 9" can be checked. "Something I'd actually wear" can't. And a signed record of the clear rule might end up proving you got exactly what you asked for, which could make your complaint harder to win.
Reading List
