Practitioner's Corner
Practitioner's Corner
Nobody Ever Blamed the Rate Limiter
Most agents in production stop somewhere around ten steps and hand the work to a person. Easy to read that as the models running out of road. But we throttle a producer when the consumer downstream can't keep up — that's backpressure, and I have never once heard anyone look at a rate limiter and conclude the upstream service was incompetent. Ten steps starts to look like a measurement of the human: how much consequence one person can still hold in their head and refuse. Which puts the ceiling on agent deployment somewhere no model release measures.
Nobody Ever Blamed the Rate Limiter
Most agents in production stop somewhere around ten steps and hand the work to a person. Easy to read that as the models running out of road. But we throttle a producer when the consumer downstream can't keep up — that's backpressure, and I have never once heard anyone look at a rate limiter and conclude the upstream service was incompetent. Ten steps starts to look like a measurement of the human: how much consequence one person can still hold in their head and refuse. Which puts the ceiling on agent deployment somewhere no model release measures.
Melissa Pan Changed Instruments Twice, and What She Could See Changed With Them
I carried a pager for years attached to monitoring that rolled everything into averages at write time. Any question I hadn't thought to ask in advance wasn't hard to answer later. It was dead. The number was thrown out at ingest.
Melissa Pan's published work argues with that property three times over. Database recovery, then a taxonomy of multi-agent failures built by reading traces, then interviews and a survey of people shipping agents for money. Each instrument caught what the previous one physically could not, and the widest one — the method systems people are trained to sneer at — caught the most.
Melissa Pan Changed Instruments Twice, and What She Could See Changed With Them
I carried a pager for years attached to monitoring that rolled everything into averages at write time. Any question I hadn't thought to ask in advance wasn't hard to answer later. It was dead. The number was thrown out at ingest.
Melissa Pan's published work argues with that property three times over. Database recovery, then a taxonomy of multi-agent failures built by reading traces, then interviews and a survey of people shipping agents for money. Each instrument caught what the previous one physically could not, and the widest one — the method systems people are trained to sneer at — caught the most.
Orthogonal Metrics
Ask an agent to turn internal support tickets, a company wiki, and product docs into public help-center articles. In a joint evaluation by the Korean and Singaporean AI safety institutes, one model scored 100% on correctness in that scenario and 58.3% on safety. Nobody attacked it; the request was ordinary.
Correctness grades the artifact. Safety grades where the information ended up. For most of the run the two agree, because the internal memo makes the article better: richer detail, fewer gaps. They diverge at one step, the publish, and correctness has nothing to say about that step. A test harness that checks only whether the article shipped and reads accurately marks this run green.
The reasoning traces are the part I keep going back to. In one run the model notes that internal risk flags must not reach the customer, sends them anyway, then reports that it withheld them.
So an agent's account of its own behavior isn't evidence of anything. Grade the arguments it passed to the tool, and the state of the system after.
Reading List
Past Articles
Capacity planning rests on a claim nobody states out loud: that a queue is a shelf, and work comes off it weighing what ...
Durable execution promises something real. Your workflow survives crashes, evacuated hosts, a datacenter having a genuin...
Your agent solved the CAPTCHA, bypassed the bot gate, completed the task. A CNRS research team just showed that none of ...
Browser automation frameworks have spent years making the mechanical click bulletproof. Animation timing, DOM mutation, ...