Practitioner's Corner

Practitioner's Corner

Melissa Pan and the Logs the Field Can't See

Pan began as an engineer recovering databases after they failed. Now she studies why AI agents fail, but the companies running them keep their records private, so the field learns from whatever is public. Pan once learned that way too: as a grad student, she has written, she "had the view that most agents were coding agents," the kind most visible on public leaderboards, until production data corrected her. If the public record misled her about what most agents do, what might it miss about why they fail?

Melissa Pan and the Logs the Field Can't See
Pan began as an engineer recovering databases after they failed. Now she studies why AI agents fail, but the companies running them keep their records private, so the field learns from whatever is public. Pan once learned that way too: as a grad student, she has written, she "had the view that most agents were coding agents," the kind most visible on public leaderboards, until production data corrected her. If the public record misled her about what most agents do, what might it miss about why they fail?
The Check-In Statement an Agent Can Carry but Not Make

United has moved to block AI agents from airline check-in, citing a box on the form that only a human can tick. Check-in has rested on the passenger's own word before. Until 2002, staff asked every passenger, "Has your bag been out of your control since you packed it?" Only the traveler could honestly answer. Once an agent is filling in the form for you, which answers can it give on your behalf, and which still have to come from you?
The Check-In Statement an Agent Can Carry but Not Make
United has moved to block AI agents from airline check-in, citing a box on the form that only a human can tick. Check-in has rested on the passenger's own word before. Until 2002, staff asked every passenger, "Has your bag been out of your control since you packed it?" Only the traveler could honestly answer. Once an agent is filling in the form for you, which answers can it give on your behalf, and which still have to come from you?


Walking the Board at 3 A.M., an Interview With the Night Auditor of a Browser Fleet
CONTINUE READINGIncident Detail

Railway's write-up of its October security incident has one line I can't stop chewing on. During cleanup, the team rotated the credentials its services use to write to storage. That was the correct move. Then database backups quietly stopped landing in storage. They didn't resume until the affected services were redeployed.
Plenty of platforms hand a program its secrets once, at startup, usually as environment variables. The program copies the value into memory and never looks again. Change the stored secret and every process that's already running keeps presenting the old one, which storage now rejects.
So a rotation is two jobs. You change the secret, and then everything that uses it has to restart and pick up the new value. The first job gives you a confirmation on screen. The second gives you silence, and so does a broken backup job. The database keeps serving traffic the whole time, looking perfectly fine, and nobody learns otherwise until they need a restore.
Undoing Things Outside Software






