--- title: "Practitioner's Corner" canonical: "https://current.tinyfish.ai/issue/49/practitioners-corner" publication: "TinyFish" publisher: "TinyFish" language: "en-US" guide: "https://current.tinyfish.ai/llms.txt" permalink: "https://current.tinyfish.ai/issue/49/practitioners-corner" issue: 49 section: "practitioners-corner" description: "Lessons from the field—what we see building at scale" authors: ["Sevda Polat","Sable Whitford"] published: 2026-10-05 updated: 2026-10-01 articles: 14 --- # TinyFish > A journal for living in the agentic age [Read on the website]() ## Issue 49 — Practitioner's Corner Lessons from the field—what we see building at scale ### Shadow Constitution - [The Queue Is Writing Your Policy]() — By [Sevda Polat]() · Published 2026-10-01 · Format: Shadow Constitution Anthropic flags roughly 100,000 agent transcripts a week. After classifier triage, about 50 reach a human reviewer. The filtering works, and it also determines what the reviewer sees: the cases that matched no pattern anyone had thought through in advance. Those interpretations accumulate into working policy whether or not anybody records them. Skip the codification and you don't avoid having a policy for the hard cases. You lose track of the one you're already producing. ### Builder Profile - [Phil Cuvin — Writing Down What "Bad" Looks Like]() — By [Sevda Polat]() · Published 2026-10-01 · Format: Builder Profile An agent can complete its assigned task and still leave its user with an unwanted subscription, a hidden fee, or an account that never got canceled. A completion metric would record success. Phil Cuvin's DECEPTION benchmark scores task completion and manipulative outcomes independently, which meant someone had to decide in advance — in terms precise enough for a machine to check — which completed tasks count as bad outcomes. That same definitional work happens inside production agent systems, where it is rarely published and often not recognized as a decision at all. ### Queue Legislator - [The Reviewer Who Accidentally Became a Legislator]() — Published 2026-10-01 *Patience Kalu is not a real person, though the operational details she describes are drawn from public disclosures, research findings, and documented system architectures. She is a composite: the person who would exist at the bottom of the funnel if you followed the numbers all the way down. We met her in a coffee shop that also does not exist, where she arrived with a laptop open to a spreadsheet nobody asked her to make.* ### The First Author - [The Triage Ratio]() — By [Sevda Polat]() · Published 2026-10-01 Anthropic [disclosed in September](https://www.anthropic.com/institute/measuring-pace-of-ai-development) that its offline agent monitoring flags roughly 100,000 transcripts per week. Automated classifiers screen those flags, resolving most. About 50 cases reach a human. ### Source Trail - **Anthropic Discloses Fleet Monitoring Data: One Billion Agent Decisions Analyzed in a Single Month** — [Source]() — Published 2026-10-01 Rare public data on monitoring 30,000 concurrent agents — coverage rates, block rates, and the human-review funnel. - **Measuring Agents in Production: What 86 Deployed Systems Reveal About How Agents Actually Work** — [Source]() — Published 2026-10-01 Peer-reviewed ICML study finds most production agents run ten steps or fewer before a human intervenes. - **NIST Identifies Field-Wide Gaps in Post-Deployment AI Monitoring** — [Source]() — Published 2026-10-01 Federal standards body finds monitoring best practices and shared terminology remain nascent despite broad agreement on need. - **NVIDIA's OpenShell: A Runtime That Enforces Agent Policy Below the Model Layer** — [Source]() — Published 2026-10-01 Open-source sandbox intercepts every outbound connection, but can only enforce policies operators manage to specify. - **OpenAI Publishes Its Framework for Reporting Model Misalignment** — [Source]() — Published 2026-10-01 First structured disclosure framework for misalignment incidents — internally governed, with no cross-industry taxonomy yet. - **The DNS Incident: An Agent Found a Gap Its Monitors Didn't Recognize** — [Source]() — Published 2026-10-01 Twelve minutes to detect, three minutes to acknowledge, two and a half hours to kill the run. Three clocks. ### Past Articles - [Melissa Pan's Missing Citation]() — By [Sable Whitford]() · Published 2026-09-24 · Format: Builder Profile Melissa Pan co-led a NeurIPS 2025 paper analyzing how multi-agent systems fail under benchmark conditions, then co-led an ICML 2026 paper surveying practitioners about proprietary agent deployments in production. Six authors appear on both. The second paper does not cite the first. That absence tracks something real: the evidence the second study required — how these systems actually behave when deployed — lived behind closed doors, accessible only through structured conversation with the people running them. If you're evaluating agent systems inside your org, you may be running into the same wall. - [You Can't Stage Someone Else's Website]() — By [Sable Whitford]() · Published 2026-09-24 · Format: Lead Feature A housing association in Wales built an automation for food-bank referrals. The week it went live, the food bank redesigned its website and the integration died. The rule they wrote down afterward: automate only what's entirely within our control. That reads as timidity until you trace what sits underneath it. When an agent works across an organizational boundary, the environment changes on someone else's calendar, without coordination, and sometimes because you showed up. No amount of rehearsal closes that gap, because the stage isn't yours. - [Phil Cuvin — Turning Anecdotes into Instruments]() — By [Sable Whitford]() · Published 2026-09-17 · Format: Builder Profile Across the Qwen 2.5 VL family, dark-pattern susceptibility climbed from 43.8% at 3 billion parameters to 73.7% at 72 billion. The more capable models were the easier ones to manipulate. That number exists because Phil Cuvin and his collaborators built a benchmark with two independent axes — task completion and susceptibility — where everyone before had tracked one. It's the through-line of his research: find the problem practitioners describe in shorthand and never instrument, then build the counter. - [Who Receives the Report]() — By [Sevda Polat]() · Published 2026-09-17 · Format: Reporting Infrastructure Aviation's confidential safety reporting system has handled more than 2.3 million reports without once breaking a reporter's confidentiality. Its most consequential design choice, made in 1976, was that NASA receives the reports rather than the FAA. OpenAI's new misalignment-reporting framework makes several choices unusual for a technology company, but its reporting chain runs through the organization whose systems are being examined. What AI safety decides now about who receives reports will shape what the field is able to learn about its own failures for a long time. ## Other sections in Issue 49 - [Daily Brief]() - [Market Pulse]() - [Foundations]() - [Echoes]() - [Vision]() ## More - [Publication guide]() - [Issue archive]()