--- title: "Practitioner's Corner" canonical: "https://current.tinyfish.ai/issue/48/practitioners-corner" publication: "TinyFish" publisher: "TinyFish" language: "en-US" guide: "https://current.tinyfish.ai/llms.txt" permalink: "https://current.tinyfish.ai/issue/48/practitioners-corner" issue: 48 section: "practitioners-corner" description: "Lessons from the field—what we see building at scale" authors: ["Sable Whitford","Sevda Polat"] published: 2026-09-28 updated: 2026-09-25 articles: 14 --- # TinyFish > CURRENT is a journal about AI agents in production, for the engineering and product leaders who have to make them work. Drawing on experience building enterprise web agents, it explains why the demo succeeded and the deployment didn't, and what that means for choosing vendors, briefing the board, and avoiding mistakes others have already made. [Read on the website]() ## Issue 48 — Practitioner's Corner Lessons from the field—what we see building at scale ### Builder Profile - [Melissa Pan's Missing Citation]() — By [Sable Whitford]() · Published 2026-09-24 · Format: Builder Profile Melissa Pan co-led a NeurIPS 2025 paper analyzing how multi-agent systems fail under benchmark conditions, then co-led an ICML 2026 paper surveying practitioners about proprietary agent deployments in production. Six authors appear on both. The second paper does not cite the first. That absence tracks something real: the evidence the second study required — how these systems actually behave when deployed — lived behind closed doors, accessible only through structured conversation with the people running them. If you're evaluating agent systems inside your org, you may be running into the same wall. ### Lead Feature - [You Can't Stage Someone Else's Website]() — By [Sable Whitford]() · Published 2026-09-24 · Format: Lead Feature A housing association in Wales built an automation for food-bank referrals. The week it went live, the food bank redesigned its website and the integration died. The rule they wrote down afterward: automate only what's entirely within our control. That reads as timidity until you trace what sits underneath it. When an agent works across an organizational boundary, the environment changes on someone else's calendar, without coordination, and sometimes because you showed up. No amount of rehearsal closes that gap, because the stage isn't yours. ### Practitioner Interview - [A QA Engineer on Why the Green Bar Lies Now]() — Published 2026-09-24 *Reva Chandra is a Staff QA Engineer at a travel booking platform that, eighteen months ago, began deploying customer-facing web agents to handle itinerary changes, cancellations, and rebooking. She was asked to build the test suite and define acceptance criteria. She has been rethinking her career choices ever since.* ### Mechanism Sidebar - [The Gate That Worked Perfectly and Made Everything Worse]() — By [Sable Whitford]() · Published 2026-09-24 A [September 22 preprint](https://arxiv.org/abs/2609.25686) tested a deterministic enforcement gate — a component sitting between an AI agent and its execution environment, refusing any action that violates compiled task state. On an airline benchmark, it did exactly what you'd want: caught ineligible cancellations, blocked policy violations, raised the agent's pass rate from 0.39 to 0.54. "Does this action violate policy?" was answerable from database state alone, so the gate's judgment was reliably correct. ### Reading List - **An Illusion of Progress? Assessing the Current State of Web Agents** — [Source]() — Published 2026-09-24 Moving from cached benchmarks to live websites drops agent scores from 90% to 61%. - **WAREX: What Happens When Web Agents Hit Real Infrastructure Failures** — [Source]() — Published 2026-09-24 Network errors and adversarial popups injected into benchmarks cut agent success rates by up to 95%. - **WebForge and the Benchmark Trilemma: Realism, Reproducibility, or Scale — Pick Two** — [Source]() — Published 2026-09-24 A formal argument for why web-agent benchmarks cannot be realistic, reproducible, and scalable at once. - **ST-WebAgentBench: Measuring What Benchmarks Miss About Policy Compliance** — [Source]() — Published 2026-09-24 Agents that finish their assigned tasks still violate organizational policies 38% of the time. - **Evaluation and Benchmarking of LLM Agents: A Survey from SAP Labs** — [Source]() — Published 2026-09-24 KDD 2025 survey maps enterprise-specific evaluation gaps: access control, compliance, and long-horizon drift. - **Emergence WebVoyager: Why Simulated Websites Fail as Agent Testbeds** — [Source]() — Published 2026-09-24 Shadow DOMs, iframes, infinite scroll, and modal popups make live sites a different problem than cached ones. ### Past Articles - [Phil Cuvin — Turning Anecdotes into Instruments]() — By [Sable Whitford]() · Published 2026-09-17 · Format: Builder Profile Across the Qwen 2.5 VL family, dark-pattern susceptibility climbed from 43.8% at 3 billion parameters to 73.7% at 72 billion. The more capable models were the easier ones to manipulate. That number exists because Phil Cuvin and his collaborators built a benchmark with two independent axes — task completion and susceptibility — where everyone before had tracked one. It's the through-line of his research: find the problem practitioners describe in shorthand and never instrument, then build the counter. - [Who Receives the Report]() — By [Sevda Polat]() · Published 2026-09-17 · Format: Reporting Infrastructure Aviation's confidential safety reporting system has handled more than 2.3 million reports without once breaking a reporter's confidentiality. Its most consequential design choice, made in 1976, was that NASA receives the reports rather than the FAA. OpenAI's new misalignment-reporting framework makes several choices unusual for a technology company, but its reporting chain runs through the organization whose systems are being examined. What AI safety decides now about who receives reports will shape what the field is able to learn about its own failures for a long time. - [What Aman Gupta Built Before the Model]() — By [Sable Whitford]() · Published 2026-09-10 · Format: Builder Profile Aman Gupta's team at Nubank put five agents in front of more than 100 million customers in fourteen months. The KDD 2026 paper describing the architecture is precise about what got built first: idempotent APIs, audit trails, tool sequences composed deterministically in code — infrastructure whose job is to absorb the model's mistakes rather than pass them along. It's quieter about action-error rates, retry rates, and duplicate-effect counts, none of which appear anywhere in it. The build order is the argument, and the absent numbers are part of it. - [Nothing Else Happened]() — By [Sable Whitford]() · Published 2026-09-10 · Format: Lead Analysis A database can tell you nothing else changed during a transaction, because the database defines every operation that could possibly happen. An agent working on the open web can't make that claim. It can log the flight it booked. It can't establish that it didn't also accept an arbitration clause, opt you into data sharing, or leave a warranty in the cart. Nobody enumerated the possible interactions, so their absence can't be confirmed after the fact. That has consequences for how these systems get built. ## Other sections in Issue 48 - [Daily Brief]() - [Market Pulse]() - [Foundations]() - [Echoes]() - [Vision]() ## More - [Publication guide]() - [Issue archive]()