--- title: "Practitioner's Corner" canonical: "https://current.tinyfish.ai/issue/5/practitioners-corner" publication: "TinyFish" publisher: "TinyFish" language: "en-US" guide: "https://current.tinyfish.ai/llms.txt" permalink: "https://current.tinyfish.ai/issue/5/practitioners-corner" issue: 5 section: "practitioners-corner" description: "Lessons from the field—what we see building at scale" authors: ["Rina Takahashi"] published: 2025-12-01 updated: 2025-11-27 articles: 11 --- # TinyFish > CURRENT is a journal about AI agents in production, for the engineering and product leaders who have to make them work. Drawing on experience building enterprise web agents, it explains why the demo succeeded and the deployment didn't, and what that means for choosing vendors, briefing the board, and avoiding mistakes others have already made. [Read on the website]() ## Issue 5 — Practitioner's Corner Lessons from the field—what we see building at scale ### Dispatches from Production Scale - [The Hidden Economics of Retry Logic]() — By [Rina Takahashi]() · Published 2025-11-27 · Format: Operations Field Notes One authentication check fails. The system retries. Within seconds, that single failure becomes fifteen authentication attempts, five rate limit violations, and a blocked IP address. The logic seems sound: if at first you don't succeed, try again. But in web automation, retry logic doesn't just repeat operations—it multiplies them across layers, amplifies costs, and can transform a recoverable failure into a multi-hour outage. ### Mapping the Invisible Web - [When Twenty Services Pretend to Be One Website]() — By [Rina Takahashi]() · Published 2025-11-27 · Format: Field Dispatch A page loads. Product listings appear, prices populate, checkout button ready. The user sees one coherent website. Operationally, dozens of independent services just assembled themselves—each from different infrastructure, each on its own timeline, each capable of failing while the page still renders. Users never notice this coordination problem. The page looks functional. But is the payment processor actually ready? Has fraud detection finished? Are required scripts loaded? At scale, these questions become operational reality. ### Theory Meets Production Reality - [Why Perfect Bot Detection Is Operationally Impossible]() — By [Rina Takahashi]() · Published 2025-11-27 · Format: Parallax View Block a legitimate customer and watch them abandon their cart. Let a scraper through and it extracts competitive intelligence. Websites must achieve precision that's operationally impossible: filtering half of all internet traffic without touching revenue. The bot security market hit $668 million in 2024. Building web agent infrastructure means encountering these detection systems thousands of times daily. We see what defenders actually pay for precision they can't fully achieve. - [Why Reliable Automation Requires More Infrastructure Than Detection]() — By [Rina Takahashi]() · Published 2025-11-27 · Format: Parallax View We run millions of requests daily through enterprise web agent infrastructure, maintaining 98%+ success rates while detection systems evolve. The operational complexity concentrates entirely on the automation side. Defense should be harder than offense, right? But persistence at scale requires more infrastructure than precision. Getting through detection is just the beginning. Maintaining reliability across those millions of requests, adapting to whatever defenders deploy—that's where the real operational weight lives. ### The Number That Matters - [4GB of RAM After 2,500 Pages]() — Published 2025-11-27 A Selenium-based scraper hits 4GB of RAM consumption after roughly 2,500 page accesses. Not 25,000. Not 250,000. Twenty-five hundred. ### Field Notes from the Ecosystem November brought failures and adaptations in roughly equal measure. APIs without rate limits. Bot detection that detects nothing. Enterprises averaging ten observability tools while complaining about complexity. Then the adaptations: Kubernetes learning to schedule GPUs efficiently. Organizations cutting observability costs 75% through smarter sampling. The usual pattern of systems breaking, then getting patched, then breaking differently. One observation stands out. Your infrastructure provider can see exactly what you're building. API patterns, token usage, query types. The hyperscalers have comprehensive competitive intelligence about the application layer. Infrastructure position provides visibility that most customers haven't considered. - **Enterprise Size Buys Nothing** — [Source]() — Published 2025-11-27 Tag: Bot Detection DataDome tested 17,000 domains. Result: 61% of large enterprises (10,000+ employees) let every test bot through. Only 2.16% caught all bots. AI-driven bots now target value: 64% hit forms, 23% login pages, 5% checkout. Size correlates with nothing. - **Six Characters, Eight Million Records** — [Source]() — Published 2025-11-27 Tag: API Security Avelo Airlines' reservation API: query with 6-character codes, no last name verification. No WAF, no rate limiting, no CAPTCHA. Math: 2.18 billion combinations, 8 million tickets sold. Hit rate 1 in 270. Patched November 13, 2025. - **Ten Tools at Scale** — [Source]() — Published 2025-11-27 Tag: Observability Sprawl Grafana surveyed 1,255 organizations. Smallest companies (≤10 employees) average 4 observability technologies. Largest (5,000+ employees) average 10. Complexity and overhead top concern across all sizes. Alert fatigue outpaces other incident response obstacles 2:1. - **The Ten Percent Benchmark** — [Source]() — Published 2025-11-27 Tag: Observability Economics Organizations spend average 17% of infrastructure costs on observability. Median and mode both land at 10%. Range: 0% (OSS users ignoring overhead) to 50%. Smart sampling and tiered storage cuts costs 60-80% while preserving critical insights. - **Kubernetes Learns Accelerators** — [Source]() — Published 2025-11-27 Tag: GPU Orchestration GPU instances bill whether busy or idle. Kubernetes adapted: Kueue for workload management, Dynamic Resource Allocation for GPUs, topology-aware scheduling. Model sizes now exceed single GPU capacity. Work splits across multiple accelerators or doesn't run. - **Hyperscalers See What You Build** — [Source]() — Published 2025-11-27 Tag: Infrastructure Intelligence Princeton CITP analysis: hyperscalers observe customer activity through API patterns, token usage, query types. Microsoft sees what enterprises build with OpenAI. Google sees which Gemini applications gain traction. Infrastructure position provides comprehensive competitive intelligence about the application layer. ## Other sections in Issue 5 - [Daily Brief]() - [Market Pulse]() - [Foundations]() - [Echoes]() - [Vision]() ## More - [Publication guide]() - [Issue archive]()