--- title: "Practitioner's Corner" canonical: "https://current.tinyfish.ai/issue/10/practitioners-corner" publication: "TinyFish" publisher: "TinyFish" language: "en-US" guide: "https://current.tinyfish.ai/llms.txt" permalink: "https://current.tinyfish.ai/issue/10/practitioners-corner" issue: 10 section: "practitioners-corner" description: "Lessons from the field—what we see building at scale" authors: ["Rina Takahashi","Nora Kaplan"] published: 2026-01-05 updated: 2025-12-31 articles: 15 --- # TinyFish > CURRENT is a journal about AI agents in production, for the engineering and product leaders who have to make them work. Drawing on experience building enterprise web agents, it explains why the demo succeeded and the deployment didn't, and what that means for choosing vendors, briefing the board, and avoiding mistakes others have already made. [Read on the website]() ## Issue 10 — Practitioner's Corner Lessons from the field—what we see building at scale ### Dispatches from Production Scale - [How Three Retries Become Sixty-Four Database Attempts]() — By [Rina Takahashi]() · Published 2025-12-31 · Format: Field Dispatch Error rates double every thirty seconds—4%, 8%, 16%, 32%—while every recovery system reports normal behavior. Frontend shows three retry attempts per failure. API gateway executes its own retry logic. Backend service follows documented procedures. Yet one failed authentication becomes sixty-four database attempts before any layer notices. When we're running thousands of concurrent sessions, this pattern appears constantly: systems designed to provide resilience somehow amplifying failures exponentially instead. ### The Builders - [Kate Blair and the Quest to Build HTTP for Agent Coordination]() — By [Nora Kaplan]() · Published 2025-12-31 · Format: Builder Profiles When Tyson Foods and Gordon Food Service deployed AI agents to manage supply chain operations, they discovered something almost quaint: the agents couldn't talk to each other. Custom point-to-point integration was the only option. Exactly the kind of brittle solution that breaks when you're coordinating across dozens of suppliers. ### The Number That Matters - [97% of Traffic Disappeared After Bot Detection]() — Published 2025-12-31 OffenderWatch processed 80 million API requests daily. After implementing rate limiting and bot protection, that number dropped to 2.5 million. A 97% reduction. ### Field Notes from the Ecosystem December closed with production systems behaving as they do. Automation meant to protect became the failure mode: Cloudflare's security fix took down 28% of its traffic. Threat actors moved from disclosure to exploitation in hours. Supply chain compromises hit browser extensions, API keys, the seams between systems. The teams building agent systems learned what operators already know: scale reveals what small tests hide. Multi-agent architectures that worked with ten agents fell apart at fifty. Context engineering turned out to be the entire job, not a feature. The gap between demo and production stayed wide and expensive. - **Cloudflare WAF Buffer Triggers 25-Minute Outage** — [Source]() — Published 2025-12-31 Tag: Infrastructure Outages Cloudflare increased WAF buffer size from 128KB to 1MB on December 5 while patching CVE-2025-55182. The change failed, impacting 28% of HTTP traffic for 25 minutes. Second major outage in under a month. Company response: "Code Orange: Fail Small." - **React2Shell Exploited Hours After Disclosure** — [Source]() — Published 2025-12-31 Tag: Threat Response China state-nexus groups exploited CVE-2025-55182 within hours of December 3 disclosure. One threat cluster spent nearly an hour troubleshooting exploitation attempts. AWS Sonaris processes 200 billion events per minute detecting real-time exploitation patterns. - **Trust Wallet Extension Loses $7M via Leaked Key** — [Source]() — Published 2025-12-31 Tag: Supply Chain Attacker used leaked Chrome Web Store API key to publish malicious Trust Wallet extension v2.68 on December 24. One million users affected, $7 million in losses. Attacker registered exfiltration domain December 8, began requests December 21. - **Google ADK Treats Context as Compiled View** — [Source]() — Published 2025-12-31 Tag: Context Engineering Google Agent Development Kit documentation shows model cost and time-to-first-token grow quickly with context size. Framework treats context as compiled view over stateful system, distinguishing durable state (Sessions) from per-call views (working context). - **Microsoft's 50-Agent System Shows Bimodal Results** — [Source]() — Published 2025-12-31 Tag: Agent Architecture Azure SRE Agent team's multi-agent system with 50+ sub-agents produced bimodal results: when handoffs worked, everything worked. When they failed, agents got lost. Poorly tuned sub-agents pollute orchestrator context with confused outputs, causing infinite loops. - **Docker Hardened Images Cut Vulnerabilities 95%** — [Source]() — Published 2025-12-31 Tag: Container Security Docker announced December 17 that Hardened Images reduce vulnerabilities up to 95% using distroless runtime. Docker Hub handles 20 billion pulls monthly. Extending hardening to MCP server images as supply chain attacks projected to cost $60 billion in 2025. ### Practitioner Resources - **OpenAI's Telemetry Cascade: When Observability Kills Availability** — [Source]() — Published 2025-12-31 Four-hour outage from metrics deployment. Watch how telemetry volume can overwhelm the systems it's meant to monitor. - **GenAI Governance: IBM's $4.45M Lesson in Codified Controls** — [Source]() — Published 2025-12-31 Vector poisoning losses documented. AWS architecture showing why governance needs enforcement in code, not policy documents. - **Data Engineering 2024: PayPal's Spark 3 Migration Math** — [Source]() — Published 2025-12-31 70% cost reduction from version upgrade and GPU optimization. Includes Medium's pipeline work and Expedia's SLO platform. - **Data Center Buildout: 80 GW by 2030** — [Source]() — Published 2025-12-31 McKinsey's infrastructure projections with cooling trade-offs. Know the power constraints before you plan for scale. - **Uber's Distributed Systems: P99 Latency Over Averages** — [Source]() — Published 2025-12-31 Production monitoring practices from actual scale. Alert management and on-call sustainability that survives real operations. - **Engineering Blogs Worth Following: LinkedIn's 4 Trillion Events** — [Source]() — Published 2025-12-31 Curated list of company engineering blogs. LinkedIn's Kafka and Pinot work shows what billion-user infrastructure actually requires. ## Other sections in Issue 10 - [Daily Brief]() - [Market Pulse]() - [Foundations]() - [Echoes]() - [Vision]() ## More - [Publication guide]() - [Issue archive]()