--- title: "Practitioner's Corner" canonical: "https://current.tinyfish.ai/issue/8/practitioners-corner" publication: "TinyFish" publisher: "TinyFish" language: "en-US" guide: "https://current.tinyfish.ai/llms.txt" permalink: "https://current.tinyfish.ai/issue/8/practitioners-corner" issue: 8 section: "practitioners-corner" description: "Lessons from the field—what we see building at scale" authors: ["Rina Takahashi"] published: 2025-12-22 updated: 2025-12-18 articles: 15 --- # TinyFish > CURRENT is a journal about AI agents in production, for the engineering and product leaders who have to make them work. Drawing on experience building enterprise web agents, it explains why the demo succeeded and the deployment didn't, and what that means for choosing vendors, briefing the board, and avoiding mistakes others have already made. [Read on the website]() ## Issue 8 — Practitioner's Corner Lessons from the field—what we see building at scale ### Dispatches from Production Scale - [The Six-Hour Price Check]() — By [Rina Takahashi]() · Published 2025-12-17 · Format: Field Dispatch An analyst opens their laptop at 9 AM. The task: check competitor pricing across five major players in three product categories. Document changes. Flag anything significant. Seems straightforward. By 3 PM, they're still at it. This is "just check the website" in practice. Six hours navigating to pricing pages, documenting numbers in spreadsheets, cross-referencing yesterday's data. Repeat tomorrow. And the day after. ### The Builders - [The Oncologist Who Builds AI for Two-Minute Cancer Decisions]() — By [Rina Takahashi]() · Published 2025-12-18 · Format: Builder Profiles Two minutes. That's how long most UK tumor boards have to discuss each patient's cancer treatment plan. In that window, clinicians synthesize radiology reports, pathology results, biomarker tests, and treatment guidelines to make decisions that directly affect survival. When information gaps appear—missing test results, incomplete staging—the case gets postponed. This happens in 7% of cases. ### The Number That Matters - [Data Teams Spend 40% of Time Chasing Freshness]() — Published 2025-12-17 Data professionals spend 40% of their time on quality tasks, with freshness violations causing nearly a third of all data downtime. Engineers end up running constant fire drills instead of building new pipelines. ### Field Notes from the Ecosystem December delivered the kind of operational lessons that only surface when systems run at scale. Compromised services hijacked entire fleets. Configuration changes cascaded into global outages. The gap between compliant bots and evasive scrapers that outnumber them eight to one became impossible to ignore. We also saw what production optimization actually looks like in numbers. Latency dropped fivefold. Uptime percentages slipped just enough to matter. These observations come from infrastructure meeting real workload, from authentication protocols getting deployed, from the distance between pilot programs and production systems becoming measurable in downtime minutes. - **Compromised Next.js Services Hijack Railway Fleet** — [Source]() — Published 2025-12-17 Tag: Security Incident Vulnerable Next.js services on Railway ran malicious binaries December 16, causing CPU spikes across Europe West. Engineers misdiagnosed it as performance degradation for hours before identifying the attack vector at 08:30 UTC. - **AWS WAF Adds Cryptographic Bot Verification** — [Source]() — Published 2025-12-17 Tag: Bot Authentication AWS WAF deployed Web Bot Authentication on November 21, using HTTP Message Signatures (RFC 9421) for cryptographic verification. The protocol issues four verification labels, letting organizations whitelist verified bots while blocking impersonators. - **Capital One Cuts Agent Latency Fivefold** — [Source]() — Published 2025-12-17 Tag: Agent Performance Capital One reduced Chat Concierge latency by 5x since launch through proprietary multi-agentic workflow optimization. Building their own stack instead of using third-party platforms enabled direct latency tuning at the infrastructure level. - **API Uptime Drops 60% Year-Over-Year** — [Source]() — Published 2025-12-17 Tag: Infrastructure Reliability Average API uptime fell from 99.66% in Q1 2024 to 99.46% in Q1 2025. That's 60% more downtime. Multi-carrier benchmarks show European shippers hit hardest during peak season with inconsistent rate limiting headers. - **Evasive Scrapers Outnumber Compliant Bots 8:1** — [Source]() — Published 2025-12-17 Tag: Bot Detection Akamai data shows web scraping volume detected by advanced methods was eight times that of compliant AI bots for a subset of publishers. The real threat remains unverified scrapers, not compliant agents from Google, OpenAI, Meta. - **Malformed Config Cascades Through Cloudflare Network** — [Source]() — Published 2025-12-17 Tag: Configuration Risk Cloudflare's November 18 outage started with a flawed ClickHouse database permission change that generated malformed Bot Management configuration files. The oversized files, bloated with duplicated data, overwhelmed size limits and cascaded globally. ### Practitioner Resources - **Cloudflare's Log Loss: When Subsystems Stop Protecting Themselves** — [Source]() — Published 2025-12-17 Postmortem showing how a single overloaded component cascaded into 55% data loss. The subsystem isolation assumptions that broke. - **GitHub Engineering: Deployment Practices at Repository Scale** — [Source]() — Published 2025-12-17 How teams actually ship when you're running thousands of deployments. Specific practices for monorepo operations and release coordination. - **Engineering Blogs Worth Following: Curated Technical Resources** — [Source]() — Published 2025-12-17 Aggregated list of company engineering blogs. Useful for finding teams publishing real operational experience, though quality varies significantly. - **Slack's Grid Rearchitecture: Moving Beyond Workspace Boundaries** — [Source]() — Published 2025-12-17 Complete system redesign when initial assumptions break at scale. Shows what it costs to fix foundational architecture decisions years later. - **Fiverr's Four-Phase Auth Migration: Trading Elegance for Latency** — [Source]() — Published 2025-12-17 Why they chose gradual migration over clean-slate microservices. Real numbers on latency costs and the operational risk of big-bang rewrites. - **Engineering Blog Directory: Finding Teams Publishing Real Experience** — [Source]() — Published 2025-12-17 Comprehensive list of company engineering publications. Start here to identify teams consistently sharing production insights and architectural decisions. ## Other sections in Issue 8 - [Daily Brief]() - [Market Pulse]() - [Foundations]() - [Echoes]() - [Vision]() ## More - [Publication guide]() - [Issue archive]()