--- title: "Echoes" canonical: "https://current.tinyfish.ai/issue/27/echoes" publication: "TinyFish" publisher: "TinyFish" language: "en-US" guide: "https://current.tinyfish.ai/llms.txt" permalink: "https://current.tinyfish.ai/issue/27/echoes" issue: 27 section: "echoes" description: "Past infrastructure decisions echoing in today's production realities" authors: ["Rina Takahashi"] published: 2026-05-04 updated: 2026-04-30 articles: 10 --- # TinyFish > CURRENT is a journal about AI agents in production, for the engineering and product leaders who have to make them work. Drawing on experience building enterprise web agents, it explains why the demo succeeded and the deployment didn't, and what that means for choosing vendors, briefing the board, and avoiding mistakes others have already made. [Read on the website]() ## Issue 27 — Echoes Past infrastructure decisions echoing in today's production realities ### The Sign in the Forest - [The Bargain That Held for Thirty Years]() — By [Rina Takahashi]() · Published 2026-04-29 · Format: The Sign in the Forest In 1994, a science fiction novelist's web crawler kept crashing someone's server. The fix was a plain text file asking automated visitors to stay off certain pages. That file, robots.txt, held for thirty years, because obeying it paid. ### Twenty Thousand to One - [20,583 Pages Per Referral: What the Crawl-to-Click Ratio Actually Looks Like Now]() — Published 2026-04-29 Googlebot crawls roughly five pages for every visitor it sends back. That ratio held for two decades. It was the deal. ### The Polite Request - [The Man Who Put a Sign on the Internet]() — Published 2026-04-29 *This conversation is imagined. Martijn Koster is very real — a Dutch software engineer who created robots.txt in 1994 and, by most accounts, the internet's first search engine. His actual words, where they exist in the public record, are cited. Everything else is constructed from what's known about his perspective, his temperament, and the dry pragmatism that runs through three decades of his public writing. He did not participate in this interview, has not endorsed it, and would probably find the whole exercise slightly odd.* ### Blind Spots and Inherited Frames - [The Invisible Agent]() — By [Rina Takahashi]() · Published 2026-04-29 · Format: Blind Spots and Inherited Frames Robots.txt governs web crawling through one mechanism: matching a text label that automated visitors voluntarily attach to themselves. For thirty years, crawlers wore that label because they needed to. The AI agents arriving now don't crawl. They open real browsers, render pages, click through interfaces. They carry the same user-agent string as the person who launched them. The protocol that governs automated access to the web literally cannot see what's visiting. - [The Successor Trap]() — By [Rina Takahashi]() · Published 2026-04-29 · Format: Blind Spots and Inherited Frames When robots.txt stopped working, replacements arrived fast. Llms.txt, ai.txt, Google's WebMCP — each citing robots.txt as ancestor, each proposing better signals and finer permissions. But the companies building browser-based agents benefit commercially from accessing content freely. The sites trying to restrict access are the ones whose content feeds those agents. That's a conflict of interest. And the new standards may have inherited exactly the framing that keeps it invisible. ### Threads Worth Following - **Cloudflare Radar Data Reveals the Real Economics of AI Crawling** — [Source]() — Published 2026-04-29 The numbers behind the broken bargain. - **Rethinking robots.txt: Why Successor Standards Haven't Arrived Yet** — [Source]() — Published 2026-04-29 A clear map of what's been proposed and what hasn't stuck. - **Internet Governance in 2026: When Authority Moves Upstream of Consensus** — [Source]() — Published 2026-04-29 The macro context for why voluntary norms are losing ground. - **Is Chrome the New Internet Explorer? The Chokepoint Mechanism That's Harder to Reverse** — [Source]() — Published 2026-04-29 Same structural concentration, opposite mechanism, harder to contest. - **Cloudflare Acknowledges the Symbiosis Is Over** — [Source]() — Published 2026-04-29 The infrastructure layer confirms what publishers already knew. ## Other sections in Issue 27 - [Daily Brief]() - [Market Pulse]() - [Foundations]() - [Practitioner's Corner]() - [Vision]() ## More - [Publication guide]() - [Issue archive]()