--- title: "Echoes" canonical: "https://current.tinyfish.ai/issue/20/echoes" publication: "TinyFish" publisher: "TinyFish" language: "en-US" guide: "https://current.tinyfish.ai/llms.txt" permalink: "https://current.tinyfish.ai/issue/20/echoes" issue: 20 section: "echoes" description: "Past infrastructure decisions echoing in today's production realities" authors: ["Rina Takahashi","Nora Kaplan"] published: 2026-03-16 updated: 2026-03-12 articles: 10 --- # TinyFish > CURRENT is a journal about AI agents in production, for the engineering and product leaders who have to make them work. Drawing on experience building enterprise web agents, it explains why the demo succeeded and the deployment didn't, and what that means for choosing vendors, briefing the board, and avoiding mistakes others have already made. [Read on the website]() ## Issue 20 — Echoes Past infrastructure decisions echoing in today's production realities ### The Full Arc - [A Text File and a Handshake]() — By [Rina Takahashi]() · Published 2026-03-11 · Format: The Full Arc In 1994, a software engineer whose server had just been crashed by someone's web crawler proposed a fix: a plain text file. A note listing which pages to please leave alone. It governed web crawling for thirty years. The file itself controlled nothing. It couldn't. The handful of people who found it chose to read it, and that was enough. A surprising amount of the web's infrastructure was quietly resting on top of that courtesy. ### The Unread Sign - [Most AI Crawlers Never Even Check robots.txt]() — Published 2026-03-11 A Duke University study presented at ACM's 2025 Internet Measurement Conference tracked 3.9 million requests over 40 days. The sharpest finding: AI search crawlers and AI assistants had the lowest robots.txt re-check rates of any bot category. Fewer than 40% checked within a seven-day window. Some never checked at all. ### The File's Author - [An Imagined Conversation with the Man Who Asked the Web to Behave]() — Published 2026-03-11 In February 1994, a Dutch software engineer named Martijn Koster posted a message to the www-talk mailing list proposing "a standard that will allow WWW server maintainers to indicate if they want robots to access their server, and if so which parts." The standard was a plain text file. No authentication, no enforcement mechanism, no semantic layer. It didn't require changes to servers, clients, or protocols. It just asked nicely. ### Two Responses - [The File on the Server]() — By [Nora Kaplan]() · Published 2026-03-11 · Format: Two Responses In 1994, a text file placed on a server governed web crawling through nothing but mutual goodwill. It worked for thirty years. Now a coalition of major publishers has proposed a replacement: richer syntax, granular permissions, licensing terms expressed in XML. The cooperative norm that made the original file work is already broken. And yet a better format remains the instinct, even though the format was never what held the system together. - [When Enforcement Becomes a Product]() — By [Nora Kaplan]() · Published 2026-03-11 · Format: Two Responses When a governance convention breaks and nobody repairs it, the vacuum doesn't stay empty for long. But the entity that fills it shapes what comes next. In July 2025, Cloudflare launched a service letting publishers charge AI crawlers for content access, turning a cooperative norm's failure into a billing product. The web's answer to "who gets to crawl what" migrated from protocol to commerce, operated by a company with its own reasons to keep the problem alive. ### Further Reading - **Robots.txt in 2025** — [Source]() — Published 2026-03-11 Traces the full arc from Martijn Koster's 1994 server crash to Perplexity's stealth crawlers in a single, clean read. If you want the origin story before pulling on the thread, start here. - **Does Robots.txt Matter Anymore?** — [Source]() — Published 2026-03-11 Duke University researchers found that many AI bots never even check the file. This piece follows that finding to its uncomfortable logical end: a cooperative norm only works if someone bothers to look at the sign. - **New AI Web Standards and Scraping Trends in 2026: Rethinking Robots.txt** — [Source]() — Published 2026-03-11 Surveys every proposed successor format, from ai.txt to llms.txt, then documents how few platforms have actually adopted any of them. Useful for seeing how the instinct to fix the format keeps recurring, even when the format was never the load-bearing part. - **How AI Is Changing Web Scraping in 2026** — [Source]() — Published 2026-03-11 A practitioner-level look at why scraping infrastructure evolved so much faster than any governance mechanism could keep up with. The maintenance economics alone explain a lot about where the incentives actually point. - **State of Web Scraping 2026 Report** — [Source]() — Published 2026-03-11 Introduces the term "Advanced Persistent Scraping" to describe always-on, distributed, autonomous collection operating at infrastructure scale. Worth reading for the framing alone: it names what robots.txt is now up against, and the gap between a voluntary text file and an adversarial system becomes visceral. ## Other sections in Issue 20 - [Daily Brief]() - [Market Pulse]() - [Foundations]() - [Practitioner's Corner]() - [Vision]() ## More - [Publication guide]() - [Issue archive]()