Echoes
Echoes
The Bargain That Held for Thirty Years
The Bargain That Held for Thirty Years
Twenty Thousand to One
Blind Spots and Inherited Frames
The Invisible Agent
Robots.txt governs web crawling through one mechanism: matching a text label that automated visitors voluntarily attach to themselves. For thirty years, crawlers wore that label because they needed to. The AI agents arriving now don't crawl. They open real browsers, render pages, click through interfaces. They carry the same user-agent string as the person who launched them. The protocol that governs automated access to the web literally cannot see what's visiting.
The Successor Trap
When robots.txt stopped working, replacements arrived fast. Llms.txt, ai.txt, Google's WebMCP — each citing robots.txt as ancestor, each proposing better signals and finer permissions. But the companies building browser-based agents benefit commercially from accessing content freely. The sites trying to restrict access are the ones whose content feeds those agents. That's a conflict of interest. And the new standards may have inherited exactly the framing that keeps it invisible.
Threads Worth Following
