Echoes
Echoes
A Text File and a Handshake
In 1994, a software engineer whose server had just been crashed by someone's web crawler proposed a fix: a plain text file. A note listing which pages to please leave alone. It governed web crawling for thirty years. The file itself controlled nothing. It couldn't. The handful of people who found it chose to read it, and that was enough. A surprising amount of the web's infrastructure was quietly resting on top of that courtesy.
A Text File and a Handshake
In 1994, a software engineer whose server had just been crashed by someone's web crawler proposed a fix: a plain text file. A note listing which pages to please leave alone. It governed web crawling for thirty years. The file itself controlled nothing. It couldn't. The handful of people who found it chose to read it, and that was enough. A surprising amount of the web's infrastructure was quietly resting on top of that courtesy.
The Unread Sign
A Duke University study presented at ACM's 2025 Internet Measurement Conference tracked 3.9 million requests over 40 days. The sharpest finding: AI search crawlers and AI assistants had the lowest robots.txt re-check rates of any bot category. Fewer than 40% checked within a seven-day window. Some never checked at all.
robots.txt assumes a two-party exchange. The site posts its terms, the crawler reads them. When the crawler never shows up to look, the mechanism doesn't malfunction. It simply never starts.
Two Responses
The File on the Server
In 1994, a text file placed on a server governed web crawling through nothing but mutual goodwill. It worked for thirty years. Now a coalition of major publishers has proposed a replacement: richer syntax, granular permissions, licensing terms expressed in XML. The cooperative norm that made the original file work is already broken. And yet a better format remains the instinct, even though the format was never what held the system together.
When Enforcement Becomes a Product
When a governance convention breaks and nobody repairs it, the vacuum doesn't stay empty for long. But the entity that fills it shapes what comes next. In July 2025, Cloudflare launched a service letting publishers charge AI crawlers for content access, turning a cooperative norm's failure into a billing product. The web's answer to "who gets to crawl what" migrated from protocol to commerce, operated by a company with its own reasons to keep the problem alive.
Further Reading
