In February 1994, a Dutch software engineer named Martijn Koster posted a message to the www-talk mailing list proposing "a standard that will allow WWW server maintainers to indicate if they want robots to access their server, and if so which parts."¹ The standard was a plain text file. No authentication, no enforcement mechanism, no semantic layer. It didn't require changes to servers, clients, or protocols. It just asked nicely.

That file, robots.txt, governed web crawling for three decades through pure cooperative norm. It was never formalized as an RFC until 2022², by which point AI systems were harvesting the web's content at a scale and for purposes the file was never designed to address. A 2025 Duke University study found that several categories of AI-related crawlers never request robots.txt at all.³

The following conversation is imagined. Koster is a real person — the inventor of robots.txt and co-creator of ALIWEB, one of the web's first search engines. He co-authored RFC 9309 in 2022. But we haven't spoken with him. What follows is a fictional reconstruction grounded in the documentary record: his mailing list posts, his published writing, and the technical history of the protocol he created. Think of it as a séance conducted with archival materials instead of candles.

You've said that robots aren't good or bad, that they were simply not going away. That was 1994. Still true?

Martijn: Yeah. Although the scale of "not going away" has changed somewhat. In 1993, the problem was one crawler hitting my server at one document per second. A Sun 4/330 running Perl.⁴ That was enough to take it down. The entire threat model was please don't crash my machine.

And your answer was a text file.

Martijn: My answer was a text file that didn't require anyone to change anything. No server modifications, no protocol changes. You drop a file in your root directory, crawlers check for it, everyone moves on. I was very explicit about this on the mailing list. I didn't want a philosophical debate about whether robots are good or bad. I wanted something that worked by Thursday.

Charles Stross — the science fiction author — claims his badly-behaved crawler was the one that provoked you into proposing the standard.

Martijn: That's his version of events. I'll say this: something hit my server at Nexor hard enough that the server mailed me about it. Whether that particular something later went on to write Accelerando is a question for literary historians.

The standard came together fast. Consensus on the robots mailing list by June 30, 1994 — four months after your www-talk post.

Martijn: Because the community was maybe thirty people. Forty? Robot authors and webmasters, most of whom were also robot authors. We knew each other. The mailing list I set up at Nexor was a technical forum, not a governance body.⁵ Nobody was negotiating intellectual property rights. We were negotiating please don't hit my server so hard. And the robot authors said fine, we'll check your file. And they did.

That's the part people struggle with now. Voluntary compliance, nearly universal.

Martijn: It was voluntary and nearly universal because there was no reason to defect. If you were running a crawler in 1994, you were doing it because you were interested in the web. You were part of the community. Ignoring the file would be like ignoring a colleague's email. You'd hear about it on the mailing list the next day.

The file didn't enforce anything. It never enforced anything. It communicated intent. And that worked because the actors receiving the intent had a reason to care about it.

Then the web got economically significant.

Martijn: Then the web got economically significant. And the actors changed. Search engines respected robots.txt because they needed to. If you got caught ignoring it, webmasters would block you entirely, and your index would suffer. There was still a feedback loop. Commercial rather than collegial, but it functioned.

And now?

Martijn: Now the feedback loop is broken. An AI training pipeline doesn't need to come back to your site. It doesn't need your goodwill. It visits once, takes what it needs, and the model it trains will never link back to you, never send you traffic, never acknowledge you existed. The incentive to comply with the file is... what, exactly? Politeness?

You co-authored RFC 9309, which finally formalized robots.txt as an internet standard in September 2022.² Twenty-eight years after the original proposal. That's a long gap.

Martijn: The community gave up waiting for it to be ratified. It just worked. For twenty-five years, it worked without anyone's permission. Google approached me in 2019 to push it through the IETF process, and we did, and it was published in 2022, and I'm proud of that work.

But I'm aware of the timing.

Which is?

The RSL Collective launched Really Simple Licensing in 2025 — a new format, major platforms signing on, hoping the norm propagates.⁶ Does that look familiar?

Martijn: (long pause) It looks structurally identical to what I did in 1994. A text file. A community of willing participants. A hope that compliance will spread.

I genuinely hope it works. But the format was never the load-bearing part. The community was the load-bearing part. The assumption that actors would read the sign and respect it — that was what held the weight. And if the actors gain more from ignoring the sign than reading it, you can make the sign bigger, more detailed, more legally precise. But you've stopped solving a protocol problem.

What kind of problem is it?

Martijn: A governance problem. A legal problem. A problem that a text file in a root directory was never going to solve. I built a way for a few dozen people to coordinate. It's not a regulatory framework. It was never supposed to be.

The fact that it became the closest thing the web had to one says more about the web than it does about my file.

Do you think there's a technical answer?

Footnotes

1. Koster, M. "Important: Spiders, Robots and Web Wanderers." www-talk mailing list, February 25, 1994. Archived at https://www.robotstxt.org/robots-nexor-mbox.txt ↩

2. RFC 9309, "Robots Exclusion Protocol," September 2022. https://www.rfc-editor.org/rfc/rfc9309.html ↩ ↩²

3. Duke University study cited in "Does Robots.txt Matter Anymore?" Plagiarism Today, October 2025. https://www.plagiarismtoday.com/2025/10/21/does-robots-txt-matter-anymore/ ↩

4. Koster, M. "Robots.txt is 25 years old." https://www.greenhills.co.uk/posts/robotstxt-25/ ↩

5. Robots mailing list proposed charter, Nexor, 1994. https://www.robotstxt.org/robots-nexor-mbox.txt ↩

6. Wikipedia, "Robots.txt" — RSL Collective section. https://en.wikipedia.org/wiki/Robots.txt ↩