In February 1994, Martijn Koster posted a proposal to an email list. Web crawlers were hammering servers. They fired off requests as fast as they could, re-fetched the same files over and over, and wandered into CGI scripts that actually did things when you called them. Koster wanted a way for the people running those servers to ask the bots to knock it off. What he described had two fields: which bot, and which paths to skip. By June there was a rough consensus, and the spec said right up front that it was "neither official nor enforced." Bot operators honored it because they chose to.
A few years later I was a teenager with root on a university's machines, and by then robots.txt was furniture. Nobody thought about it much. We've written before about why it worked: the parties had aligned interests, they could see each other, and the file gave them a cheap way to coordinate. That was good enough for almost thirty years.
In 2022, twenty-eight years after Koster's post, the IETF standardized robots.txt as RFC 9309. Read this sentence from it slowly:
"These rules are not a form of access authorization."
So a robots.txt file is a request. It doesn't lock anything, it doesn't bind anyone, it doesn't license anything. The RFC spells that out because by 2022 people were already treating the file as if it did, and you don't put a sentence like that into a standard unless you're tired of watching it get misread. Whether or not the authors meant it this way, that line is a diagnosis. It marks the edge of what the convention can actually hold, at the exact spot where people had started piling other things on top.
Then AI training showed up and changed what people were asking for. By late 2023, news publishers were adding new bot names to their robots.txt files to block AI crawlers. But a lot of those entries didn't really mean "don't visit." They meant "don't train a model on what you find here." That's a restriction on downstream purpose, and robots.txt has no words for it. The file gets checked when the bot shows up. The use publishers care about happens later, maybe months later, maybe at some company that bought the training data secondhand.
A 2024 IAB workshop, whose report came out this year as RFC 9969, names this gap directly: the distance between "crawl time" and "inference time." I have spent a lot of years watching systems get asked to enforce properties they can't even observe, and this is that problem again. Something that runs at fetch time can't govern what happens at training time, for the same reason a door lock can't enforce a lease. They sit at different layers, run on different timescales, and involve different parties.
The report finds more structural problems. The file covers the whole site, so individual authors on a shared platform can't set their own terms. The site operator controls it, not the person who wrote the content. And the rules don't travel with the content: copy an article somewhere else and the robots.txt stays behind on the original server. One workshop participant put it bluntly: "robots.txt is broken and we broke it." The report itself was more measured. Robots.txt "cannot be the only solution," it said, but nothing else is anywhere near as widely recognized.
A separate IETF Internet-Draft from August 2026 (still a work in progress, not an approved standard) takes a run at listing what's actually needed when agents interact with the web: identity, authentication, authorization, delegation, payment, audit, revocation. It says plainly that no single layer should handle all of these. Each one is an institutional function. It needs parties who can identify themselves, make commitments, and get held accountable when those commitments break. A two-field text file was never going to do that work, and honestly it's kind of amazing how much it carried for as long as it did. Where it's buckling tells you what was never built: purpose, authorship, persistence, accountability. Right now the cracks in the file are the clearest map we have of where those pieces need to go.
- Crawl time versus inference time: The IAB workshop report records a fundamental mismatch — a preference visible when content is collected may concern uses that happen months later, possibly at a third party that acquired the data secondhand — and proposes starting with location-based and embedded attachment mechanisms alongside a common vocabulary.
- Agent architecture requirements draft: An individual IETF Internet-Draft separates discovery, identity, authorization, delegation, payment, and audit into distinct architectural layers, explicitly warning that no single mechanism should be expected to handle all of them.
- NIST on agent identity: A public consultation on software and agentic AI identity found broad support for distinguishing human users from authorized agents and from bots merely claiming authorization, with respondents favoring persistent trust anchors paired with short-lived, narrowly scoped credentials.
- Purpose tokens in robots.txt: Some AI operators now use robots.txt entries not as crawler identifiers but as signals governing whether content gathered by existing crawlers may later be used for training — a shift documented in current operator pages that moves the file from "who may visit" toward "what may be done with what's found."