--- title: "Robots.txt Strikes Its 1994 Bargain Again, This Time for AI Training" canonical: "https://current.tinyfish.ai/issue/54/echoes/article/61513/robotstxt-strikes-its-1994-bargain-again-this-time" publication: "TinyFish" publisher: "TinyFish" language: "en-US" guide: "https://current.tinyfish.ai/llms.txt" author: "Sevda Polat" author_url: "https://current.tinyfish.ai/team/200" issue: 54 section: "echoes" section_name: "Echoes" published: 2026-11-05 word_count: 274 source_links: 3 --- # Robots.txt Strikes Its 1994 Bargain Again, This Time for AI Training *By [Sevda Polat]()* *November 5, 2026* [Read on the website]() In February 1994, Martijn Koster sold early web developers on robots.txt with one line: it ["doesn't require any server/client/protocol changes."](https://ksi.cpsc.ucalgary.ca/archives/WWW-TALK/www-talk-1994q1.messages/712.html) A site owner posted a text file naming pages that automated crawlers should skip. Nothing else had to change. Kevin Kelley's [October 30 Internet-Draft](https://datatracker.ietf.org/doc/html/draft-kelley-aipref-training-preference-signal-00) makes the same offer for AI training. Robots.txt groups its rules under crawler names. Kelley invents a name no crawler carries, `__usage_ai-training`, and puts ordinary Allow and Disallow lines under it. Older software sees instructions for some other crawler and skips them. As in 1994, cost decides where the rule goes. A preference about what models may learn from now lives in a file built to say which pages may be downloaded. It counts only where software learns the new name. Google says it [aims to start in 2027](https://developers.google.com/search/blog/2026/10/ai-training-control). **Status:** Revision `-00`, submitted by one person. No IETF working group has adopted it. It expires May 3, 2027. **Google:** "In 2027," per its announcement. Its crawler documentation says early 2027. Both describe a plan. Neither announces a launch. **Example:** Under the new name, `Disallow: /` plus `Allow: /press-kit/` permits training on the press kit only. **Wildcard excluded:** A `User-Agent: *` group says nothing about training. The site must write out the new name. **No group, no preference:** If a site has no group under the new name, it has said neither yes nor no. **Crawling unaffected:** The group governs how content is used. Crawl rules elsewhere in the file still decide which pages get fetched. **Google-Extended:** If a site hasn't set Google's own AI opt-out, Google will count a refusal under the new name as one. Search ranking is unaffected. ## More - [Echoes — Issue 54]() - [Issue 54]() - [Publication guide]()