---
title: "Most AI Crawlers Never Even Check robots.txt"
canonical: "https://current.tinyfish.ai/issue/20/echoes/article/11185/most-ai-crawlers-never-even-check-robotstxt"
publication: "TinyFish"
publisher: "TinyFish"
language: "en-US"
guide: "https://current.tinyfish.ai/llms.txt"
issue: 20
section: "echoes"
section_name: "Echoes"
published: 2026-03-11
updated: 2026-03-12
word_count: 80
source_links: 0
---

# Most AI Crawlers Never Even Check robots.txt

*March 11, 2026*

[Read on the website](<https://current.tinyfish.ai/issue/20/echoes/article/11185/most-ai-crawlers-never-even-check-robotstxt>)

A Duke University study presented at ACM's 2025 Internet Measurement Conference tracked 3.9 million requests over 40 days. The sharpest finding: AI search crawlers and AI assistants had the lowest robots.txt re-check rates of any bot category. Fewer than 40% checked within a seven-day window. Some never checked at all.

robots.txt assumes a two-party exchange. The site posts its terms, the crawler reads them. When the crawler never shows up to look, the mechanism doesn't malfunction. It simply never starts.

## More

- [Echoes — Issue 20](<https://current.tinyfish.ai/issue/20/echoes/markdown>)
- [Issue 20](<https://current.tinyfish.ai/issue/20/markdown>)
- [Publication guide](<https://current.tinyfish.ai/llms.txt>)