A Publisher's Web Ops Engineer on Sorting Agent Traffic From Everything Else

On July 1, 2026, Cloudflare replaced its binary "Block AI Bots" toggle with three independently manageable traffic categories — Search, Agent, and Training — available to all customers.¹ The move formalized what publisher-side operators had been doing informally for months: treating agent traffic as something fundamentally different from crawlers that index and crawlers that train. For the roughly 23% of websites sitting behind Cloudflare's infrastructure,² this was less a revelation than a vocabulary for decisions they were already making.

We wanted to understand how those decisions actually get made. Not from the platform side, but from behind the dashboard. What follows is a conversation with Dot Parsons, a senior web operations engineer at a mid-large media publisher, who has managed bot traffic policy for over fifteen years. Dot is not a real person. She is a composite character grounded in publicly observable operational realities, research findings, and the specific constraints that publisher-side operators navigate daily. Her opinions are her own, which is to say, they are ours, which is to say, they belong to no one in particular. But the problems she describes are very real.

You've been managing bot policy since before AI agents were a category. What changed?

Dot: The volume. I know that sounds mundane, but it's the honest answer. AI "user action" crawling, what Cloudflare now labels the Agent category, increased over fifteen times on their network in 2025 alone.³ That's not background noise anymore.

When I started doing this work, bot policy was basically three rules: let Googlebot in, rate-limit everything suspicious, go home. Now I'm staring at traffic that doesn't fit either bucket, and there's a lot of it.

What makes agent traffic different from a search crawler or a training scraper?

Dot: A search crawler builds a database of your site so it can answer questions later. A training scraper takes your content to fine-tune a model. An agent is supposedly acting right now, in real time, on behalf of a specific person trying to get something done.¹ That's Cloudflare's framing, and it's useful because each one maps to a different economic relationship with my site.

Search sends traffic back. That was the deal for twenty years, and everyone understood it. Training takes content and gives nothing back. Agents occupy this weird middle ground. An agent booking a flight through our travel section could drive a transaction. An agent summarizing our reporting so the user never visits? That's extraction wearing a helpful hat.

So how do you decide what to let through?

Dot: Right now it's a layered mess, honestly.

First layer: does it identify itself? Cloudflare's BotBase gives me classification and crawl-to-referral ratios, meaning how often a company crawls my content versus how many visitors it actually refers back.⁴ That's the basic economic signal. If you're crawling ten thousand pages a day and referring back twelve visitors, we should talk.

Second layer: is it cryptographically verified? Cloudflare's signed agent infrastructure, Web Bot Auth, lets a bot sign each request with a key I can verify against a published record.⁵ That's real progress. Verified AI agents reportedly cover about 84% of identified AI browser traffic as of June 2026.⁶

But "identified" is doing a lot of work in that sentence.

Third layer: behavioral signals. And this is where it gets interesting.

The fingerprinting research.

Dot: Yeah. At least three published studies from this spring show that agents are distinguishable from humans, and from each other, through behavioral patterns alone. Typing cadence. Scroll timing. The absence of continuous mouse movement between clicks. One study achieved perfect classification scores by combining browser and behavioral features.⁷ Another showed you can identify the underlying language model from interaction traces alone, with up to 96% accuracy.⁸

But here's the finding that actually rewired how I think about all of this:

That's counterintuitive.

Dot: Only if you've never watched traffic logs for a living. A normal browser session has a fingerprint. A bot pretending to be a normal browser session has a different fingerprint, and the pretending itself is a signal. Think of it as the uncanny valley, but for HTTP requests. The effort to look natural produces its own kind of unnatural.

So now I have leverage I didn't have two years ago. I can say to an agent operator: look, I can probably detect your traffic whether you identify yourself or not. So why not just tell me who you are, and we can have a negotiated relationship instead of an adversarial one?

What does robots.txt give you in this situation?

Dot: Almost nothing useful. RFC 9309 was designed for a world where crawlers self-declare and voluntarily comply.¹⁰ It literally says the rules are "not a form of access authorization."¹⁰ And now you've got operators like OpenAI taking the position that user-initiated agent fetches aren't subject to robots.txt at all¹¹, which, fine, I understand the argument. But it means my robots.txt is a suggestion that some of the biggest players have explicitly said doesn't apply to their most interesting product.

A 2025 study found scrapers selectively respect robots.txt based on site characteristics and their own operational goals.¹² So it's not even a reliable suggestion.

What about RSL, Really Simple Licensing?

Dot: RSL is the right direction. Machine-readable licensing terms, pay-per-crawl, pay-per-inference. That's the vocabulary I actually need.¹³ Robots.txt can say "don't come here." RSL can say "you can come here, and here's what it costs." Completely different conversation.

But enforcement remains the same open question. RSL requires AI companies to voluntarily respect it,¹⁴ just like robots.txt. We keep building better vocabulary for a negotiation where one side can still walk away from the table whenever it likes.

Starting September 15, Cloudflare's new defaults block Training and Agent bots on pages that display ads.¹⁵ What does that mean for you?

Dot: It means Cloudflare said the quiet part out loud: an ad is a signal that a publisher meant for a human to land on that page.¹⁵ If an agent extracts the content and the user never sees the ad, the publisher's business model breaks. Nobody needed a PhD to figure that out. But nobody with Cloudflare's market position had codified it as a default before.

For me operationally, it's a starting point. I'll keep Search open, block Training on ad-supported pages, and make case-by-case decisions on Agent traffic based on verification status and referral ratios. The default saves me from having to justify the obvious. The exceptions are where the actual work lives.

If you could make one request of every agent operator, what would it be?

Dot: Separate your crawlers by purpose. Cloudflare's already asking for this¹⁶. If you do search and training and agent work, run three separate crawlers so I can make three separate decisions. When a mixed-purpose crawler shows up, I have to treat it by its worst-case use, and that usually means blocking everything.

And sign your requests. I don't care about your user-agent string. Those are trivially spoofed.⁵ Give me a cryptographic signature I can verify. That's the difference between a conversation and a guessing game.

You don't sound adversarial about this.

Dot: I'm not. Some of this traffic is genuinely useful. An agent that helps a user complete a transaction on my site? Great. An agent that surfaces my content in a way that drives subscriptions? Wonderful. I would love to let those through with a smile and a fast lane.

I just need to be able to tell that agent apart from the one strip-mining my archive to train a model. Right now, the agents that declare themselves clearly get through. The ones that try to look human get treated like threats.

Footnotes

1. Cloudflare, "Your site, your rules: new AI traffic options for all customers," July 1, 2026. https://blog.cloudflare.com/content-independence-day-ai-options/ ↩ ↩²

2. Cloudflare reports sitting in front of roughly 22.7% of websites as of May 2026. ↩

3. Cloudflare, "Radar 2025 Year in Review," January 2026. https://blog.cloudflare.com/radar-2025-year-in-review/ ↩

4. Cloudflare Bots Changelog. https://developers.cloudflare.com/changelog/product/bots/ ↩

5. Cloudflare, "Forget IPs: using cryptography to verify bot and agent traffic," March 13, 2026. https://blog.cloudflare.com/web-bot-auth/ ↩ ↩²

6. Pravinkumar.co, "Cloudflare's June 2026 Bot Management Update," June 4, 2026. https://www.pravinkumar.co/blog/cloudflare-june-2026-bot-management-webflow-protection-2026 ↩

7. Wang, Shafiq, Vekaria (UC Davis), "FP-Agent: Fingerprinting AI Browsing Agents," arXiv:2605.01247, May 2026. https://arxiv.org/abs/2605.01247 ↩

8. Lugoloobi et al., "Known By Their Actions: Fingerprinting LLM Browser Agents via UI Traces," arXiv:2605.14786, May 2026. https://arxiv.org/html/2605.14786v1 ↩

9. Fayolle et al., "On the Internet, Nobody Knows You're an LLM Bot," arXiv:2606.30119, June 2026. https://arxiv.org/abs/2606.30119 ↩

10. RFC 9309, "Robots Exclusion Protocol," IETF, September 2022. https://www.rfc-editor.org/rfc/rfc9309 ↩ ↩²

11. As of 2026, OpenAI removed language stating that its ChatGPT-User crawler would comply with robots.txt, taking the position that user-initiated actions through AI agents are not subject to robots.txt restrictions. ↩

12. Kim et al., "Scrapers selectively respect robots.txt directives," arXiv:2505.21733, 2025. https://arxiv.org/pdf/2505.21733 ↩

13. RSL Standard, "Really Simple Licensing." https://rslstandard.org/ ↩

14. WebProNews, "Really Simple Licensing 1.0 Empowers Publishers on AI Content Use," December 2025. https://www.webpronews.com/really-simple-licensing-1-0-empowers-publishers-on-ai-content-use/ ↩

15. Cloudflare Developers Changelog, "New options to manage AI traffic," July 1, 2026. https://developers.cloudflare.com/changelog/post/2026-07-01-ai-traffic-options/ ↩ ↩²

16. Cloudflare, "Your site, your rules," July 1, 2026. https://blog.cloudflare.com/content-independence-day-ai-options/ ↩