Share

Purpose Over Detection

Verified, Then Refused

By Sevda Polat— August 5, 2026

Feature image for article: Verified, Then Refused

Site operators are writing robots.txt rules against crawlers that were switched off years ago, and against crawlers nobody has ever identified. You could read that as carelessness, in which case there is nothing to learn from it. Read it as improvisation and it tells you a great deal: people building a classification system that no one has issued them, out of the only material lying around, which is whatever name a client announces about itself.

What they are sorting for is purpose. Indexing, training, a live assistant checking a page on someone's behalf, a checkout. And that sets up a failure your retry logic has no way to see: your agent proves exactly what it is, and gets turned away for it.

Purpose Over Detection

Verified, Then Refused

By Sevda Polat— August 5, 2026

Site operators are writing robots.txt rules against crawlers that were switched off years ago, and against crawlers nobody has ever identified. You could read that as carelessness, in which case there is nothing to learn from it. Read it as improvisation and it tells you a great deal: people building a classification system that no one has issued them, out of the only material lying around, which is whatever name a client announces about itself.

What they are sorting for is purpose. Indexing, training, a live assistant checking a page on someone's behalf, a checkout. And that sets up a failure your retry logic has no way to see: your agent proves exactly what it is, and gets turned away for it.

Protocol Boundaries

Web Bot Auth Signs the Sender, Not the Errand

By Sable Whitford— August 5, 2026

Feature image for article: Web Bot Auth Signs the Sender, Not the Errand

For years my entire toolkit for identifying a crawler was a user-agent string, an IP range, and a reverse DNS record that any motivated adult could dress up to look wholesome. Web Bot Auth swaps that guess for a cryptographic check, and a narrower one than most coverage suggests. It tells you a key holder sent something to this host inside a time window. Who the agent is working for, and why, are ruled out of scope on purpose. Here's what actually gets signed, and what comes back to your desk.

Protocol Boundaries

Web Bot Auth Signs the Sender, Not the Errand

By Sable Whitford— August 5, 2026

For years my entire toolkit for identifying a crawler was a user-agent string, an IP range, and a reverse DNS record that any motivated adult could dress up to look wholesome. Web Bot Auth swaps that guess for a cryptographic check, and a narrower one than most coverage suggests. It tells you a key holder sent something to this host inside a time window. Who the agent is working for, and why, are ruled out of scope on purpose. Here's what actually gets signed, and what comes back to your desk.

Recognition Reading

Use Cases for Authentication of Web Bots

Nottingham's IETF draft lays out the live debates: centralization risk, operator-only identity, and what's still unresolved.

NIST NCCoE: Accelerating AI Agent Identity and Authorization

The first U.S. government concept paper proposing OAuth and SPIFFE-based identity frameworks for autonomous agents.

2026 Bad Bot Report: Bad Bots in the Agentic Age

How Cryptographically Signed AI Agents Change the Closing Web

Cloudflare AI Bot Traffic Controls: Search vs. Agent vs. Training

Bot Management: What Works in 2026?

Recognition Reading

Use Cases for Authentication of Web BotsNottingham's IETF draft lays out the live debates: centralization risk, operator-only identity, and what's still unresolved.

NIST NCCoE: Accelerating AI Agent Identity and AuthorizationThe first U.S. government concept paper proposing OAuth and SPIFFE-based identity frameworks for autonomous agents.

Quick links

2026 Bad Bot Report: Bad Bots in the Agentic Age

How Cryptographically Signed AI Agents Change the Closing Web

Cloudflare AI Bot Traffic Controls: Search vs. Agent vs. Training

Bot Management: What Works in 2026?

Recognition Stack

When an automated agent connects to a website, the site can try to answer a series of questions. Which network sent this request? Who claims to operate it? Can that claim be verified cryptographically? What is the agent trying to do?

Each question maps to a different identification layer: IP verification, user-agent strings, cryptographic signatures, traffic classification, bot directories, behavioral fingerprinting. Together they form a recognition stack. Each layer establishes something real, and each hits a hard boundary where its proof runs out. Knowing where those boundaries fall shapes what policies a site can actually enforce.

Recognition Stack

When an automated agent connects to a website, the site can try to answer a series of questions. Which network sent this request? Who claims to operate it? Can that claim be verified cryptographically? What is the agent trying to do?

Each question maps to a different identification layer: IP verification, user-agent strings, cryptographic signatures, traffic classification, bot directories, behavioral fingerprinting. Together they form a recognition stack. Each layer establishes something real, and each hits a hard boundary where its proof runs out. Knowing where those boundaries fall shapes what policies a site can actually enforce.

Network Origin

Reverse DNS Confirms Network Ownership and Nothing Else

Forward-confirmed reverse DNS verifies that a connecting IP belongs to a stated organization's network. Google publishes its crawler IP ranges as JSON files for direct checking. This confirms which network sent a request, but not which user triggered it, or why.

Self Declaration

User-Agent Strings Claim an Identity Without Any Proof

Any HTTP client can set a user-agent header claiming to be Googlebot or Chrome. OpenAI Operator runs inside real Chromium and sends a genuine browser string. Modern spoofing extends to navigator objects and client hints. User-agent works as a first-pass filter, never a final answer.

Cryptographic Identity

Web Bot Auth Signs Requests With Verifiable Keys

The IETF WebBotAuth protocol uses HTTP Message Signatures (RFC 9421) to let bot operators cryptographically sign requests with Ed25519 keys. Sites verify signatures against published public keys. This establishes operator identity per-request, replacing spoofable strings with provable claims about who sent the traffic.

Operator Limits

Signatures Prove the Operator, Not the User's Intent

The IETF charter explicitly excludes end-user authentication and intent vocabulary from Web Bot Auth's scope. A valid OpenAI signature proves OpenAI's infrastructure sent the request. It says nothing about which user initiated it, what task they authorized, or whether the agent stayed within scope.

Purpose Categories

Traffic Classification Sorts Search, Agent, and Training Traffic

Edge networks now classify bot traffic by purpose: Search (indexing), Agent (real-time browsing for a user), and Training (model fine-tuning). A site can permit an operator for search while blocking it for training. Classification draws on self-declaration, directory entries, and behavioral signals.

Verified Directories

Bot Directories Verify Operators Based on Observed Behavior

Verified bot directories list known operators with behavior-based classifications, validated through IP ranges, reverse DNS, or Web Bot Auth. Directory listing lets sites apply consistent rules without per-request verification. But a listing reflects behavior at enrollment time — categories shift if observed patterns later diverge.

Behavioral Signals

Behavioral Fingerprinting Detects Automation Below the HTTP Layer

Recent research shows behavioral signals like typing latency, scroll timing, and mouse movement continuity can identify AI browsing agents with near-perfect accuracy in controlled settings. These signals work even when agents send genuine browser user-agents and route through residential proxies. Evasion-optimized agents may close that gap.

Persistent Gap

The Combined Stack Still Cannot Identify End Users

Together, these layers tell a site which network sent a request, which operator signed it, what purpose category applies, and whether behavior looks automated. They cannot tell a site which end user initiated the action, what task was authorized, or whether the agent honored its mandate.

Purpose Policy Map

Websites used to make one decision about automated traffic: let it through or block it. That's splintering. Cloudflare's July 2026 retirement of its single "block AI bots" toggle, replaced by three independent category controls, reflects a broader move toward purpose-based governance, where rules depend on why a bot visits.

Five traffic categories are taking shape, each carrying different default policies, identity requirements, and enforcement realities. The category labels look tidy. The boundaries between them are considerably less stable, and the mechanisms for verifying what a bot actually intends remain early and unevenly adopted.

Purpose Policy Map

Websites used to make one decision about automated traffic: let it through or block it. That's splintering. Cloudflare's July 2026 retirement of its single "block AI bots" toggle, replaced by three independent category controls, reflects a broader move toward purpose-based governance, where rules depend on why a bot visits.

Five traffic categories are taking shape, each carrying different default policies, identity requirements, and enforcement realities. The category labels look tidy. The boundaries between them are considerably less stable, and the mechanisms for verifying what a bot actually intends remain early and unevenly adopted.

Search Crawling

Search Crawlers Remain Welcome With Tangled Strings

Search crawling stays allowed by default, even under Cloudflare's September 2026 policy changes. But crawlers like Googlebot bundle search indexing and AI training into the same request. Blocking training means blocking the whole crawler unless operators use granular opt-out tokens like Google-Extended rather than category-level controls.

Agent Action

Real-Time Agents Get Blocked on Ad Pages

Agents acting on a user's behalf in real time are blocked by default on ad-supported pages, on the logic that ads are designed for human attention. Enforcement gets messy: Google's user-triggered fetchers ignore robots.txt entirely, so sites need WAF rules or IP blocking to actually stop them.

Data Training

Training Crawlers Hit Mass Opt-Outs and Paywalls

Over 2.5 million websites now block AI training crawlers. Cloudflare's September 2026 defaults extend this by blocking training on ad-supported pages automatically. A third option is emerging alongside allow and block: HTTP 402 pay-per-crawl, currently in private beta, lets sites price access to their content.

Transaction Agents

Transaction Bots Tracked but Missing Managed Controls

Cloudflare's BotBase classifies transaction bots as a separate category from general agents, but there's no managed preset for them yet. Authorization depends on payment network rules (Visa added agentic-payment requirements in April 2026) and existing anti-fraud systems, not web infrastructure categories.