Verified, Then Refused

An audit of 14,000 domains feeding the major AI training corpora found rules in robots.txt — the file that tells automated visitors what they may fetch — written against agents that were decommissioned years ago, and against agents nobody has ever been able to identify. Sloppiness is the obvious explanation. I don't buy it. Nobody hand-writes rules against ghosts unless they badly need a category system that no one has given them.

So they improvise one, from the only material available: whatever name a client announces about itself. HTTP Archive's July 2025 crawl of 16.2 million sites turned up ten distinct AI-related crawler names at measurable prevalence, the most common appearing on 4.5% of desktop sites, up from 2.9% a year earlier. Millions of files now name specific machines by hand.

The interesting thing about those files is not what they contain but what they do. An operator with three named rules is not tightening one door. They are writing three different answers into a single file: yes to the crawler that sends readers back, no to the one ingesting pages for a training run, unresolved for everything else. That is the whole shift, sitting there in plain text. Recognition is what makes differentiation possible, and the moment differentiation is possible, blanket allow and blanket block both start to look like crude instruments.

The categories people are reaching for are categories of purpose. Indexing a page so it surfaces in search results is one activity. Ingesting the same page to train a model is another. Fetching it this second because a person asked an assistant to go check something is a third, and completing a checkout is a fourth. Search indexing was always a bargain: take my page, send me readers. Training keeps the taking and quietly drops the sending. A checkout is not an exchange of attention at all. It is money, and it carries liability.

The edge of the web is turning from a wall into a border. A wall performs one operation on everyone who arrives. A border sorts, and sorting requires classes.

The comparison survives contact with the details, which analogies usually don't. The U.S. State Department is unusually blunt about it: a visa does not authorize entry. It authorizes you to travel to a port of entry and ask. Whether you actually come in gets decided by a different agency altogether, partly on whether what you intend to do matches the class you were admitted under.

Engineering instinct goes wrong here. Cryptographic identity for automated clients looked like the hard problem and is turning out to be the tractable one, because identity is something the sender can prove. Purpose can only be asserted. The receiving side decides whether to believe the assertion and then whether it likes the answer. The IETF working group on authenticating automated clients keeps intent vocabulary explicitly outside its scope, which is honest of them — identity is a standards problem, and purpose is a policy problem being settled in provider dashboards and default configurations. Defaults are policy.

So the failure worth planning around is this one. Verified, then refused. Your agent presents valid credentials, is recognized as precisely what it claims to be, and is turned away because its class is not admitted here.

And the refusal arrives illegible. 403 Forbidden says only that the server understood you and declines. No reason travels with it, so a purpose refusal is indistinguishable from a firewall rule, a botched handshake, a misconfiguration. A draft licensing protocol circulating in this space proposes attaching a scope header and a pointer to governing terms to that 403, which tells you the gap is fillable and unfilled.

It stays unfilled for reasons that have nothing to do with difficulty. The party doing the classifying gains nothing by explaining the classification. A legible refusal costs engineering hours and hands somebody a surface to argue with. An illegible one costs nothing and passes for ordinary flakiness.

Meanwhile your retry logic reads a policy decision as weather, and backs off politely, forever.

In issue 38 we argued that a successful click and a 200 don't establish that the receiving system regarded your agent as legitimate. The inverse deserves equal attention. A failure won't tell you what kind of failure it is, and the gap between a flaky server and a closed door is the gap between waiting and negotiating.