HTTP status code 402 has existed since 1997. Its full definition in the original spec: "Reserved for future use." Nobody implemented it. For twenty-eight years it sat in the protocol, an empty room in a building, waiting for someone to decide what it was for.
On July 1, 2025, Cloudflare moved in. The company launched Pay per Crawl, a service that lets website owners charge AI companies for access to their content. When an AI crawler requests a page from a participating site, Cloudflare returns a 402 response with a price header. The crawler can pay and retry, or leave.
Cloudflare called the launch date "Content Independence Day," which tells you something about how the company sees its role. Liberation, here, means a commercial intermediary standing between publishers and the crawlers extracting their content. The product rested on data Cloudflare was uniquely positioned to collect. The company serves roughly 20% of the web's traffic, which means it can observe how often AI crawlers visit a site versus how often they send a human visitor back. Google's crawler scraped sites 14 times for every click it returned. OpenAI's ratio was 1,700 to 1. Anthropic's was 73,000 to 1.
The billing product got the headlines. The quieter decision may matter more. Alongside Pay per Crawl, Cloudflare changed a default: new websites on its platform would now block all AI crawlers automatically. More than a million existing customers had already used the one-click opt-out tool. The default flip formalized what the market was already choosing. There's a difference between offering a tool and setting a default, though. Defaults are governance decisions. This one was made by a company that serves a fifth of the web.
The mechanics of Pay per Crawl reveal where the leverage concentrates. Both publisher and crawler must hold Cloudflare accounts. Cloudflare acts as Merchant of Record, handling billing, authentication, and payment distribution. As of late 2025, the product remained in private beta with no public adoption figures. Both sides of the transaction pass through a single infrastructure provider.
For thirty years, the web's answer to "who gets to crawl what" was a cooperative convention. When the convention stopped working, no protocol emerged to replace it. No regulatory body stepped in. An infrastructure company that already controlled DNS proxying, CDN delivery, bot detection, and web application firewalls added one more capability to the stack: billing. Cloudflare was already sitting in the right place in the network, handling the traffic, identifying the bots, serving the pages. The tollbooth was a natural extension of the road.
Whether the product works is probably beside the point. For sites on Cloudflare's network, it probably will. The enforcement layer for web governance is now operated by a commercial actor whose revenue grows with the problem it solves. A protocol wants to be adopted universally and then forgotten. A product wants to be adopted universally and then renewed.