In 1994, Martijn Koster proposed robots.txt after a misbehaving crawler accidentally knocked his own server offline. The fix was a plain text file at the root of a website telling crawlers what they could and couldn't look at. It was never formally standardized as an RFC until 2022, twenty-eight years later. It didn't need to be. The convention held because the stakes were low. Crawlers only read.
Over the next two decades, the web kept building infrastructure for its non-human visitors. XML sitemaps made pages easier to discover. Schema.org gave machines a shared vocabulary for interpreting what they found. Each layer solved a harder version of the same problem: helping automated visitors understand what they were looking at. And every one of them rested on an assumption so fundamental nobody bothered stating it. The visitor is here to read.
The pattern is starting over. Files like llms.txt tell AI agents what a site contains and how to navigate it. Structured endpoints make web content machine-parseable. The early energy feels familiar. The web is adapting its surface for a new class of visitor, and the instinct to reach for the search-engine playbook makes sense. That playbook worked.
It worked because reading is cheap. Reading carries no liability. A crawler that misreads a page wastes its own time. Nobody gets billed.
In September 2025, an open standard emerged that lets AI agents complete purchases inside a chat interface using encrypted payment tokens. By December, a major payment network reported hundreds of agent-initiated transactions with production partners and predicted millions of consumers would use agents to buy things by the 2026 holiday season.
These are live checkout flows. They snap the historical analogy cleanly.
Robots.txt is a permission system for looking. Nothing in the search-era infrastructure stack addresses what happens when the visitor can spend money. There is no broadly adopted standard that communicates "you are authorized to transact here." The IETF has active drafts working on agent authentication and delegation. But commercial protocols from payment networks and platform companies shipped first, within months of each other. The governance assumptions for how agents transact on the web are being set by whoever builds fastest. That's a familiar infrastructure pattern, and it tends to be difficult to reverse.
Meanwhile, the web is reaching for the tools it already has. Cloudflare recently extended robots.txt with a "Content-Signal" directive that lets sites specify whether AI can use their content for training, for inference, or for search. A thoughtful addition, and a read-permission tool being stretched to cover a world where the visitor might buy something. You can see the gap in the shape of the solution itself.
When an AI agent completes a purchase, someone is bound by that transaction. The agent itself cannot be. Software has no legal personhood.
Agency law offers a framework: the agent acts on behalf of a principal, the principal bears the consequences. Courts are only beginning to test this. In 2024, a Canadian tribunal held an airline liable for its chatbot's misleading fare advice. That was an advisory chatbot, one that merely described a policy incorrectly. The liability surface when agents commit real dollars, negotiate terms, or accept offers faster than a human can intervene is a question the legal system hasn't yet had to answer.
The search-era parallel is genuinely useful for understanding the shape of what's happening. Standards are emerging in the same messy, overlapping way. Adoption will be uneven. The pattern rhymes. And the search-engine mental model quietly assumes the worst thing a visitor can do is misread the page. The infrastructure the web needs now has to handle authorization, permissioned action, and accountability when something goes wrong. Legibility never had to touch any of that. Different plumbing entirely.
-
NIST enters the picture: The federal AI Agent Standards Initiative, launched in February 2026, is targeting an AI Agent Interoperability Profile by Q4 2026 that includes fundamental research on agent authentication and identity infrastructure.
-
The alphabet soup thickens: Google's developer blog acknowledges the growing wall of competing acronyms in agent protocols — MCP, A2A, UCP, AP2, A2UI, AG-UI — a landscape that looks a lot like the pre-consolidation era of early web standards.
-
EU liability gets concrete: The new EU Product Liability Directive, which member states must implement by December 2026, explicitly includes software and AI as "products" subject to strict liability if found defective — the first major legislative framework that could apply directly to agent-initiated transactions.
-
Governance lags everywhere equally: McKinsey's 2026 AI Trust Maturity Survey found that governance and agentic AI controls lag behind data and technology capabilities across all regions, suggesting the gap between what agents can do and what organizations can govern is globally consistent, not a maturity curve some regions have already climbed.