--- title: "Foundations" canonical: "https://current.tinyfish.ai/issue/41/foundations" publication: "TinyFish" publisher: "TinyFish" language: "en-US" guide: "https://current.tinyfish.ai/llms.txt" permalink: "https://current.tinyfish.ai/issue/41/foundations" issue: 41 section: "foundations" description: "Conceptual clarity earned from building at scale" authors: ["Sevda Polat","Sable Whitford"] published: 2026-08-10 updated: 2026-08-06 articles: 27 --- # TinyFish > CURRENT is a journal about AI agents in production, for the engineering and product leaders who have to make them work. Drawing on experience building enterprise web agents, it explains why the demo succeeded and the deployment didn't, and what that means for choosing vendors, briefing the board, and avoiding mistakes others have already made. [Read on the website]() ## Issue 41 — Foundations Conceptual clarity earned from building at scale ### Purpose Over Detection - [Verified, Then Refused]() — By [Sevda Polat]() · Published 2026-08-05 · Format: Purpose Over Detection Site operators are writing robots.txt rules against crawlers that were switched off years ago, and against crawlers nobody has ever identified. You could read that as carelessness, in which case there is nothing to learn from it. Read it as improvisation and it tells you a great deal: people building a classification system that no one has issued them, out of the only material lying around, which is whatever name a client announces about itself. ### Protocol Boundaries - [Web Bot Auth Signs the Sender, Not the Errand]() — By [Sable Whitford]() · Published 2026-08-05 · Format: Protocol Boundaries For years my entire toolkit for identifying a crawler was a user-agent string, an IP range, and a reverse DNS record that any motivated adult could dress up to look wholesome. Web Bot Auth swaps that guess for a cryptographic check, and a narrower one than most coverage suggests. It tells you a key holder sent something to this host inside a time window. Who the agent is working for, and why, are ruled out of scope on purpose. Here's what actually gets signed, and what comes back to your desk. ### Recognition Reading - **Use Cases for Authentication of Web Bots** — [Source]() — Published 2026-08-05 Nottingham's IETF draft lays out the live debates: centralization risk, operator-only identity, and what's still unresolved. - **NIST NCCoE: Accelerating AI Agent Identity and Authorization** — [Source]() — Published 2026-08-05 The first U.S. government concept paper proposing OAuth and SPIFFE-based identity frameworks for autonomous agents. - **2026 Bad Bot Report: Bad Bots in the Agentic Age** — [Source]() — Published 2026-08-05 Thales carves out AI agents as a third traffic category distinct from good bots and bad bots. - **How Cryptographically Signed AI Agents Change the Closing Web** — [Source]() — Published 2026-08-05 A walkthrough of Web Bot Auth's architecture and why forged User-Agent strings are losing their usefulness. - **Cloudflare AI Bot Traffic Controls: Search vs. Agent vs. Training** — [Source]() — Published 2026-08-05 Three-category classification in practice: how site operators get granular control over distinct AI bot behaviors. - **Bot Management: What Works in 2026?** — [Source]() — Published 2026-08-05 Practical look at which detection approaches hold up as AI agents increasingly operate through real browser environments. ### Recognition Stack When an automated agent connects to a website, the site can try to answer a series of questions. Which network sent this request? Who claims to operate it? Can that claim be verified cryptographically? What is the agent trying to do? Each question maps to a different identification layer: IP verification, user-agent strings, cryptographic signatures, traffic classification, bot directories, behavioral fingerprinting. Together they form a recognition stack. Each layer establishes something real, and each hits a hard boundary where its proof runs out. Knowing where those boundaries fall shapes what policies a site can actually enforce. - **Reverse DNS Confirms Network Ownership and Nothing Else** — [Source]() — Published 2026-08-05 Tag: Network Origin Forward-confirmed reverse DNS verifies that a connecting IP belongs to a stated organization's network. Google publishes its crawler IP ranges as JSON files for direct checking. This confirms which network sent a request, but not which user triggered it, or why. - **User-Agent Strings Claim an Identity Without Any Proof** — [Source]() — Published 2026-08-05 Tag: Self Declaration Any HTTP client can set a user-agent header claiming to be Googlebot or Chrome. OpenAI Operator runs inside real Chromium and sends a genuine browser string. Modern spoofing extends to navigator objects and client hints. User-agent works as a first-pass filter, never a final answer. - **Web Bot Auth Signs Requests With Verifiable Keys** — [Source]() — Published 2026-08-05 Tag: Cryptographic Identity The IETF WebBotAuth protocol uses HTTP Message Signatures (RFC 9421) to let bot operators cryptographically sign requests with Ed25519 keys. Sites verify signatures against published public keys. This establishes operator identity per-request, replacing spoofable strings with provable claims about who sent the traffic. - **Signatures Prove the Operator, Not the User's Intent** — [Source]() — Published 2026-08-05 Tag: Operator Limits The IETF charter explicitly excludes end-user authentication and intent vocabulary from Web Bot Auth's scope. A valid OpenAI signature proves OpenAI's infrastructure sent the request. It says nothing about which user initiated it, what task they authorized, or whether the agent stayed within scope. - **Traffic Classification Sorts Search, Agent, and Training Traffic** — [Source]() — Published 2026-08-05 Tag: Purpose Categories Edge networks now classify bot traffic by purpose: Search (indexing), Agent (real-time browsing for a user), and Training (model fine-tuning). A site can permit an operator for search while blocking it for training. Classification draws on self-declaration, directory entries, and behavioral signals. - **Bot Directories Verify Operators Based on Observed Behavior** — [Source]() — Published 2026-08-05 Tag: Verified Directories Verified bot directories list known operators with behavior-based classifications, validated through IP ranges, reverse DNS, or Web Bot Auth. Directory listing lets sites apply consistent rules without per-request verification. But a listing reflects behavior at enrollment time — categories shift if observed patterns later diverge. - **Behavioral Fingerprinting Detects Automation Below the HTTP Layer** — [Source]() — Published 2026-08-05 Tag: Behavioral Signals Recent research shows behavioral signals like typing latency, scroll timing, and mouse movement continuity can identify AI browsing agents with near-perfect accuracy in controlled settings. These signals work even when agents send genuine browser user-agents and route through residential proxies. Evasion-optimized agents may close that gap. - **The Combined Stack Still Cannot Identify End Users** — [Source]() — Published 2026-08-05 Tag: Persistent Gap Together, these layers tell a site which network sent a request, which operator signed it, what purpose category applies, and whether behavior looks automated. They cannot tell a site which end user initiated the action, what task was authorized, or whether the agent honored its mandate. ### Purpose Policy Map Websites used to make one decision about automated traffic: let it through or block it. That's splintering. Cloudflare's July 2026 retirement of its single "block AI bots" toggle, replaced by three independent category controls, reflects a broader move toward purpose-based governance, where rules depend on why a bot visits. Five traffic categories are taking shape, each carrying different default policies, identity requirements, and enforcement realities. The category labels look tidy. The boundaries between them are considerably less stable, and the mechanisms for verifying what a bot actually intends remain early and unevenly adopted. - **Search Crawlers Remain Welcome With Tangled Strings** — [Source]() — Published 2026-08-05 Tag: Search Crawling Search crawling stays allowed by default, even under Cloudflare's September 2026 policy changes. But crawlers like Googlebot bundle search indexing and AI training into the same request. Blocking training means blocking the whole crawler unless operators use granular opt-out tokens like Google-Extended rather than category-level controls. - **Real-Time Agents Get Blocked on Ad Pages** — [Source]() — Published 2026-08-05 Tag: Agent Action Agents acting on a user's behalf in real time are blocked by default on ad-supported pages, on the logic that ads are designed for human attention. Enforcement gets messy: Google's user-triggered fetchers ignore robots.txt entirely, so sites need WAF rules or IP blocking to actually stop them. - **Training Crawlers Hit Mass Opt-Outs and Paywalls** — [Source]() — Published 2026-08-05 Tag: Data Training Over 2.5 million websites now block AI training crawlers. Cloudflare's September 2026 defaults extend this by blocking training on ad-supported pages automatically. A third option is emerging alongside allow and block: HTTP 402 pay-per-crawl, currently in private beta, lets sites price access to their content. - **Transaction Bots Tracked but Missing Managed Controls** — [Source]() — Published 2026-08-05 Tag: Transaction Agents Cloudflare's BotBase classifies transaction bots as a separate category from general agents, but there's no managed preset for them yet. Authorization depends on payment network rules (Visa added agentic-payment requirements in April 2026) and existing anti-fraud systems, not web infrastructure categories. - **Security Scanners Run on Contracts Not Protocols** — [Source]() — Published 2026-08-05 Tag: Security Testing Security testing bots can be classified in BotBase, but authorization for scanning still runs through bug bounty scopes and pentest contracts. Robots.txt has never governed this space. The gap grows as AI agents autonomously probe application logic during ordinary browsing tasks. - **Categories Break Where HTTP Requests Look Identical** — [Source]() — Published 2026-08-05 Tag: Blurred Boundaries A retrieval crawler and a user-triggered agent look the same at the HTTP layer. A price-monitoring scraper traverses the same pages as a transaction bot. Agentic browsers using standard Chrome user agents carry no bot signal at all. Purpose classification depends on operator honesty, which adversarial traffic won't provide. - **Cryptographic Bot Identity Exists but Adoption Lags** — [Source]() — Published 2026-08-05 Tag: Identity Layer Web Bot Auth lets operators sign requests with Ed25519 cryptographic keys, and a companion IETF draft adds purpose declarations through Signature Agent Cards. Adoption remains thin: Cloudflare and Akamai verify signatures at the edge, Google signs experimentally, and most agent operators haven't implemented it. ### Past Articles - [Somebody Has to Hold the Basket]() — By [Sevda Polat]() · Published 2026-07-29 · Format: Where State Lives Now A basket, a second item, a card that only the person at the keyboard can produce, and a refund that may take a day to land. Four calls, one story. - [The Exponent Borrows Two Things Nobody Has Measured]() — By [Sevda Polat]() · Published 2026-07-29 · Format: Reliability As Shape Every agent deployment estimate runs the same arithmetic. Per-step success rate, raised to the number of steps, watch the floor open. Directionally right, and nobody should be argued out of it. But it borrows two things the evidence hasn't supplied: one per-step rate to raise, and steps that fail independently of each other. Five runs of the same task, same agent, same instruction, different bill. Horizon is doing most of the damage in the reliability results, and it is the variable those studies measure least directly. - [The Evidence Locker]() — By [Sable Whitford]() · Published 2026-07-23 · Format: Evidence Locker Concept An agent updates a billing tier, submits a regulatory form, merges a PR. Three days later someone shows up asking who authorized it. You reach for what you have: traces built for engineers debugging execution, outputs built for users seeing results. Neither answers the question. Both look like evidence because they're detailed, but specificity isn't verification. The evidence locker is a third artifact, designed for the stakeholder who arrives after the fact needing to accept, contest, or unwind what an agent did. Its contents scale with how hard the work is to reverse. - [What a Query Result Needs Before It Counts]() — By [Sable Whitford]() · Published 2026-07-23 · Format: SQL Evidence Bundle An agent runs a query. $14.2M comes back, clean execution, no errors. Looks like an answer. But the best-evaluated system in the EntSQL benchmark hit 15.9% accuracy when enterprise domain knowledge was required, and the benchmarks themselves carry annotation error rates above 50%. That number isn't an answer. It's a claim wearing a suit. What follows builds the evidence bundle that makes such a claim contestable, element by element, each layer entering because the previous one left something dangerously exposed. ## Other sections in Issue 41 - [Daily Brief]() - [Market Pulse]() - [Practitioner's Corner]() - [Echoes]() - [Vision]() ## More - [Publication guide]() - [Issue archive]()