--- title: "Foundations" canonical: "https://current.tinyfish.ai/issue/33/foundations" publication: "TinyFish" publisher: "TinyFish" language: "en-US" guide: "https://current.tinyfish.ai/llms.txt" permalink: "https://current.tinyfish.ai/issue/33/foundations" issue: 33 section: "foundations" description: "Conceptual clarity earned from building at scale" authors: ["Nora Kaplan","Rina Takahashi"] published: 2026-06-15 updated: 2026-06-12 articles: 22 --- # TinyFish > CURRENT is a journal about AI agents in production, for the engineering and product leaders who have to make them work. Drawing on experience building enterprise web agents, it explains why the demo succeeded and the deployment didn't, and what that means for choosing vendors, briefing the board, and avoiding mistakes others have already made. [Read on the website]() ## Issue 33 — Foundations Conceptual clarity earned from building at scale ### Reliability Lens - [The Number You Didn't Ask About]() — By [Nora Kaplan]() · Published 2026-06-11 · Format: Reliability Lens An agent succeeds at a task 70% of the time. Frame the results one way and you can report 97%. Frame them differently and you report 34%. Same system, same math, a 63-point gap created entirely by which question the benchmark decided to answer. Nearly every major agent benchmark chooses the generous framing. The number that captures what production actually feels like, consecutive successes across a full workload, is one that almost nobody reports. ### Perception Walkthrough - [The Accessibility Tree Wasn't Built for AI Agents]() — By [Rina Takahashi]() · Published 2026-06-11 · Format: Perception Walkthrough Accessibility trees were built for screen readers, not AI agents. But both share the same constraint: neither can see a web page. Both need it translated into structured, sequential text. Playwright MCP builds its entire browser automation architecture around that convergence, using accessibility snapshots that cost a fraction of what screenshots require per interaction. The token savings are real and they compound across multi-step workflows. So does the downside: on 95.9% of the web's top million pages, the underlying markup has detectable failures. The agent inherits every one of them. ### Further Reading - **The Three Architectures of Browser Agents** — [Source]() — Published 2026-06-11 Maps the full perception design space the walkthrough occupies one corner of. - **Demystifying Evals for AI Agents** — [Source]() — Published 2026-06-11 A layered eval methodology for teams ready to move from measuring reliability to building around it. - **The Reliability Gap: Agent Benchmarks for Enterprise** — [Source]() — Published 2026-06-11 Extends pass^k into category-level verdicts on which agent tasks are production-ready today. - **How AI Agents See Your Website: The Accessibility Tree Explained** — [Source]() — Published 2026-06-11 The web-quality reality check behind accessibility-tree agents, with data on how broken the substrate actually is. ### Key Distinctions The agent ecosystem adopts vocabulary faster than it pressure-tests it. A term catches on, architectures get built around the assumption, and then someone discovers that two genuinely different concepts have been sharing a label. By then the confusion has already shaped security postures, eval pipelines, and deployment decisions. These six distinctions can look like semantics from a distance. Up close, the gaps are structural. A sixty-percentage-point swing between two eval metrics applied to the same agent. A protocol everyone treats as a governance layer that explicitly declines to be one. Vocabulary worth sharpening before it costs you something. - **pass@k and pass^k Measure Completely Different Things** — [Source]() — Published 2026-06-11 Tag: Eval Metrics Confused: treating "succeeded in k tries" and "succeeded every try" as variations of the same score. They aren't. An agent with 70% per-trial success hits ~97% on pass@3 (capability) but just 34% on pass^3 (reliability). Same agent, opposite conclusions. - **Injection and Poisoning Enter Through Different Doors** — [Source]() — Published 2026-06-11 Tag: Security Vectors Confused: lumping both under "prompt attacks." Prompt injection is an input-validation problem where malicious instructions slip through unsanitized user content. Tool poisoning is a supply-chain problem where server-side metadata the agent trusts was authored by an attacker. Defending the input channel does nothing about the supply chain, and vice versa. - **Who Decides the Next Step: Code or Model** — [Source]() — Published 2026-06-11 Tag: System Design Confused: calling anything that uses an LLM an "agent." A system that calls GPT-4 ten times in a fixed sequence is a workflow if code controls the routing. A system where the model picks its own tools and loops is an agent. The dividing line is control architecture, not model size or capability. - **MCP Standardizes Plumbing, Not Policy or Governance** — [Source]() — Published 2026-06-11 Tag: Tool Protocols Confused: assuming a standard integration protocol handles authorization and security. MCP defines how agents discover and call tools. It does not enforce least privilege, scope tokens, define audit policy, or verify tool quality. Authorization remains optional in the spec. Trend Micro found 492 MCP servers exposed with zero authentication. - **Durable Execution Survives Crashes, Regular Execution Doesn't** — [Source]() — Published 2026-06-11 Tag: Runtime Infrastructure Confused: assuming a workflow that runs will keep running. Execution means the code ran. Durable execution means it checkpointed progress, survived a crash, and resumed without duplicating side effects. A server restart at 3am either loses the customer's half-finished task or retries and doubles the order. That's the gap. - **Browser Agents Inherit the User's Full Privilege Surface** — [Source]() — Published 2026-06-11 Tag: Agent Surfaces Confused: treating browser-based and API-based agents as interchangeable approaches. API-first agents call documented endpoints with typed inputs and narrow contracts. Browser-first agents operate through the user's most privileged interface, crossing app boundaries and acting with whatever credentials the session holds. The privilege surface is everything a logged-in user can reach. ### Benchmark Landscape Six benchmarks cover the agent capability landscape from coding and web navigation to function calling and workplace autonomy. Scores are climbing fast. Nearly all of them answer the same narrow question, though: can the agent do this once? One benchmark in this set, τ-bench, was built to ask the harder version: can it do this reliably, every time? That asymmetry shapes everything downstream. How we evaluate, what we miss when we deploy, and which failures never surface until production. Scores shift as new models enter evaluation. Dates are noted on each card. Verify against primary leaderboards before making decisions. - **SWE-bench Verified: Real GitHub Bug Fixes** — [Source]() — Published 2026-06-11 Tag: Coding Tasks Agents tackle human-validated software engineering tasks drawn from actual GitHub issues. Leading score hit 74.4% by end of 2025, though scores have climbed further since, accompanied by growing contamination concerns. Reports pass@k only. Check the primary leaderboard for current numbers. - **GAIA Pushes Multi-Step Reasoning With Tools** — [Source]() — Published 2026-06-11 Tag: Reasoning Tasks Tests real-world assistant tasks that chain together web browsing, tool use, and multi-step reasoning across three difficulty tiers. Orchestrated agent systems reached ~90% by end of 2025. Bare models score far lower. Reports pass@k only. - **τ-bench Exposes What Repeated Runs Reveal** — [Source]() — Published 2026-06-11 Tag: ⚑ Reliability Metric Policy-adherent tool use in retail and airline customer service. Agents hit 80% on a single attempt, but pass^8 drops below 25% in original experiments. The only benchmark here reporting both capability (pass@k) and reliability (pass^k). Worth sitting with that gap. - **WebArena Navigates Realistic E-Commerce and SaaS** — [Source]() — Published 2026-06-11 Tag: Web Navigation Multi-step web navigation across e-commerce sites, forums, and content management systems. GPT-4's original baseline sat at 14.41%. Recent agents clear 60%, with top entries approaching 69% by mid-2026. Reports pass@k only. - **BFCL Scores Tool Calling Across Languages** — [Source]() — Published 2026-06-11 Tag: Function Calling Function-calling accuracy evaluated via AST matching across Python, Java, JavaScript, and REST APIs. Leading score around 77.5% at end of 2025, now updated to V4 with agentic evaluation categories. Reports pass@k only. - **TheAgentCompany: Full Workplace Autonomy, Humbling Results** — [Source]() — Published 2026-06-11 Tag: Workplace Tasks Simulated workplace tasks that blend web browsing, coding, program execution, and communication with simulated coworkers. The best agent completed 30% of tasks in the NeurIPS 2025 version. Comfortably the hardest benchmark in this set. Reports pass@k only. ### Past Articles - [What a Website Chooses to Tell an Agent]() — By [Rina Takahashi]() · Published 2026-06-03 · Format: WebMCP Walkthrough A new browser protocol lets websites publish structured tool definitions that agents can call directly, skipping the page entirely. A site registers a product search with four categories in the enum. The actual catalog has fifty. The agent receives that simplified view as the complete picture and has no mechanism to ask what else exists. When a website decides what to tell an agent about itself, it's also deciding what disappears. - [Where You Can Afford to Be Surprised]() — By [Rina Takahashi]() · Published 2026-06-03 · Format: The Control Question "Agent" means at least three different things right now, and the confusion isn't just semantic. Underneath the terminology sits a design decision with real operational consequences: who picks the next step, your code or the model? Code-determined paths produce bounded, explainable failures. Model-determined paths produce creative ones you didn't plan for. Most production systems need both. The hard part is drawing the line between them, and it's a risk-tolerance decision that happens to look like an architecture one. - [What Browser Agents See Before They Think]() — By [Rina Takahashi]() · Published 2026-05-28 · Format: The Perception Layer Before a browser agent reasons about a task, it needs a representation of the page. Raw HTML, a screenshot, or the accessibility tree. That choice, made once and early, sets the token budget, failure modes, and reliability ceiling for everything downstream. OpenAI, Microsoft, and Perplexity built their agent systems independently and all three landed on the same primary perception layer: infrastructure originally designed for people who can't see screens. The convergence is worth understanding. - [Where Agent Behavior Actually Gets Decided]() — By [Nora Kaplan]() · Published 2026-05-28 · Format: Context Over Instruction A developer's coding agent worked perfectly for thirty minutes, then started contradicting its own decisions. The prompt was the same. The model was the same. Everything else in the context window had been accumulating. His fix was radical: stop carrying context forward entirely. It worked. Across production agent systems, the same asymmetry kept surfacing — a carefully engineered prompt drowning in accumulated tool outputs performed worse than a simple prompt in a clean, focused context. The field needed a word for the thing practitioners were actually optimizing, because "prompt" didn't describe it. ## Other sections in Issue 33 - [Daily Brief]() - [Market Pulse]() - [Practitioner's Corner]() - [Echoes]() - [Vision]() ## More - [Publication guide]() - [Issue archive]()