Foundations — Issue 39

Evidence Locker Concept

The Evidence Locker

By Sable Whitford— July 23, 2026

Feature image for article: The Evidence Locker

An agent updates a billing tier, submits a regulatory form, merges a PR. Three days later someone shows up asking who authorized it. You reach for what you have: traces built for engineers debugging execution, outputs built for users seeing results. Neither answers the question. Both look like evidence because they're detailed, but specificity isn't verification. The evidence locker is a third artifact, designed for the stakeholder who arrives after the fact needing to accept, contest, or unwind what an agent did. Its contents scale with how hard the work is to reverse.

Evidence Locker Concept

The Evidence Locker

By Sable Whitford— July 23, 2026

An agent updates a billing tier, submits a regulatory form, merges a PR. Three days later someone shows up asking who authorized it. You reach for what you have: traces built for engineers debugging execution, outputs built for users seeing results. Neither answers the question. Both look like evidence because they're detailed, but specificity isn't verification. The evidence locker is a third artifact, designed for the stakeholder who arrives after the fact needing to accept, contest, or unwind what an agent did. Its contents scale with how hard the work is to reverse.

SQL Evidence Bundle

What a Query Result Needs Before It Counts

By Sable Whitford— July 23, 2026

Feature image for article: What a Query Result Needs Before It Counts

An agent runs a query. $14.2M comes back, clean execution, no errors. Looks like an answer. But the best-evaluated system in the EntSQL benchmark hit 15.9% accuracy when enterprise domain knowledge was required, and the benchmarks themselves carry annotation error rates above 50%. That number isn't an answer. It's a claim wearing a suit. What follows builds the evidence bundle that makes such a claim contestable, element by element, each layer entering because the previous one left something dangerously exposed.

SQL Evidence Bundle

What a Query Result Needs Before It Counts

By Sable Whitford— July 23, 2026

An agent runs a query. $14.2M comes back, clean execution, no errors. Looks like an answer. But the best-evaluated system in the EntSQL benchmark hit 15.9% accuracy when enterprise domain knowledge was required, and the benchmarks themselves carry annotation error rates above 50%. That number isn't an answer. It's a claim wearing a suit. What follows builds the evidence bundle that makes such a claim contestable, element by element, each layer entering because the previous one left something dangerously exposed.

Evidence Across the Stack

GenAI Observability

Traces capture the skeleton of agent execution. Prompt content and tool arguments stay hidden behind opt-in gates.

EntSQL: Grounding Text-to-SQL in Long-Context Enterprise Knowledge

Top systems hit 15.9% accuracy when queries required business knowledge the schema alone couldn't provide.

Pervasive Annotation Errors Break Text-to-SQL Benchmarks and Leaderboards

A First Measurement Study on Authentication Security in Real-World Remote MCP Servers

Adoption and Impact of Command-Line AI Coding Agents at Microsoft

MCP-ITP: An Automated Framework for Implicit Tool Poisoning in MCP

Evidence Across the Stack

GenAI ObservabilityTraces capture the skeleton of agent execution. Prompt content and tool arguments stay hidden behind opt-in gates.

EntSQL: Grounding Text-to-SQL in Long-Context Enterprise KnowledgeTop systems hit 15.9% accuracy when queries required business knowledge the schema alone couldn't provide.

Quick links

Pervasive Annotation Errors Break Text-to-SQL Benchmarks and Leaderboards

A First Measurement Study on Authentication Security in Real-World Remote MCP Servers

Adoption and Impact of Command-Line AI Coding Agents at Microsoft

MCP-ITP: An Automated Framework for Implicit Tool Poisoning in MCP

What's in the Locker

An agent run produces a trace. It logged the calls, returned the output, maybe even succeeded. But a trace tells you what happened. It doesn't tell you whether the outcome was allowed to stand.

That second question demands a record, not a log. And the shape of that record depends entirely on what kind of agent ran and what kind of scrutiny might follow. A dashboard annotation and a wire transfer don't warrant the same evidence bundle. Below are five cards, one per agent type, listing the minimum artifacts that should survive a run. Think of them as checklists shaped by the specific challenge each agent type will most likely face later.

What's in the Locker

An agent run produces a trace. It logged the calls, returned the output, maybe even succeeded. But a trace tells you what happened. It doesn't tell you whether the outcome was allowed to stand.

That second question demands a record, not a log. And the shape of that record depends entirely on what kind of agent ran and what kind of scrutiny might follow. A dashboard annotation and a wire transfer don't warrant the same evidence bundle. Below are five cards, one per agent type, listing the minimum artifacts that should survive a run. Think of them as checklists shaped by the specific challenge each agent type will most likely face later.

SQL Agents

The Number Means Nothing Without the Query

Versioned metric definition and source documentation consulted. Generated SQL with dialect assumptions noted. Raw execution results plus sample rows. Known caveats and exclusions. Reviewer context: who saw the full evidence bundle before the number traveled downstream. Business meaning dies once prose leaves the query behind.

Browser Agents

Browser Runs Start With Who Said Go

Delegated authority chain showing who authorized the action and its scope. Target-site recognition status (did the site accept automated access, or resist it?). State snapshots before and after. Session evidence including auth source and challenges encountered. Documented remedy path for reversal or escalation.

Coding Agents

Keep the Diff, Not the Agent's Summary

The actual diff, not the agent's description of it. Test results and full CI output. Code-owner review record with real review depth, not rubber stamps. Rollback path documentation. Issue linkage proving the change was authorized. The review machinery already exists; evidence should flow through it.

MCP Tools

Tool Calls Need Governance the Protocol Skips

Tool identity and version. Credential scope with token audience and expiry. Exact invocation arguments. Raw tool response, including error states. A bound approval record showing what the human actually confirmed. Server trust basis documenting your organization's review of tool metadata before it was ever called.

RAG Retrieval

Permissions Can Change Between Retrieval and Action

Source document provenance with version and ingestion timestamp. Access-control verification method and when it ran. Chunk lineage showing the parsing and retrieval path. Embedding model version. Freshness timestamps covering source update, re-embedding date, and index propagation delta. What was permitted at retrieval time may not be permitted later.

What's Missing

Agent runs produce logs, traces, outputs. What they routinely fail to produce is the evidence you'd actually need if someone challenged the result.

These seven gaps share a shape. The action completes. The tool was called, the query returned rows, the page loaded, the PR merged. But the institutional meaning of that action stays unresolved. Who authorized it? Under what scope? Against which definitions? With what recognition from the counterparty?

Call them ordinary omissions. They stay invisible until a question arrives that the run's own records can't answer. The ecosystem keeps improving how agents act. It hasn't caught up on how they prove what they did.

What's Missing

Agent runs produce logs, traces, outputs. What they routinely fail to produce is the evidence you'd actually need if someone challenged the result.

These seven gaps share a shape. The action completes. The tool was called, the query returned rows, the page loaded, the PR merged. But the institutional meaning of that action stays unresolved. Who authorized it? Under what scope? Against which definitions? With what recognition from the counterparty?

Call them ordinary omissions. They stay invisible until a question arrives that the run's own records can't answer. The ecosystem keeps improving how agents act. It hasn't caught up on how they prove what they did.

Unsigned Mandates

No Record Exists of Who Authorized the Agent

MCP authorization remains optional. A session log proves the agent held a working token. It does not prove that a specific person delegated a specific task with defined scope and expiry. When the result gets challenged, the challenge has nowhere to land.

Unscoped Credentials

Agent Holds Broader Access Than the Task Needs

AWS warns operators not to treat a readonly flag as a security boundary. When credentials exceed what the task requires and no runtime scope evidence gets captured, demonstrating least privilege after the fact is the problem you cannot solve.

Ungrounded Metrics

The Query Ran Clean but Meaning Went Unverified

EntSQL's best system reached 15.9% execution accuracy on enterprise questions, even with domain documentation provided. SQL can execute cleanly while the business term behind the question was never resolved against an authoritative definition. A polished number and a grounded one look identical.

Unrecognized Visits

Browser Agent Visited but the Site Saw Nothing

Cloudflare now separates AI traffic into Search, Training, and Agent categories, with default blocking planned for agent visits on ad-supported pages. A successful browser session today may face retroactive challenge if neither party recorded what the visit was for.