An agent runs a query. A number comes back: $14.2M. No errors. Specific. Looks settled.
It is not settled.
The best-evaluated system in the EntSQL benchmark hit 15.9% accuracy when enterprise domain knowledge was required. The benchmarks themselves aren't clean either: audits found annotation error rates of 52.8% in BIRD Mini-Dev and 62.8% in Spider 2.0-Snow. So that $14.2M is a claim. The evidence bundle exists to give the claim enough structure that someone can actually challenge it, and I want to walk through the construction because each layer only makes sense once you feel the gap it closes.
Metric definition comes first, because without it you don't even know what you're looking at. $14.2M of what? "Active customers" at Wayfair means individuals who purchased at least once in the preceding twelve months. At Torrid, it means customers who completed at least one transaction in the preceding four quarters, identified through loyalty-program data, covering both in-store and online. Same term. Different qualifying event, different identity rule, different channel boundary. None of that lives in column names. Without the definition pinned to the result, you picked a meaning without knowing you picked.
Fine. You have a definition. But where did it come from? Source documents earn their keep here. Two teams pull the same metric. Finance works from a governed semantic layer updated last quarter. Sales works from a wiki page last touched in 2023, before the company changed its return-treatment policy. Both definitions look authoritative. The agent picks one. Nobody downstream knows which, and you absolutely cannot tell from the number itself. Pinning the source to the definition turns "this is what we mean by revenue" from an assertion into something traceable. When two teams get different numbers from the same question, the source documents are how you find out where the fork happened.
Now: the generated SQL. You need it because the translation from definition to query is exactly where things break. EntSQL's error analysis found 54.6% of failures were wrong filters, 14.4% wrong scope, 10.0% wrong aggregation. Every single one of those errors produces a result that executes cleanly and looks perfectly reasonable. The SQL is the bridge between what you meant and what you got. If it's not visible, nobody can check whether the bridge holds.
But even correct SQL on a correct definition can produce a number that implies a completeness it doesn't have. That's what caveats close. Wayfair's "orders delivered" includes orders that may later be returned, and actual delivery dates may be estimated when unavailable. Torrid can attribute 97% of net sales, excluding private-label credit card funds, to active customers through its loyalty program. Even that 97% has a boundary condition baked in. A revenue number without those boundaries surfaced is silently asserting a precision about timing, finality, and coverage that the underlying data does not support. The caveat is the difference between a number someone can reason about and a number that just looks clean enough to stop asking questions. I have watched people make eight-figure decisions on numbers that looked clean enough to stop asking questions.
Which brings you to the layer that makes the rest of them useful: reviewer context. Not a summary of what's in the bundle. A judgment the bundle cannot make on its own. The same revenue figure might be perfectly adequate for a weekly team standup and catastrophically insufficient for a board presentation or a deal valuation. Fitness-for-purpose doesn't live in the SQL, the definition, or the caveats. It lives in the relationship between the number and the decision it's about to inform, and only a human who can see the full chain (question to definition to source to SQL to result to caveats) can make that call.
Each layer entered because the one before it left something exposed. The full bundle is the minimum structure that makes a query result contestable rather than merely consumable.
Contestability is what separates an answer you can act on from one you're just hoping is right. And hope, as anyone who has ever been paged at 3 AM already knows, is not how you run anything.
- Traces aren't records yet: OpenTelemetry's GenAI semantic conventions now capture model identity, token counts, and tool-call structure, but prompt contents and tool arguments aren't logged by default because they can contain sensitive data, which means the trace alone won't reconstruct the evidence chain a reviewer needs.
- Revenue recognition is deeper: IFRS 15 requires identifying performance obligations, estimating variable consideration, and determining point-in-time versus over-time recognition, all of which shape what "revenue" means in a query before a single SQL filter is written.
- EntSQL's failure taxonomy: The benchmark's error analysis categorizes wrong filters, wrong scope, and wrong aggregation as distinct failure modes, and 96% of its 1,066 examples require domain knowledge beyond question and schema to resolve correctly.
- Benchmark ground truth itself: A 2026 audit of major text-to-SQL benchmarks found that correcting annotations shifted model rankings by up to nine positions, which means leaderboard placement may reflect annotation quality as much as system capability.