The EntSQL benchmark asked text-to-SQL systems to answer enterprise questions requiring domain knowledge: metric definitions, reporting conventions, organizational rules. The best evaluated English-input system reached 15.9% accuracy. Set that aside for a moment and consider what the other 84% looks like once it ships.
In coding, wrong output hits a diff, a test suite, a reviewer. Distrust already has a workflow. In analytics, the query produces rows. Rows become a number, and the number becomes a sentence in a deck. Somewhere between the SQL and the sentence, the metric definition, the filters, the caveats all fall away. Nobody later argues with the query. They argue with the conclusion, long after anyone could reconstruct whether "active customers" included trials, paused accounts, or subsidiaries.
A bad database mutation tends to be loud. A bad number can be quiet and durable. It looks precise enough to travel.
The coding contrast: Diffs, tests, CI, code review, ownership, rollback — all native to the domain. Analytics has none of these by default.
Why 15.9% isn't an outlier: On Spider 2.0, an o1-preview agent solved 17.0% of enterprise tasks vs. 91.2% on the toy benchmark. The drop is consistent.
Where meaning rots: Terms like "churn," "net retention," or "open incidents" can return rows before the organization has resolved what the term includes.
What would need to travel with the number: Metric definition and version, source tables and filters, executed SQL, known exclusions, reviewer decision, downstream use.
Where agents move fastest: Domains where the work product is already shaped for challenge. Analytics isn't one of them.