Ask a financial database for Nvidia's revenue from 2020 to 2025, and the query that comes back may never mention revenue. In a worked example from a paper on FLINT, a system Kensho researchers presented at the EMNLP 2026 conference, the SQL filters on dataItemId = 28 and periodTypeId = 1. A separate lookup table records that 28 means Total Revenue and 1 means annual. The schema, the description of tables and columns a language model reads before writing a query, says only that both columns hold integers.
So the business meaning lives somewhere the model can't see by default. Some of it sits in rows of reference tables. The rest sits in conventions analysts carry in their heads and rarely write down.
FLINT translates plain-English questions into SQL; its authors compared it with several published methods running on the same underlying model. Every system, FLINT included, produced queries that executed at least 99% of the time. None of the competing methods answered even half the questions correctly. The most common failure was a valid query aimed at the wrong code: Net Revenue instead of Total Revenue, quarterly figures instead of annual. The paper's appendix states the mechanism plainly:
"the query executes successfully with wrong IDs, so no error signal triggers refinement."
A database will sum whatever item you hand it. It has no way of knowing 28 was the wrong one, so a system that waits for an error before retrying never hears anything went wrong.
Instead, FLINT had finance experts cut a catalog of 11,412 data items to roughly 2,500 in common use, write example queries by hand, and record the conventions:
- revenue means the annual figure unless stated otherwise
- stock price means the closing price
- market capitalization uses the primary listing
- cross-country comparisons convert to one currency
The Nvidia query got "annual" from the first; the question never said it.
With this in place, FLINT answered 68.8% of questions correctly on its financials schema and 65.2% on its transactions schema. "Correctly" here means matching reference answers experts wrote under the same conventions. That is fair, since someone must decide what revenue means, but it makes the benchmark a test of fidelity to house definitions. In production there is no reference answer, so the authors watch indirect signals such as users rephrasing a question, often a sign that the system's definition was not theirs.
The number returned is exact, and it really is in the database. What's false is the confidence that it answers the question asked, a gap worth naming: false precision. Where nobody has recorded what "revenue" means, a clean result reports exactly on a choice no one made. FLINT's contribution is to turn that silent default into a decision someone can point to.
The evidence is thin in particular places. The financial results come from two proprietary schemas, evaluated by the system's builders against baselines they adapted themselves: some reimplemented, some given FLINT's own lookup layer, at least one missing part of its original design. The paper also leaves out what I'd most want to know as a buyer: how the conventions are versioned, who approves changes, and how disputes get settled.
The way the field measures progress makes those questions easy to skip. Execution rates are simple to report, and they flatter everyone. A definitions file is slow, unglamorous work that never shows up in a demo, yet whoever maintains it decides what the agent's answers mean. So ask a vendor where the definitions live, who can change them, and what happens when finance and investor relations disagree about revenue.
-
Scores carry their instruments: In the ASSERT audit study from the EMNLP Industry Track, changing the judge, the simulated user, or the evidence threshold changes reported rates and can reorder system rankings, which is a useful lens for any accuracy figure graded against expert-written answers.
-
Knowledge versus architecture: A myeloma-records study found that giving iterative retrieval the same expert skill library as the agent matched the agent's aggregate concordance, which raises the question of how much of FLINT's lead comes from its curated conventions rather than its pipeline.
-
Definitions as authored policy: Sable Whitford's "The Document Nobody Wants to Write" argues that a runtime cannot know what an application does unless a written policy carries that knowledge, the same burden FLINT places on its expert-maintained configuration files.
-
Checking the persisted result: "The Agent Clicked Save. Check the Database Anyway" makes the case for verifying outcomes in durable data, and FLINT's failures show that such a check also needs an agreed definition of which row counts as right.
