You have a text-to-SQL prediction file and a benchmark that scored it. Run one command and get, per question, whether the prediction and the gold disagree on the shipped data, the rows that differ, and what the benchmark's own scorer would have said: for a question scored 0, the rows that decided it, and for one scored 1, whether a comparison that keeps duplicates, order and types agrees with the set comparison that credited it.
attestql audit --engine sqlite --dsn sandbox/fixture.sqlite \
--questions tools/audit-sandbox-sqlite/questions.json \
--predictions tools/audit-sandbox-sqlite/predictions.json --out audit
That run is the repository's own sandbox: the three BIRD Mini-Dev golds that upstream reports found wrong (q1029, q879, q207) on a fixture of a few rows, each paired with a corrected statement as its prediction, plus three synthetic questions. Its output, unedited:
q1029 european_football_2 R-ORD NOT_EQUAL smells=none audit/q1029/ q879 formula_1 R-ORD NOT_EQUAL smells=ordering-over-numeric-text audit/q879/ q207 toxicology R-SET NOT_EQUAL smells=none audit/q207/ q900001 synthetic R-ORD GOLD-ONLY smells=not-a-function-of-the-data audit/q900001/ q900002 synthetic R-ORD GOLD-ONLY smells=arbitrary-cut,not-a-function-of-the-data audit/q900002/ q900005 synthetic R-ORD EQUAL smells=ordering-over-numeric-text audit/q900005/ 6 questions: 3 NOT_EQUAL, 5 smells fired, 0 credited by BIRD but NOT_EQUAL (0 multiplicity, 0 type, 0 order, 0 truncation), 0 timed out (0 gold, 0 prediction)
Every NOT_EQUAL line has a directory. audit/q879/counterexample.json says why that one disagrees, trimmed here to the fields a reader opens first:
"gold": "... ORDER BY T2.fastestLapSpeed DESC LIMIT 1" -> "Norwegian"
"second": "... ORDER BY CAST(T2.fastestLapSpeed AS REAL) DESC LIMIT 1" -> "Peruvian"
"bird_ex": {"value": 0, "method": "set(second_rows) == set(gold_rows), ..."}
"differing_rows": {"in_gold_not_in_second": [["Norwegian"]], "in_second_not_in_gold": [["Peruvian"]]}
and audit/q879/smells.json names the mechanism: fastestLapSpeed is a text column holding only numbers, so the gold sorts 9.5 above 10 and the fastest lap is not the one it returns. The same directory holds the two evidence records, each with the statement, the role, the engine, the session settings, the result and its hash, and how to run it again.
NOT_EQUAL never means the gold is wrong. It means these two statements disagree on this data under this rule; here are the rows; decide.
Python 3.11 or later. SQLite needs nothing else; PostgreSQL 16 needs a server.
uv tool install attestql # or: pip install attestql
The README has the three measurements on BIRD's own files, the six upstream reports and their replies, and how to run the command on your own files. Every number has an owning artifact in the claims register.