AttestQL

a static viewer over the JSON an audit writes

Audit text-to-SQL gold and predictions on PostgreSQL with typed replay evidence.

Ten minutes

Two commands, no server and no account. The second audits the sandbox the package carries and writes a directory of JSON, one per question.

uv tool install attestql        # or: pip install attestql
attestql demo --out demo

The output below is that command's own, printed while this page was built. The run it names is this build's: run audit-b91cefcf-535b-4595-be15-2bc84948c208, data as of 2026-09-08T06:49:21.412539+00:00. Yours will carry your own.

q1029 european_football_2 R-ORD  NOT_EQUAL  smells=none  demo/audit/q1029/
q879  formula_1   R-ORD  NOT_EQUAL  smells=ordering-over-numeric-text  demo/audit/q879/
q207  toxicology  R-SET  NOT_EQUAL  smells=none  demo/audit/q207/
q900001 synthetic   R-ORD  GOLD-ONLY  smells=not-a-function-of-the-data  demo/audit/q900001/
q900002 synthetic   R-ORD  GOLD-ONLY  smells=arbitrary-cut,not-a-function-of-the-data  demo/audit/q900002/
q900005 synthetic   R-ORD  EQUAL      smells=ordering-over-numeric-text  demo/audit/q900005/
6 questions: 3 NOT_EQUAL, 5 smells fired, 0 credited by BIRD but NOT_EQUAL (0 multiplicity, 0 type, 0 order, 0 truncation), 0 timed out (0 gold, 0 prediction)
rerun: attestql audit --engine sqlite --dsn demo/fixture.sqlite --questions demo/questions.json --predictions demo/predictions.json --out demo/audit

One of them, in full

NOT_EQUAL states that these two statements disagree on this data under this rule. It does not state which of them is wrong.

in gold, not in the prediction, 1 row

from counterexample.json, 1 row, up to 25 shown per side

side nationalityTEXT
gold Norwegian

in the prediction, not in gold, 1 row

from counterexample.json, 1 row, up to 25 shown per side

side nationalityTEXT
second Peruvian

NOT_EQUAL never means the gold is wrong. It means these two statements disagree on this data under this rule; the rows above are the disagreement; a maintainer decides.

The whole of that question, with both statements, both evidence records, the probes and the JSON each part was rendered from.

What the published runs found

credited by BIRD's own check and NOT_EQUAL here
164161 of them have a page here; the other 3 are whole in the release assets.
of those, classified by hand as a gold that does not answer its question
6966 of them have a page here; the other 3 are whole in the release assets.
BIRD dev golds classified by hand the same way
23
credited by BIRD's own check and NOT_EQUAL here, cut into what was classified by hand and the rest, from aggregate.json69 of those, classified by hand as a gold that does not answer its question95 the rest
credited by BIRD's own check and NOT_EQUAL here: 69 of those, classified by hand as a gold that does not answer its question, 95 the rest

The runs

Each row links to a page rendered from a summary.json the audit wrote. Where a connection is one database file, one prediction file is a group of eleven runs and the row is the group. Every benchmark is indexed on its own page.

sandbox

run date engine question set question set digest
demo 2026-09-08T06:49:21.412539+00:00 sqlite demo/questions.json sha256:7ad58addc070b425cabfc9d67bbd9c95840253160cf3b73d672e9aeb4897f5ff

bird-dev-sqlite

group runs audited compared GOLD-ONLY probes fired credited by BIRD and NOT_EQUAL ERROR
dev-20251106 11 1,534 0 1,531 47 0 3

minidev-pg

run date engine question set question set digest
gpt-35-turbo 2026-09-08T05:21:06.155844+00:00 postgresql https://bird-bench.oss-cn-beijing.aliyuncs.com/minidev.zip (sha256 cc48ba16838204e4e214512030cb572eeb5f7bcdd999bae4b9b6ff12ec13b92f, downloaded 2026-09-07), member minidev/MINIDEV/mini_dev_postgresql.json sha256:d2731292f20b8d8569cd956dd747ffe1df13cd625076263e38ae9ebcef50b1ab
gpt-35-turbo-instruct 2026-09-08T05:20:16.520067+00:00 postgresql https://bird-bench.oss-cn-beijing.aliyuncs.com/minidev.zip (sha256 cc48ba16838204e4e214512030cb572eeb5f7bcdd999bae4b9b6ff12ec13b92f, downloaded 2026-09-07), member minidev/MINIDEV/mini_dev_postgresql.json sha256:d2731292f20b8d8569cd956dd747ffe1df13cd625076263e38ae9ebcef50b1ab
gpt-4 2026-09-08T05:23:23.924963+00:00 postgresql https://bird-bench.oss-cn-beijing.aliyuncs.com/minidev.zip (sha256 cc48ba16838204e4e214512030cb572eeb5f7bcdd999bae4b9b6ff12ec13b92f, downloaded 2026-09-07), member minidev/MINIDEV/mini_dev_postgresql.json sha256:d2731292f20b8d8569cd956dd747ffe1df13cd625076263e38ae9ebcef50b1ab
gpt-4-32k 2026-09-08T05:21:47.344133+00:00 postgresql https://bird-bench.oss-cn-beijing.aliyuncs.com/minidev.zip (sha256 cc48ba16838204e4e214512030cb572eeb5f7bcdd999bae4b9b6ff12ec13b92f, downloaded 2026-09-07), member minidev/MINIDEV/mini_dev_postgresql.json sha256:d2731292f20b8d8569cd956dd747ffe1df13cd625076263e38ae9ebcef50b1ab
gpt-4-turbo 2026-09-08T05:22:38.535771+00:00 postgresql https://bird-bench.oss-cn-beijing.aliyuncs.com/minidev.zip (sha256 cc48ba16838204e4e214512030cb572eeb5f7bcdd999bae4b9b6ff12ec13b92f, downloaded 2026-09-07), member minidev/MINIDEV/mini_dev_postgresql.json sha256:d2731292f20b8d8569cd956dd747ffe1df13cd625076263e38ae9ebcef50b1ab
meta-llama-3-70b-instruct-2 2026-09-08T05:20:16.519959+00:00 postgresql https://bird-bench.oss-cn-beijing.aliyuncs.com/minidev.zip (sha256 cc48ba16838204e4e214512030cb572eeb5f7bcdd999bae4b9b6ff12ec13b92f, downloaded 2026-09-07), member minidev/MINIDEV/mini_dev_postgresql.json sha256:d2731292f20b8d8569cd956dd747ffe1df13cd625076263e38ae9ebcef50b1ab
meta-llama-3-8b-instruct-2 2026-09-08T05:31:08.958184+00:00 postgresql https://bird-bench.oss-cn-beijing.aliyuncs.com/minidev.zip (sha256 cc48ba16838204e4e214512030cb572eeb5f7bcdd999bae4b9b6ff12ec13b92f, downloaded 2026-09-07), member minidev/MINIDEV/mini_dev_postgresql.json sha256:d2731292f20b8d8569cd956dd747ffe1df13cd625076263e38ae9ebcef50b1ab
mistralai-mixtral-8x7b-instru-4 2026-09-08T05:24:33.608370+00:00 postgresql https://bird-bench.oss-cn-beijing.aliyuncs.com/minidev.zip (sha256 cc48ba16838204e4e214512030cb572eeb5f7bcdd999bae4b9b6ff12ec13b92f, downloaded 2026-09-07), member minidev/MINIDEV/mini_dev_postgresql.json sha256:d2731292f20b8d8569cd956dd747ffe1df13cd625076263e38ae9ebcef50b1ab
phi-3-medium-128k-instruct-1 2026-09-08T05:25:39.400422+00:00 postgresql https://bird-bench.oss-cn-beijing.aliyuncs.com/minidev.zip (sha256 cc48ba16838204e4e214512030cb572eeb5f7bcdd999bae4b9b6ff12ec13b92f, downloaded 2026-09-07), member minidev/MINIDEV/mini_dev_postgresql.json sha256:d2731292f20b8d8569cd956dd747ffe1df13cd625076263e38ae9ebcef50b1ab

minidev-pg-gold-only

run date engine question set question set digest
hf 2026-09-08T05:18:53.188748+00:00 postgresql https://huggingface.co/datasets/birdsql/bird_mini_dev/resolve/f65faf4ae3b638c1fa6df1d3370c8d92c8366301/data/mini_dev_pg-00000-of-00001.json (commit f65faf4a, downloaded 2026-09-08) sha256:7fa740ef9225389cff6c34432120e8325d0ca3008d73db1ae38731234bc10da7
zip 2026-09-08T05:18:53.188261+00:00 postgresql https://bird-bench.oss-cn-beijing.aliyuncs.com/minidev.zip (sha256 cc48ba16838204e4e214512030cb572eeb5f7bcdd999bae4b9b6ff12ec13b92f, downloaded 2026-09-07), member minidev/MINIDEV/mini_dev_postgresql.json sha256:d2731292f20b8d8569cd956dd747ffe1df13cd625076263e38ae9ebcef50b1ab

minidev-sqlite

group runs audited compared EQUAL NOT_EQUAL ERROR probes fired credited by BIRD and NOT_EQUAL
gpt-35-turbo 11 498 409 164 245 89 24 27
gpt-35-turbo-instruct 11 498 372 144 228 126 23 26
gpt-4 11 498 472 209 263 26 26 32
gpt-4-32k 11 498 460 205 255 38 25 32
gpt-4-turbo 11 498 422 190 232 24 29 76
meta-llama-3-70b-instruct-2 11 498 440 177 263 58 23 29
meta-llama-3-8b-instruct-2 11 498 311 187 207 104 16 21
mistralai-mixtral-8x7b-instru-4 11 498 248 250 157 91 14 16
phi-3-medium-128k-instruct-1 11 498 366 235 131 132 20 25

Read on