The whole of this run is a release asset
This site holds 0 of the 18 question directories this run wrote; the archive holds every one:
archive
minidev-sqlite-meta-llama-3-70b-instruct-2.tar.gz size
12,446,219 bytes
sha256
7ce19ef9fc5138ea2f75c123aa3bc7acdc2c5c45ca5f9fb82b0c6e521ebcf764
What it counted
what how many
questions audited 52 EQUAL 35 NOT_EQUAL 17 probes fired 3
the verdicts of run audit-e8bd48bc-864a-4d06-bc3c-74fda70421d0, from summary.json 35 EQUAL 17 NOT_EQUAL Of the questions this run audited: 35 EQUAL, 17 NOT_EQUAL
the probes
probe fired on
ordering-over-numeric-text 0 arbitrary-cut 3 not-a-function-of-the-data 0 float-aggregate-order 0 direction-against-question 0
how many golds each probe fired on, of 52, from summary.json 0 ordering-over-numeric-text 3 arbitrary-cut 0 not-a-function-of-the-data 0 float-aggregate-order 0 direction-against-question Of 52 golds this run read, the probes fired on: arbitrary-cut on 3.
credited by BIRD's own check and NOT_EQUAL here
what how many
credited by BIRD and NOT_EQUAL here 0 multiplicity 0 type 0 order 0 truncation 0 other 0 the test-suite check answered 1 0 the test-suite check answered 0 0
What the run was made of
run
audit-e8bd48bc-864a-4d06-bc3c-74fda70421d0
summary format
attestql/audit/summary/2
question file
data/questions/mini_dev_sqlite.json sha256:4ba5fa8de55856222f484d380d2ba872b380bf79d825de70478e2120cb0fc43b | https://bird-bench.oss-cn-beijing.aliyuncs.com/minidev.zip (sha256 cc48ba16838204e4e214512030cb572eeb5f7bcdd999bae4b9b6ff12ec13b92f, downloaded 2026-09-07), member minidev/MINIDEV/mini_dev_sqlite.json, 2024-06-19
questions in the file
500
prediction file
data/preds-sqlite/predict_mini_dev_meta-llama-3-70b-instruct-2_sqlite.json sha256:d4e0efbafaa26435c2110559b7a24c91efa272ce82b6ec6d2a7176019582e1ae | https://raw.githubusercontent.com/bird-bench/mini_dev/b3d4bcbbae9a96934ad812551eb400c7a3b23c12/llm/exp_result/sql_output_kg/predict_mini_dev_meta-llama-3-70b-instruct-2_sqlite.json, 2024-06-19
statements read
500
keyed by
position
data file
data/minidev/dev_databases/superhero/superhero.sqlite sha256:75e94a2c3236ee3bb2c01fb97a1c4b4c1c269bcefd4eab1d04be323d2d0825b1 | minidev.zip (sha256 cc48ba16838204e4e214512030cb572eeb5f7bcdd999bae4b9b6ff12ec13b92f), member minidev/MINIDEV/dev_databases, superhero/superhero.sqlite, 2024-06-13
server
SQLite 3.53.4 | file=/private/tmp/attestql-runs/data/minidev/dev_databases/superhero/superhero.sqlite | size=237568
database role
file
engine
sqlite
parser
validator audit:sqlglot-sqlite-parse, sqlglot 30.18.0, dialect sqlite
serialization
attestql/audit/2
statement timeout
30 s
fixture digest depth
counts
schema digest
sha256:921b79dd088877e3b6090a31ab919676ab6b45adf398a468edc025b74a77f750
data as of
2026-09-08T05:03:47.274143+00:00 the instant the run started
shuffled copies
prepared
shuffle seed
1
experimental probe
off
The session the run was made in
what the engine reported, 9 settings
sqlite_version
3.53.4
encoding
UTF-8
reverse_unordered_selects
0
query_only
1
journal_mode
delete
data_version
1
compile_options
ATOMIC_INTRINSICS=1,COMPILER=clang-21.0.0,DEFAULT_AUTOVACUUM,DEFAULT_CACHE_SIZE=-2000,DEFAULT_FILE_FORMAT=4,DEFAULT_JOURNAL_SIZE_LIMIT=-1,DEFAULT_MMAP_SIZE=0,DEFAULT_PAGE_SIZE=4096,DEFAULT_PCACHE_INITSZ=20,DEFAULT_RECURSIVE_TRIGGERS,DEFAULT_SECTOR_SIZE=4096,DEFAULT_SYNCHRONOUS=2,DEFAULT_WAL_AUTOCHECKPOINT=1000,DEFAULT_WAL_SYNCHRONOUS=2,DEFAULT_WORKER_THREADS=0,DIRECT_OVERFLOW_READ,ENABLE_API_ARMOR,ENABLE_COLUMN_METADATA,ENABLE_DBSTAT_VTAB,ENABLE_FTS3,ENABLE_FTS3_PARENTHESIS,ENABLE_FTS5,ENABLE_GEOPOLY,ENABLE_MATH_FUNCTIONS,ENABLE_MEMORY_MANAGEMENT,ENABLE_PERCENTILE,ENABLE_PREUPDATE_HOOK,ENABLE_RTREE,ENABLE_SESSION,ENABLE_STAT4,ENABLE_UNLOCK_NOTIFY,MALLOC_SOFT_LIMIT=1024,MAX_ATTACHED=10,MAX_COLUMN=2000,MAX_COMPOUND_SELECT=500,MAX_DEFAULT_PAGE_SIZE=8192,MAX_EXPR_DEPTH=1000,MAX_FUNCTION_ARG=1000,MAX_LENGTH=1000000000,MAX_LIKE_PATTERN_LENGTH=50000,MAX_MMAP_SIZE=0x7fff0000,MAX_PAGE_COUNT=0xfffffffe,MAX_PAGE_SIZE=65536,MAX_SQL_LENGTH=1000000000,MAX_TRIGGER_DEPTH=1000,MAX_VARIABLE_NUMBER=250000,MAX_VDBE_OP=250000000,MAX_WORKER_THREADS=8,MUTEX_PTHREADS,SYSTEM_MALLOC,TEMP_STORE=1,THREADSAFE=1,USE_URI
collation_list
RTRIM,NOCASE,BINARY
case_sensitive_like
0
What the run states about itself
ids the question file states twice
137, 138
prediction positions not compared
487, 488
The questions
question
rule
verdict
class
probes
what was asked
A question whose statement could not be run wrote no directory, so it is a row here and has no page: the summary holds the side that stopped, the step it stopped at and the engine's own message.
The JSON this page was rendered from