R-ORD GOLD-ONLY not-a-function-of-the-data
formula_1 · dev_20251106-00000-of-00001 from https://huggingface.co/datasets/birdsql/bird_sql_dev_20251106/resolve/3c11fb193e5439b338e23677fa0aae11e8b85db9/data/dev_20251106-00000-of-00001.json (commit 3c11fb19, downloaded 2026-09-07)
How many wins was achieved by the oldest racer? Indicate his/her full name.
the hint the set supplies: oldest racer refers to MIN(dob); full name refers to forename, surname.
This question was audited without a prediction beside it, so there is nothing to compare the gold with. The probes below read the gold alone.
read by hand, 2026-09-07: wrong the gold does not answer its question on this data
a bare SUM with no GROUP BY beside two ungrouped columns: the total is every driver's wins added together (7,890) and the name beside it is one arbitrary row's
a maintainer's reading of this question, out of classification.json, copied from plans/reports/bird-dev-sqlite-260907/classification.json. It is not a verdict and nothing above it was computed from it.
SELECT SUM(T1.wins),T2.forename, T2.surname FROM driverStandings AS T1 INNER JOIN drivers AS T2 on T1.driverId = T2.driverId ORDER BY T2.dob ASC LIMIT 1sha256:95586ad9cbebd6bd17ab272e5754212cef9669f964611847cfb7439ea72fc8eb
from evidence-gold.json, 1 row
| SUM(T1.wins)INTEGER | forenameTEXT | surnameTEXT |
|---|---|---|
| 7890 | Lewis | Hamilton |
A smell is a mechanical reason to read this gold statement again. It is a heuristic: it does not state that the statement is wrong, and a maintainer decides.
this statement orders by a text column holding only numbers, and ordering it as a number gives a different answer, so the gold may be sorting 9.5 above 10
no ORDER BY key resolves to a text column
{
"heuristic": true,
"reason": "no ORDER BY key resolves to a text column",
"keys": [
{
"key": "T2.dob",
"column": "drivers.dob",
"declared_type": "DATE",
"not_applicable": "the column is not declared as text"
}
]
}this statement cuts its result at a LIMIT that does not decide which rows come back, so a different but equally correct statement can return other rows and score zero
{
"heuristic": true,
"cut": 1,
"offset": 0,
"distinct_kept": false,
"unbounded_sql": "SELECT SUM(T1.wins), T2.forename, T2.surname, T2.dob AS attestql_ordering_key_0 FROM driverStandings AS T1 INNER JOIN drivers AS T2 ON T1.driverId = T2.driverId ORDER BY T2.dob ASC",
"unbounded_rows": 1,
"projected_columns": [
"SUM(T1.wins)",
"forename",
"surname"
],
"ordering_key_columns": [
"attestql_ordering_key_0"
],
"ordering_keys": [
{
"key": "T2.dob",
"direction": "asc",
"nulls": "first",
"nulls_first_in_effect": true,
"returned_rows_null_in_this_key": 0,
"fires": false
}
],
"case": null
}rerun over the same rows in another physical order this statement gives another answer, so its result depends on how the rows are stored and not only on the data
from smells.json, 1 row
| 7890 | Volker | Weidler |
{
"heuristic": true,
"rule": "R-ORD",
"baseline_result_hash": "sha256:95586ad9cbebd6bd17ab272e5754212cef9669f964611847cfb7439ea72fc8eb",
"baseline_result": {
"columns": [
{
"name": "SUM(T1.wins)",
"declared_type": "INTEGER"
},
{
"name": "forename",
"declared_type": "TEXT"
},
{
"name": "surname",
"declared_type": "TEXT"
}
],
"row_count": 1,
"truncated": false,
"rows_shown": 1,
"rows": [
[
{
"type": "int",
"value": 7890
},
{
"type": "str",
"value": "Lewis"
},
{
"type": "str",
"value": "Hamilton"
}
]
],
"result_hash": "sha256:95586ad9cbebd6bd17ab272e5754212cef9669f964611847cfb7439ea72fc8eb"
},
"planner_statistics": {},
"shuffle": {
"seed": "1",
"row_limit": 300000,
"tables": [
"driverStandings",
"drivers"
],
"tables_not_shuffled": [],
"tables_skipped_for_size": {
"lapTimes": 420369
},
"tables_not_reached_by_a_copy": {}
},
"shuffled_copies": {
"run": true,
"verdict": "not_equal",
"differs": true,
"result_hash": "sha256:48b715a576c7daabcc6b152a00c646773a078aee93f3589e7050878d7b0972cb",
"result": {
"columns": [
{
"name": "SUM(T1.wins)",
"declared_type": "INTEGER"
},
{
"name": "forename",
"declared_type": "TEXT"
},
{
"name": "surname",
"declared_type": "TEXT"
}
],
"row_count": 1,
"truncated": false,
"rows_shown": 1,
"rows": [
[
{
"type": "int",
"value": 7890
},
{
"type": "str",
"value": "Volker"
},
{
"type": "str",
"value": "Weidler"
}
]
],
"result_hash": "sha256:48b715a576c7daabcc6b152a00c646773a078aee93f3589e7050878d7b0972cb"
}
},
"plan_variant": {
"run": false,
"reason": "the plan variant was not asked for"
}
}SELECT SUM(T1.wins),T2.forename, T2.surname FROM driverStandings AS T1 INNER JOIN drivers AS T2 on T1.driverId = T2.driverId ORDER BY T2.dob ASC LIMIT 1result_hash sha256:95586ad9cbebd6bd17ab272e5754212cef9669f964611847cfb7439ea72fc8eb recomputed from this JSON: match
record_hash sha256:3a185a8d32b5fc309eded73f5dbda4f476460bf54a58bb0a9319167113ac39df recomputed from this JSON: match
from evidence-gold.json, 1 row
| SUM(T1.wins)INTEGER | forenameTEXT | surnameTEXT |
|---|---|---|
| 7890 | Lewis | Hamilton |
re-run this statement read-only against SQLite 3.53.4 | file=/private/tmp/attestql-runs/data/dev/dev_databases/formula_1/formula_1.sqlite | size=22360064 under the session settings and over the data this record's fixture digest names, and compare the two results under R-ORD