R-ORD GOLD-ONLY not-a-function-of-the-data
card_games · dev_20251106-00000-of-00001 from https://huggingface.co/datasets/birdsql/bird_sql_dev_20251106/resolve/3c11fb193e5439b338e23677fa0aae11e8b85db9/data/dev_20251106-00000-of-00001.json (commit 3c11fb19, downloaded 2026-09-07)
What is the annual average number of sets that were released between 1/1/2012 to 12/31/2015? Indicate the common langugage of the card.
the hint the set supplies: AVG(id); releaseDate BETWEEN 1/1/2012 AND 12/31/2015; the common language refers to MAX(COUNT(language))
This question was audited without a prediction beside it, so there is nothing to compare the gold with. The probes below read the gold alone.
read by hand, 2026-09-07: wrong the gold does not answer its question on this data
the gold groups by release date and cuts at the largest language count, and several dates share that count; the projected average is a sum of ids divided by a count of ids divided by four, which is not an annual number of sets either
a maintainer's reading of this question, out of classification.json, copied from plans/reports/bird-dev-sqlite-260907/classification.json. It is not a verdict and nothing above it was computed from it.
SELECT (CAST(SUM(T1.id) AS REAL) / COUNT(T1.id)) / 4, T2.language FROM sets AS T1 INNER JOIN set_translations AS T2 ON T1.id = T2.id WHERE T1.releaseDate BETWEEN '2012-01-01' AND '2015-12-31' GROUP BY T1.releaseDate ORDER BY COUNT(T2.language) DESC LIMIT 1sha256:b5693f77f64f6a785dd2054ae28ad09604520931c209ed4100e61d9d52e85457
from evidence-gold.json, 1 row
| (CAST(SUM(T1.id) AS REAL) / COUNT(T1.id)) / 4REAL | languageTEXT |
|---|---|
| 66.95833333333333 | Japanese |
A smell is a mechanical reason to read this gold statement again. It is a heuristic: it does not state that the statement is wrong, and a maintainer decides.
this statement orders by a text column holding only numbers, and ordering it as a number gives a different answer, so the gold may be sorting 9.5 above 10
no ORDER BY key resolves to a text column
{
"heuristic": true,
"reason": "no ORDER BY key resolves to a text column",
"keys": [
{
"key": "COUNT(T2.language)",
"not_applicable": "the key is an expression and not a column"
}
]
}this statement cuts its result at a LIMIT that does not decide which rows come back, so a different but equally correct statement can return other rows and score zero
{
"heuristic": true,
"cut": 1,
"offset": 0,
"distinct_kept": false,
"unbounded_sql": "SELECT (CAST(SUM(T1.id) AS REAL) / COUNT(T1.id)) / 4, T2.language, COUNT(T2.language) AS attestql_ordering_key_0 FROM sets AS T1 INNER JOIN set_translations AS T2 ON T1.id = T2.id WHERE T1.releaseDate BETWEEN '2012-01-01' AND '2015-12-31' GROUP BY T1.releaseDate ORDER BY COUNT(T2.language) DESC",
"unbounded_rows": 68,
"projected_columns": [
"(CAST(SUM(T1.id) AS REAL) / COUNT(T1.id)) / 4",
"language"
],
"ordering_key_columns": [
"attestql_ordering_key_0"
],
"ordering_keys": [
{
"key": "COUNT(T2.language)",
"direction": "desc",
"nulls": "last",
"nulls_first_in_effect": false,
"returned_rows_null_in_this_key": 0,
"fires": false
}
],
"case": null
}rerun over the same rows in another physical order this statement gives another answer, so its result depends on how the rows are stored and not only on the data
from smells.json, 1 row
| 66.95833333333333 | Korean |
{
"heuristic": true,
"rule": "R-ORD",
"baseline_result_hash": "sha256:b5693f77f64f6a785dd2054ae28ad09604520931c209ed4100e61d9d52e85457",
"baseline_result": {
"columns": [
{
"name": "(CAST(SUM(T1.id) AS REAL) / COUNT(T1.id)) / 4",
"declared_type": "REAL"
},
{
"name": "language",
"declared_type": "TEXT"
}
],
"row_count": 1,
"truncated": false,
"rows_shown": 1,
"rows": [
[
{
"type": "dec",
"value": "66.95833333333333"
},
{
"type": "str",
"value": "Japanese"
}
]
],
"result_hash": "sha256:b5693f77f64f6a785dd2054ae28ad09604520931c209ed4100e61d9d52e85457"
},
"planner_statistics": {},
"shuffle": {
"seed": "1",
"row_limit": 300000,
"tables": [
"sets",
"set_translations"
],
"tables_not_shuffled": [],
"tables_skipped_for_size": {
"legalities": 427907
},
"tables_not_reached_by_a_copy": {}
},
"shuffled_copies": {
"run": true,
"verdict": "not_equal",
"differs": true,
"result_hash": "sha256:3247ce0a9c4792afd7064e498dd6c5511e78f2662752705016175d1c07d26904",
"result": {
"columns": [
{
"name": "(CAST(SUM(T1.id) AS REAL) / COUNT(T1.id)) / 4",
"declared_type": "REAL"
},
{
"name": "language",
"declared_type": "TEXT"
}
],
"row_count": 1,
"truncated": false,
"rows_shown": 1,
"rows": [
[
{
"type": "dec",
"value": "66.95833333333333"
},
{
"type": "str",
"value": "Korean"
}
]
],
"result_hash": "sha256:3247ce0a9c4792afd7064e498dd6c5511e78f2662752705016175d1c07d26904"
}
},
"plan_variant": {
"run": false,
"reason": "the plan variant was not asked for"
}
}SELECT (CAST(SUM(T1.id) AS REAL) / COUNT(T1.id)) / 4, T2.language FROM sets AS T1 INNER JOIN set_translations AS T2 ON T1.id = T2.id WHERE T1.releaseDate BETWEEN '2012-01-01' AND '2015-12-31' GROUP BY T1.releaseDate ORDER BY COUNT(T2.language) DESC LIMIT 1result_hash sha256:b5693f77f64f6a785dd2054ae28ad09604520931c209ed4100e61d9d52e85457 recomputed from this JSON: match
record_hash sha256:43a42ed51844aaa3365d332206b179a89320bece11aa53e582f6dbee4a1d9ca4 recomputed from this JSON: match
from evidence-gold.json, 1 row
| (CAST(SUM(T1.id) AS REAL) / COUNT(T1.id)) / 4REAL | languageTEXT |
|---|---|
| 66.95833333333333 | Japanese |
re-run this statement read-only against SQLite 3.53.4 | file=/private/tmp/attestql-runs/data/dev/dev_databases/card_games/card_games.sqlite | size=261820416 under the session settings and over the data this record's fixture digest names, and compare the two results under R-ORD