The method

what is computed beside every verdict

What one question is put through

What one question is put through: two statements, parsed, run under one session's preconditions, serialized, hashed, and compared under the gold's rule Two columns, the gold statement on the left and the second statement on the right. Each is parsed, and a statement the parser cannot read stops there. Both are then run against one database in one session, under the preconditions that session states. Each result is serialized under one canonical rendering and hashed. The two hashes meet at a comparison under the rule the gold declared, R-SET or R-ORD, and the comparison answers EQUAL, NOT_EQUAL or NOT_COMPARABLE. NOT_COMPARABLE is the answer when a precondition differed, and NOT_EQUAL states that the two disagree on this data under this rule and not which of them is wrong. the gold statement as the question set ships it the second statement a prediction, or another gold parse tables, ordering, the rule parse refused here is run nowhere run, both in one session the preconditions are read off the server and recorded, not set serialize one canonical rendering serialize the same one, or nothing is comparable hash over every row, not a sample hash the record carries both compare, under the gold's rule R-SET, or R-ORD when the gold states an order EQUAL, NOT_EQUAL or NOT_COMPARABLE NOT_EQUAL states that these two disagree here, and nothing about which of them to keep

The two replay rules

A question declares one of these, and the comparison is performed under the one the gold declared. The two paragraphs below are the two comparisons' own text.

R-ORD
R-ORD: byte-identical canonical rendering under the recorded ordering and serialization, with each column named by its position so an alias is not part of the bytes. The verdict is NOT_COMPARABLE, naming what differed, when a precondition or a rule field differs; otherwise the renderings are compared byte for byte, in the order each result holds its rows.
R-SET
R-SET: typed semantic equality of the row multiset, order disregarded. The verdict is NOT_COMPARABLE, naming what differed, when a precondition or a rule field differs; otherwise the projections, by position and declared type, and the typed row multisets are compared.

NOT_EQUAL states that these two statements disagree on this data under this rule. It does not state which of them is wrong.

The rendering both results are compared under

One descriptor for the whole tool, restated in every record it writes: two results rendered under different rules are not two results a reader may compare.

version
attestql/audit/2
numeric scale
6
timestamp format
%Y-%m-%dT%H:%M:%S.%fZ
timezone
UTC
null rendering
NULL
encoding
utf-8

The 7 settings two postgresql records must agree on

A record states the session it was made in. Where two records disagree on one of these, the verdict is NOT_COMPARABLE and the page names what differed, because the two were not made under one rule. These 7 are named as postgresql names them, and they are that engine's.

On the other engine the tool has, the backend states this of itself: The nine settings a SQLite file can be asked for, and the seven it does not hold. Read once and repeated after that, as the identity is. None of the nine blocks a comparison: a SQLite record states no session setting that decides comparability, which is why the seven named fields are absent and everything read is recorded.

What makes two results unequal

The class states what makes these two results unequal under this rule, read off the two results and nothing else. It does not state which of the two statements is wrong.

The probes, and what each would mean

A smell is a mechanical reason to read this gold statement again. It is a heuristic: it does not state that the statement is wrong, and a maintainer decides.

arbitrary-cut
this statement cuts its result at a LIMIT that does not decide which rows come back, so a different but equally correct statement can return other rows and score zero
direction-against-question
the question asks for a maximum or a minimum and the statement orders the other way, so the bounded result may be the wrong end of the list; experimental
float-aggregate-order
this statement aggregates floating point numbers, so its last digits depend on the order the rows were summed in; the values agree to six significant digits
not-a-function-of-the-data
rerun over the same rows in another physical order this statement gives another answer, so its result depends on how the rows are stored and not only on the data
ordering-over-numeric-text
this statement orders by a text column holding only numbers, and ordering it as a number gives a different answer, so the gold may be sorting 9.5 above 10

The two benchmark readings computed beside every verdict

Neither decides anything here. They are what another evaluator's own check says about the same two results, so a reader can see where a benchmark's answer and this comparison part company.

BIRD's own check
on postgresql, set(second_rows) == set(gold_rows), float4/float8 cells as Python float as psycopg2 returns them, numeric as Decimal; on sqlite, set(second_rows) == set(gold_rows), REAL cells as Python float as sqlite3 returns them, so Python equality holds 1 == 1.0 == True as the benchmark's own scorer does. Read from https://github.com/bird-bench/mini_dev/blob/main/evaluation/evaluation_ex.py
the test-suite check
on postgresql, result_eq: equal row counts and equal column counts, each row unordered as a quick rejection, then the two equal as a list when the gold text holds ORDER BY and as a multiset otherwise, under some permutation of the columns; DISTINCT is not stripped and re-executed, and the cells are PostgreSQL's as psycopg2 returns them; on sqlite, result_eq: equal row counts and equal column counts, each row unordered as a quick rejection, then the two equal as a list when the gold text holds ORDER BY and as a multiset otherwise, under some permutation of the columns; DISTINCT is not stripped and re-executed, and the cells are SQLite's as sqlite3 returns them. Read from ruiqi-zhong/test-suite-sql-eval, exec_eval.py, result_eq, at commit 48cb78ec: https://github.com/ruiqi-zhong/test-suite-sql-eval/blob/48cb78ecf7f610620206283846c76751b18a1326/exec_eval.py

The principle

NOT_EQUAL never means the gold is wrong.

Every flag, every default and every state the run writes about itself are in the audit command's documentation. This site serves the pages and not the documentation, so that link leaves it.