isitright

isitright is an evidence-first tool for stress-testing text-to-SQL evaluations. Given reference and candidate SQL, the v0.1 engine will search for a valid database state on which their outputs differ and emit a minimized, replayable counterexample. A disagreement is evidence, not automatic attribution of fault.

Current status: Stage 4 review gate

The v0.1 candidate safely compares queries on the supplied SQLite database and runs a deterministic, schema-aware generated-state search. Search harvests referenced tables, columns, and literals and tries boundary, NULL, duplicate, join, filtering, and aggregation mutations in priority order.

Every generated state is checked against SQLite integrity and foreign-key rules, an optional JSON Schema constraints document, and an optional trusted validator. Finding no divergence is reported as AGREEMENT_ON_TESTED_STATES; it is not a claim of equivalence. Findings are minimized across generated and background rows, revalidated, and exported as deterministic counterexample.sql files. Suite mode, ranked reports, and opt-in --fail-on policies are active.

Quick start

from isitright import check

result = check(
    database="example.sqlite",
    reference_sql="SELECT province FROM tribute ORDER BY province",
    candidate_sql="SELECT DISTINCT province FROM tribute ORDER BY province",
)
print(result.headline)
print(result.original_state.outputs_match)
print(result.search.model_dump())

The CLI reads each query from a UTF-8 file and writes versioned JSON plus a concise Markdown rendering by default:

isitright check --database example.sqlite --reference gold.sql \
  --candidate agent.sql --output-dir artifacts/check

When a counterexample is found, replay the emitted fixture and verify both recorded outputs without touching the source database:

cd artifacts/check
python -m isitright.replay --report case_report.json --fixture counterexample.sql

Development

Python 3.11 or newer is required. With uv installed:

uv sync --extra dev
uv run ruff check .
uv run mypy src
uv run pytest
uv build

The package exposes single-case and JSONL suite commands:

isitright --version
isitright check --help
isitright suite --help

isitright suite --database example.sqlite --cases cases.jsonl \
  --output-dir artifacts/suite --fail-on domain-counterexample

Suite cases run independently and are ranked with domain-validated findings first, then schema-only findings, inconclusive/unsupported cases, and observed agreement. Both commands accept --fail-on none, domain-counterexample, or any-counterexample; a matched opt-in policy exits with code 3 after artifacts have been written.

The Stage 4 preview is versioned 0.1.0.dev5. It is intentionally a development release and must not be presented as the complete v0.1 search engine.

Contract highlights

• Result statuses keep observed equivalence, state validity, and attribution separate.
• Unordered query results are compared as multisets, preserving duplicates.
• Automatic ordering honors a top-level ORDER BY on either query; otherwise results are unordered.
• Engine-native values are normalized into explicit, portable canonical kinds.
• JSON Schema and trusted callback validation are applied before generated states are admitted, and every rejection is counted by reason.
• Reports use schema version 0.1; package versioning remains pre-release until the v0.1 engine is complete.

Generation is intentionally bounded and lightweight. It creates small insertions by adapting existing rows or synthesizing complete rows for empty referenced tables, including required foreign-key parent rows. It does not perform AST-diff-guided search or prove that exhausted candidates cover every admissible state. Minimization is also bounded; if its budget is exhausted, the smallest verified state found so far is returned with minimization_complete: false.

See the contract reference and the fixture proposal. The precise supported SQL surface and known limitations are documented in Supported SQL and limitations.

Contributions are welcome; see CONTRIBUTING.md. Security issues should follow the private reporting process in SECURITY.md.

License

MIT