isitright

isitright is an evidence-first tool for stress-testing text-to-SQL evaluations. Given reference and candidate SQL, the eventual v0.1 engine will search for a valid database state on which their outputs differ and emit a minimized, replayable counterexample. A disagreement is evidence, not automatic attribution of fault.

Current status: Stage 0 review gate

This repository currently contains package scaffolding and specification contracts only. SQL execution, database mutation, constraint validation, search, comparison, and minimization are deliberately not implemented. The proposed fixtures under tests/fixtures/proposals require owner approval before Stage 1 begins.

Development

Python 3.11 or newer is required. With uv installed:

uv sync --extra dev
uv run ruff check .
uv run mypy src
uv run pytest
uv build

The package exposes the future command surface now:

isitright --version
isitright check --help
isitright suite --help

Calling check or suite during Stage 0 exits with the internal/not-implemented code because no analysis engine exists yet.

The Stage 0 preview is versioned 0.1.0.dev0. It is intentionally a development release and must not be presented as an implemented analysis engine.

Contract highlights

• Result statuses keep observed equivalence, state validity, and attribution separate.
• Unordered query results will be compared as multisets, preserving duplicates.
• Automatic ordering will honor a top-level ORDER BY; otherwise results are unordered.
• SQLite values are represented with explicit storage classes.
• Reports use schema version 0.1; package versioning remains pre-release until an engine is implemented.

See the contract reference and the fixture proposal.

License

MIT