evaldata

Evaluate AI-generated SQL with pytest.

evaldata runs text-to-SQL evals in your existing test suite.

It checks semantic equivalence of SQL queries, diffs result sets in your warehouse, and uses an LLM judge for ambiguous cases.

Why evaldata

• Semantic equivalence. Parse both queries, normalize their ASTs, and compare canonical forms. It doesn't execute queries or call an LLM. When it can't confirm equivalence, it returns unknown.
• Execution in your warehouse. Run the query on DuckDB, Postgres, Databricks, Snowflake, or BigQuery and compare the results, accounting for row order, NULLs, float tolerance, and types.
• It's just pytest. Every eval is a test, run in your suite and your CI on every PR. No new runner, notebook, or dashboard.
• An LLM judge when you need one. For ambiguous questions, missing reference answers, or explanations to grade, use a grader model with explicit criteria.

Quickstart

uv add evaldata   # core, includes the DuckDB adapter

An eval is a pytest test: a case (a question and its expected answer), a solver (the system under test that writes the SQL), and a scorer (how the answer is judged).

Below, the AI's SQL has reordered predicates and different casing, but means the same thing as the reference query. observed_equivalence() confirms the match with AST normalization; no query runs.

from evaldata import CallableSolver, EvalCase, assert_eval, eval_case, observed_equivalence
from evaldata.platforms import duckdb_platform

platform = duckdb_platform(name="shop", path="shop.duckdb")


@eval_case(
    input="Name the US customers with an id above 1.",
    expected={"kind": "gold_query", "sql": "SELECT name FROM customers WHERE country = 'US' AND id > 1"},
    platform=platform,
)
def test_us_customers(case: EvalCase) -> None:
    solver = CallableSolver(lambda c: "select NAME from customers where id > 1 and country = 'US'")
    assert_eval(case, solver, scorers=[observed_equivalence()])

uv run pytest

 case               result   detail
 ──────────────────────────────────
 test_us_customers  PASS

 1 passed, 0 failed

The full runnable version is in examples/01_deterministic/test_showcase.py.

To test a real model instead of fixed SQL, swap the solver for PromptSolver(model="openai/gpt-4o-mini") (needs the evaldata[litellm] extra). To judge equivalence without a warehouse, swap the scorer for judged_equivalence(model).

More use cases

• Add execution-based SQL scoring to Pydantic Evals.
• Evaluate dbt projects against gold SQL.
• Evaluate dbt Semantic Layer queries against gold MetricFlow queries.
• Evaluate Snowflake Cortex Analyst against gold SQL.
• Evaluate BigQuery queries against a live project.
• Reproduce dbt's Semantic Layer benchmark locally on DuckDB.

Install

uv add evaldata                # core (includes the DuckDB adapter)
uv add "evaldata[postgres]"    # + Postgres adapter
uv add "evaldata[databricks]"  # + Databricks adapter
uv add "evaldata[snowflake]"   # + Snowflake adapter
uv add "evaldata[bigquery]"    # + BigQuery adapter
uv add "evaldata[cortex]"      # + Snowflake Cortex Analyst solver
uv add "evaldata[litellm]"     # + litellm, to call a model from PromptSolver
uv add "evaldata[pydantic-evals]"  # + Pydantic Evals integration

DuckDB, Postgres, Databricks, Snowflake, and BigQuery are the adapters available today.

Documentation

Full documentation: monospaceai.github.io/evaldata

• Getting started: write and run your first eval.
• Concepts: cases, solvers, scorers, and platforms.
• Scoring guides: semantic equivalence, LLM judge, composing scorers.
• Model guides: local Ollama, hosted model.
• Integration guides: Pydantic Evals, dbt project, dbt Semantic Layer.
• Platform guides: Databricks, Snowflake, BigQuery, Cortex Analyst.
• Reproduce dbt's Semantic Layer benchmark.
• Run a text-to-SQL benchmark: load a Spider/BIRD dataset and measure execution accuracy.
• API reference: the public API, generated from docstrings.

Examples

Runnable examples in examples/:

See examples/README.md for details.

Contributing

git clone https://github.com/monospaceai/evaldata.git
cd evaldata
uv sync                       # core + dev tooling
uv run pre-commit install
just check                    # lint + typecheck + local tests with coverage

Run just --list for other development commands.

Platform e2e tests

Adapter conformance for real platforms is marked e2e. CI provisions Postgres as a service container and runs the suite on every push, so the Postgres adapter is exercised against a real engine on every change.

Run it locally against Postgres with:

docker compose up -d                  # postgres:17 on localhost:5432
uv run --extra postgres pytest -m e2e # connection via POSTGRES_TEST_* env (defaults match compose)