evaldata

Test AI-generated SQL before it reaches production.

evaldata runs evals as ordinary pytest tests in your existing CI. It can prove SQL equivalence without executing queries, fall back to warehouse execution, or use an LLM judge for ambiguous cases.

Why evaldata

evaldata can often decide SQL equivalence without running the query or calling a grader. When structure is inconclusive, it falls back to warehouse execution or an LLM judge.

• Semantic equivalence. Confirm two queries have the same meaning by comparing their structure. No execution, no guessing — when it can't confirm, it returns unknown.
• Execution in your warehouse. Run the query on DuckDB, Postgres, Databricks, or Snowflake and compare the results, accounting for row order, NULLs, float tolerance, and types.
• It's just pytest. Every eval is a test, run in your suite and your CI on every PR. No new runner, notebook, or dashboard.
• An LLM judge when you need one. For ambiguous questions, missing reference answers, or explanations to grade, use a grader model with explicit criteria.

Beyond raw SQL, evaldata evaluates dbt projects and dbt Semantic Layer queries, and drives Snowflake Cortex Analyst as the AI under test.

evaldata reproduces dbt's own Semantic Layer benchmark locally on DuckDB — same dataset, questions, and model — scoring 96.4% with gpt-5.3-codex as pytest tests. See Reproduce dbt's Semantic Layer benchmark.

Quickstart

uv add evaldata   # core, includes the DuckDB adapter

An eval is a pytest test: a case (a question and its expected answer), a solver (the system under test that writes the SQL), and a scorer (how the answer is judged).

Below, the AI's SQL is written differently from the reference query — reordered predicates, different casing — but means the same thing. observed_equivalence() proves the match from the query structure alone; no query runs.

from evaldata import CallableSolver, EvalCase, assert_eval, eval_case, observed_equivalence
from evaldata.platforms import duckdb_platform

platform = duckdb_platform(name="shop", path="shop.duckdb")


@eval_case(
    input="Name the US customers with an id above 1.",
    expected={"kind": "gold_query", "sql": "SELECT name FROM customers WHERE country = 'US' AND id > 1"},
    platform=platform,
)
def test_us_customers(case: EvalCase) -> None:
    solver = CallableSolver(lambda c: "select NAME from customers where id > 1 and country = 'US'")
    assert_eval(case, solver, scorers=[observed_equivalence()])

uv run pytest

 case               result   detail
 ──────────────────────────────────
 test_us_customers  PASS

 1 passed, 0 failed

The full runnable version is in examples/01_deterministic/test_showcase.py.

To test a real model instead of fixed SQL, swap the solver for PromptSolver(model="openai/gpt-4o-mini") (needs the evaldata[litellm] extra). To judge equivalence without a warehouse, swap the scorer for judged_equivalence(model).

Install

uv add evaldata                # core (includes the DuckDB adapter)
uv add "evaldata[postgres]"    # + Postgres adapter
uv add "evaldata[databricks]"  # + Databricks adapter
uv add "evaldata[snowflake]"   # + Snowflake adapter
uv add "evaldata[cortex]"      # + Snowflake Cortex Analyst solver
uv add "evaldata[litellm]"     # + litellm, to call a model as the AI under test

DuckDB, Postgres, Databricks, and Snowflake are the adapters available today. A BigQuery adapter is planned.

Documentation

Full documentation: monospaceai.github.io/evaldata

• Getting started — write and run your first eval.
• Concepts — cases, solvers, scorers, and platforms.
• Scoring guides — semantic equivalence, LLM judge, composing scorers.
• Model guides — local Ollama, hosted model.
• Platform guides — Databricks, Snowflake, Cortex Analyst.
• dbt guides — dbt project, dbt Semantic Layer, reproduce dbt's Semantic Layer benchmark.
• Run a text-to-SQL benchmark — load a Spider/BIRD dataset and measure execution accuracy.
• API reference — the public API, generated from docstrings.

Examples

Runnable examples in examples/:

See examples/README.md for details.

Contributing

git clone https://github.com/monospaceai/evaldata.git
cd evaldata
uv sync                       # core + dev tooling
uv run pre-commit install
just check                    # lint + typecheck + tests with coverage (runs everything)

just check runs lint, typecheck, and tests with coverage (held at 100%). See the justfile for the full set of commands.

Platform e2e tests

Adapter conformance for real platforms is marked e2e. CI provisions Postgres as a service container and runs the suite on every push, so the Postgres adapter is exercised against a real engine on every change.

Run it locally against Postgres with:

docker compose up -d                  # postgres:17 on localhost:5432
uv run --extra postgres pytest -m e2e # connection via POSTGRES_TEST_* env (defaults match compose)