evaldata 0.1.0

AI evals framework for data & analytics engineering teams.

evaldata

The evaluation framework for AI-generated SQL.

Evaluate AI-generated SQL in your warehouse and catch regressions when prompts, models, or code change.

Pytest-native. CI-friendly. Built for data teams.

Install

uv add evaldata                # core (includes the DuckDB adapter)
uv add "evaldata[postgres]"    # + Postgres adapter
uv add "evaldata[databricks]"  # + Databricks adapter
uv add "evaldata[litellm]"     # + litellm, to call a model as the AI under test

DuckDB, Postgres, and Databricks are the adapters available today. Snowflake and BigQuery are planned.

Examples

Runnable examples in examples/:

Example	Shows
Deterministic	Every expected-type and scorer with fixed SQL — no model or network
Local AI	A self-hosted Ollama model as the AI under test, via litellm
Hosted AI	Hosted-model plumbing, mocked so it runs without a live call or key

See examples/README.md for details.

Develop locally

git clone https://github.com/monospaceai/evaldata.git
cd evaldata
uv sync                       # core + dev tooling
uv run pre-commit install
just check                    # lint + typecheck + tests with coverage (runs everything)

just check runs lint, typecheck, and tests with coverage (held at 100%). See the justfile for the full set of commands.

Platform e2e tests

Adapter conformance for real platforms is marked e2e. CI provisions Postgres as a service container and runs the suite on every push, so the Postgres adapter is exercised against a real engine on every change.

Run it locally against Postgres with:

docker compose up -d                  # postgres:17 on localhost:5432
uv run --extra postgres pytest -m e2e # connection via POSTGRES_TEST_* env (defaults match compose)