duckcheck

Lightweight data quality checks powered by DuckDB — the anti–Great Expectations for teams who want pip install, one YAML file, and one command.

Status: v0.8 — CSV/Parquet/SQLite/Postgres/MySQL sources, custom SQL, baselines, JUnit/JSON, Airflow + Dagster operators.

60-second try

pip install duckcheck
duckcheck run examples/clean.yaml
# or with Docker:
docker compose run --rm run-example
# live Postgres / MySQL ATTACH examples:
docker compose run --rm run-postgres-example
docker compose run --rm run-mysql-example

Why this vs alternatives

Problem

Data teams need to assert column quality in CI, but Great Expectations is heavyweight and Soda Core funnels to cloud. Ad-hoc SQL checks have no reporting standard.

Key features (v0.8)

• YAML check definitions
• DuckDB scans CSV, Parquet, SQLite, Postgres, and MySQL (ATTACH) — no separate DQ server
• Checks: not_null, unique, accepted_values, custom_sql, pattern, freshness, row_count, row_count_delta
• custom_sql ${column} / ${name} substitution; expect operators: 0, =N, >N, <N, >=N, <=N (default 0)
• --format json and --junit for CI dashboards
• ${ENV} in source URIs; --source-table for SQL ATTACH
• Orchestration: DuckCheckOperator (Airflow) and duckcheck_op (Dagster) via optional extras

Architecture

duckcheck run checks.yaml
    └── SuiteSpec (Pydantic)
            └── DuckDB in-process
                    └── source_data view from CSV/Parquet/SQLite/Postgres/MySQL

Installation

pip install duckcheck
pip install -e ".[dev]"

Usage

duckcheck health
duckcheck run examples/clean.yaml
duckcheck run examples/checks.yaml   # fixture with known failures
duckcheck run examples/clean.yaml --junit /tmp/duckcheck.xml
duckcheck run examples/clean.yaml --format json
duckcheck baseline update examples/clean.yaml

Example checks.yaml:

name: sample-suite
source: examples/sample.csv
checks:
  - name: id_not_null
    type: not_null
    column: id
  - name: status_values
    type: accepted_values
    column: status
    values: [active, inactive]
  - name: three_active
    type: custom_sql
    sql: "SELECT * FROM source_data WHERE status = 'active'"
    expect: "=3"

Postgres (compose hostname postgres; from the host set DUCKCHECK_PG_DSN):

name: postgres-suite
source: postgresql://duckcheck:duckcheck@postgres:5432/duckcheck
source_table: orders

MySQL (compose hostname mysql; from the host set DUCKCHECK_MYSQL_DSN):

name: mysql-suite
source: mysql://duckcheck:duckcheck@mysql:3306/duckcheck
source_table: orders

Docker

docker compose run --rm test
docker compose run --rm run-example
docker compose run --rm run-postgres-example
docker compose run --rm run-mysql-example

Airflow / Dagster

pip install 'duckcheck[airflow]'   # DuckCheckOperator
pip install 'duckcheck[dagster]'   # duckcheck_op(...)

from duckcheck.airflow_op import DuckCheckOperator

DuckCheckOperator(
    task_id="validate_orders",
    suite_path="checks/orders.yaml",
    source="postgresql://...",  # optional override
    junit_output="/tmp/duckcheck.xml",
)

from duckcheck.dagster_op import duckcheck_op

validate = duckcheck_op(name="validate", suite_path="checks/orders.yaml")

See examples/airflow/ and examples/dagster/.

Running tests

pytest tests/ -v
# live Postgres ATTACH (after compose postgres is up):
# compose publishes Postgres on host port 5433
DUCKCHECK_PG_DSN=postgresql://duckcheck:duckcheck@localhost:5433/duckcheck \
  pytest tests/test_runner.py::test_postgres_attach_live -v
# live MySQL ATTACH (compose publishes MySQL on host port 3307):
DUCKCHECK_MYSQL_DSN=mysql://duckcheck:duckcheck@127.0.0.1:3307/duckcheck \
  pytest tests/test_runner.py::test_mysql_attach_live -v

Roadmap

• Freshness + row_count + custom_sql + JUnit
• Row-count baseline delta store (duckcheck baseline update)
• custom_sql expect operators + --format json
• custom_sql ${column} / ${name} substitution + pattern checks
• Live Postgres ATTACH (compose example + optional DUCKCHECK_PG_DSN test)
• Live MySQL ATTACH (compose example + optional DUCKCHECK_MYSQL_DSN test)
• Airflow DuckCheckOperator + Dagster duckcheck_op (optional extras)

License

MIT

Known limitations (v0.8)

• Airflow/Dagster extras pull large frameworks — install only when needed
• examples/checks.yaml is a failing fixture; examples/clean.yaml / examples/postgres.yaml / examples/mysql.yaml are happy paths
• Checks still run against a source_data view