duckcheck

Lightweight data quality checks powered by DuckDB — the anti–Great Expectations for teams who want pip install, one YAML file, and one command.

Status: v0.6 — CSV/Parquet/SQLite/Postgres sources, custom SQL with field substitution + expect, pattern regex, freshness, baselines, JUnit, and --format json.

60-second try

pip install duckcheck
duckcheck run examples/clean.yaml
# or with Docker:
docker compose run --rm run-example
# live Postgres ATTACH example:
docker compose run --rm run-postgres-example

Why this vs alternatives

Problem

Data teams need to assert column quality in CI, but Great Expectations is heavyweight and Soda Core funnels to cloud. Ad-hoc SQL checks have no reporting standard.

Key features (v0.6)

• YAML check definitions
• DuckDB scans CSV, Parquet, SQLite, and Postgres (ATTACH) — no separate DQ server
• Checks: not_null, unique, accepted_values, custom_sql, pattern, freshness, row_count, row_count_delta
• custom_sql ${column} / ${name} substitution; expect operators: 0, =N, >N, <N, >=N, <=N (default 0)
• --format json and --junit for CI dashboards
• ${ENV} in source URIs; --source-table for SQL ATTACH

Architecture

duckcheck run checks.yaml
    └── SuiteSpec (Pydantic)
            └── DuckDB in-process
                    └── source_data view from CSV/Parquet/SQLite/Postgres

Installation

pip install duckcheck
pip install -e ".[dev]"

Usage

duckcheck health
duckcheck run examples/clean.yaml
duckcheck run examples/checks.yaml   # fixture with known failures
duckcheck run examples/clean.yaml --junit /tmp/duckcheck.xml
duckcheck run examples/clean.yaml --format json
duckcheck baseline update examples/clean.yaml

Example checks.yaml:

name: sample-suite
source: examples/sample.csv
checks:
  - name: id_not_null
    type: not_null
    column: id
  - name: status_values
    type: accepted_values
    column: status
    values: [active, inactive]
  - name: three_active
    type: custom_sql
    sql: "SELECT * FROM source_data WHERE status = 'active'"
    expect: "=3"

Postgres (compose hostname postgres; from the host set DUCKCHECK_PG_DSN):

name: postgres-suite
source: postgresql://duckcheck:duckcheck@postgres:5432/duckcheck
source_table: orders

Docker

docker compose run --rm test
docker compose run --rm run-example
docker compose run --rm run-postgres-example

Running tests

pytest tests/ -v
# live Postgres ATTACH (after compose postgres is up):
# compose publishes Postgres on host port 5433
DUCKCHECK_PG_DSN=postgresql://duckcheck:duckcheck@localhost:5433/duckcheck \
  pytest tests/test_runner.py::test_postgres_attach_live -v

Roadmap

• Freshness + row_count + custom_sql + JUnit
• Row-count baseline delta store (duckcheck baseline update)
• custom_sql expect operators + --format json
• custom_sql ${column} / ${name} substitution + pattern checks
• Live Postgres ATTACH (compose example + optional DUCKCHECK_PG_DSN test)
• Live MySQL ATTACH integration tests
• Airflow/Dagster operators

License

MIT

Known limitations (v0.6)

• MySQL ATTACH is stubbed (INSTALL/LOAD) — no live DB in CI yet
• examples/checks.yaml is a failing fixture; examples/clean.yaml / examples/postgres.yaml are happy paths
• Checks still run against a source_data view