duckcheck

Lightweight data quality checks powered by DuckDB — the anti–Great Expectations for teams who want pip install, one YAML file, and one command.

Status: v0.5 — CSV/Parquet/SQLite sources, custom SQL with field substitution + expect, pattern regex, freshness, baselines, JUnit, and --format json.

60-second try

docker compose run --rm run-example  # duckcheck run examples/clean.yaml
docker compose run --rm test         # pytest

Why this vs alternatives

Problem

Data teams need to assert column quality in CI, but Great Expectations is heavyweight and Soda Core funnels to cloud. Ad-hoc SQL checks have no reporting standard.

Key features (v0.5)

• YAML check definitions
• DuckDB scans CSV, Parquet, and SQLite locally — no server
• Checks: not_null, unique, accepted_values, custom_sql, pattern, freshness, row_count, row_count_delta
• custom_sql ${column} / ${name} substitution; expect operators: 0, =N, >N, <N, >=N, <=N (default 0)
• --format json and --junit for CI dashboards
• ${ENV} in source URIs; --source-table for SQL ATTACH

Architecture

duckcheck run checks.yaml
    └── SuiteSpec (Pydantic)
            └── DuckDB in-process
                    └── source_data view from CSV/Parquet

Installation

pip install duckcheck
pip install -e ".[dev]"

Usage

duckcheck health
duckcheck run examples/clean.yaml
duckcheck run examples/checks.yaml   # fixture with known failures
duckcheck run examples/clean.yaml --junit /tmp/duckcheck.xml
duckcheck run examples/clean.yaml --format json
duckcheck baseline update examples/clean.yaml

Example checks.yaml:

name: sample-suite
source: examples/sample.csv
checks:
  - name: id_not_null
    type: not_null
    column: id
  - name: status_values
    type: accepted_values
    column: status
    values: [active, inactive]
  - name: three_active
    type: custom_sql
    sql: "SELECT * FROM source_data WHERE status = 'active'"
    expect: "=3"

Docker

docker compose run --rm test
docker compose run --rm run-example

Running tests

pytest tests/ -v

Roadmap

• Freshness + row_count + custom_sql + JUnit
• Row-count baseline delta store (duckcheck baseline update)
• custom_sql expect operators + --format json
• custom_sql ${column} / ${name} substitution + pattern checks
• Live Postgres/MySQL ATTACH integration tests
• Airflow/Dagster operators

License

MIT

Known limitations (v0.5)

• Postgres/MySQL ATTACH is stubbed (INSTALL/LOAD) — no live DB in CI yet
• examples/checks.yaml is a failing fixture; examples/clean.yaml is the happy path
• Checks still run against a source_data view