Eliza DQ validates DataFrames and warehouse tables at any scale. It streams data through Polars LazyFrames (constant memory, no matter the dataset size), pushes checks to warehouses with parallel SQL queries, and collects failed row samples instantly via LIMIT N. No full materialization, no OOM, no waiting.

Two lines to your first check:

pip install eliza-dq

from eliza import check

result = check("data.parquet", checks={
    "order_id": ["not_null", "unique"],
    "amount":   ["not_null", "not_negative"],
    "email":    ["is_email"],
})
print(result.summary())
# 3 passed, 0 warnings, 2 failed (1,000,000 rows, 3ms)

What you get out of the box:

• 17 built-in checks with zero configuration (not_null, unique, regex, is_email, freshness, schema, FK reference, and more)
• 8 warehouse connectors with one-line YAML setup (BigQuery, Athena, Snowflake, Postgres, ClickHouse, MySQL, Databricks, Redshift)
• 3 interfaces to fit your workflow (inline dict for notebooks, YAML config for production, CLI for CI/CD)
• Slack alerts with PDF reports including donut charts, failure distribution bars, and sample tables
• Auto-learn checks from your data with eliza learn data.parquet
• 2 core dependencies (polars + pyyaml). No numpy. No pandas. No bloat.

Benchmarks

NYC Yellow Taxi dataset (Parquet). 5 identical checks. Warmup + 3 runs, min time. Apple M-series, Python 3.13.

DataFrame Engine (in-memory)

Pre-loaded Polars DataFrame - pure check speed, no I/O.

_{GX requires pandas - times include check execution only (pandas conversion adds 1-4s extra). Polars-native libraries (Eliza, Cuallee, Pandera) tested on Polars DataFrames directly.}

File Streaming (from disk)

Total wall time including file I/O - realistic workload (e.g. S3 to Lambda, local parquet files).

Eliza streams from disk via Polars LazyFrames. Data flows through in chunks without loading the full dataset into memory. Competitors must read all files into a single in-memory DataFrame before running checks. At 259M rows, that means 7.6+ GB of RAM just to hold the data.

SQL Pushdown Engine

AWS Athena, Iceberg tables, 8 not_null checks per table, pyathena connector.

_{* Soda Core OSS does not return failed row samples. DefaultSampler is a Cloud-only feature (paid). Eliza returns actual failed rows via SELECT ... WHERE ... LIMIT N.}

Eliza batches all inline checks into one SELECT and runs sample queries in parallel. Soda runs all queries sequentially. Eliza with 10 sample rows is faster than Soda without any samples on every scale tested.

Eliza vs Competitors

Features

_{* Materializes all failing rows in memory before truncating - causes OOM/timeout on large failures.
** Soda Slack alerting requires Soda Cloud (paid).}

Checks

Warehouse Support

pip install eliza-dq[bigquery]   # install only what you need
pip install eliza-dq[athena]
pip install eliza-dq[snowflake]
pip install eliza-dq[postgres]
pip install eliza-dq[clickhouse]
pip install eliza-dq[mysql]
pip install eliza-dq[databricks]
pip install eliza-dq[redshift]

DWH (SQL pushdown) - checks run as SQL queries directly on the warehouse:

# eliza_checks/analytics.yaml
connection:
  type: bigquery
  project: my-project-123

table: my-project-123.analytics.orders

checks:
  - column: order_id
    check: not_null

OLTP (local engine) - data is pulled from the database, checked locally with Polars:

# eliza_checks/prod_orders.yaml
connection:
  type: postgres
  host: prod-db.internal
  port: 5432
  user: readonly
  password: ${DB_PASSWORD}
  database: production

engine: local
table: orders
filter: "created_at >= '2024-01-01'"

checks:
  - column: order_id
    check: not_null

# BigQuery
connection:
  type: bigquery
  project: my-project
  location: US    # optional

# Athena
connection:
  type: athena
  region: us-east-1
  schema: my_database
  s3_staging_dir: s3://bucket/athena-results/

# Snowflake
connection:
  type: snowflake
  account: xy12345.us-east-1
  user: eliza_user
  password: ${SF_PASSWORD}
  warehouse: COMPUTE_WH
  database: ANALYTICS
  schema: PUBLIC

# PostgreSQL
connection:
  type: postgres
  host: localhost
  port: 5432
  user: postgres
  password: ${PG_PASSWORD}
  database: mydb

# MySQL
connection:
  type: mysql
  host: localhost
  user: root
  password: ${MYSQL_PASSWORD}
  database: mydb

# ClickHouse
connection:
  type: clickhouse
  host: localhost
  port: 8123
  user: default
  database: mydb

# Databricks
connection:
  type: databricks
  host: adb-123.azuredatabricks.net
  http_path: /sql/1.0/warehouses/abc
  token: ${DBX_TOKEN}

# Redshift
connection:
  type: redshift
  host: cluster.region.redshift.amazonaws.com
  database: analytics
  user: eliza_user
  password: ${RS_PASSWORD}

Quick Start

Inline checks (notebook / script)

import polars as pl
from eliza import check

df = pl.read_parquet("orders.parquet")

result = check(df, checks={
    "order_id": ["not_null", "unique"],
    "amount":   ["not_null", "not_negative", {"between": {"min": 0, "max": 100000}}],
    "email":    ["is_email"],
    "status":   [{"in_set": {"values": ["pending", "shipped", "delivered"]}}],
    "name":     [{"min_length": {"min": 2}}, {"max_length": {"max": 100}}],
})

print(result.summary())
result.raise_on_fail()  # exit code 1 on failure

YAML config (production)

# eliza_checks/orders.yaml
connection:
  type: bigquery
  project: my-project-123

table: my-project-123.analytics.orders

filter: "created_at >= '2024-01-01'"

samples:
  limit: 20

checks:
  - column: order_id
    check: not_null
  - column: order_id
    check: unique
  - column: amount
    check: not_negative
  - column: email
    check: not_missing
    missing_values: ["", "N/A", "null"]
  - column: updated_at
    check: freshness
    max_age: 24h
  - check: row_count
    min: 1000

from eliza import check
result = check(config="orders")

CLI

# Initialize project
eliza init

# Auto-learn checks from data
eliza learn data/orders.parquet --name orders

# Run checks (exit code: 0=pass, 1=fail, 2=error)
eliza check --config orders --source data/orders.parquet

# JSON output for orchestrators
eliza check --config orders --format json

Alerting & Reporting

from eliza import check
from eliza.alert import send_slack
from eliza.report import generate_pdf

result = check(config="orders")

# Slack message + PDF attachment
send_slack(result, token="xoxb-...", channel="C...", pdf=True, name="orders")

# PDF report with charts (donut, failure bars, sample tables)
generate_pdf(result, name="orders")
# -> eliza_orders_2026-09-07.pdf

pip install eliza-dq[report]  # for PDF reports

Slack Alert

PDF Report

CI/CD Integration

# .github/workflows/dq.yml
- run: pip install eliza-dq
- run: eliza check --config orders --source data/orders.parquet

# Airflow
@task
def dq_check():
    from eliza import check
    result = check(config="orders")
    result.raise_on_fail()
    return result.to_dict()

Architecture

Polars native (DataFrames, files): Streaming engine with per-column grouping. Files are scanned as LazyFrames - data streams through without loading into RAM. Each column group runs one collect(engine="streaming") call. Failed row samples use .filter().head(N).collect(engine="streaming") - instant, no full materialization.

SQL pushdown (warehouses): All inline checks batched into one SELECT COUNT(*), SUM(CASE WHEN ...) FROM table. Separate checks (unique, freshness) and sample queries run in parallel via ThreadPoolExecutor with thread-local connections. Sample queries use LIMIT N in SQL - never fetches all failing rows.

License

MIT