Eliza DQ validates DataFrames and warehouse tables with streaming execution, parallel SQL pushdown, and instant failure sampling. It runs on Polars LazyFrames (constant memory, no matter the data size), connects to 8 warehouses, and ships with 2 dependencies.

pip install eliza-dq

from eliza import check

result = check("data.parquet", checks={
    "order_id": ["not_null", "unique"],
    "amount":   ["not_null", "not_negative"],
    "email":    ["is_email"],
})
print(result.summary())
# 3 passed, 0 warnings, 2 failed (1,000,000 rows, 3ms)

Benchmarks

DataFrame Engine

In-memory Polars DataFrame, 5 checks, warmup + 3 runs, min time.

Eliza streams from disk via LazyFrames — constant memory regardless of file size. Competitors must load the entire dataset into RAM.

SQL Pushdown Engine

Athena, 179M rows (crossref_silver), 8 not_null checks.

Eliza batches all inline checks into one SELECT, runs separate checks and samples in parallel, and uses LIMIT N on sample queries. Soda runs queries sequentially and fetches ALL failing rows before truncating in memory.

Eliza vs Competitors

Features

_{* Materializes all failing rows in memory before truncating — causes OOM/timeout on large failures.
** Soda Slack alerting requires Soda Cloud ($25k+/yr).}

Checks

Warehouse Support

pip install eliza-dq[bigquery]   # install only what you need
pip install eliza-dq[athena]
pip install eliza-dq[snowflake]
pip install eliza-dq[postgres]
pip install eliza-dq[clickhouse]
pip install eliza-dq[mysql]
pip install eliza-dq[databricks]
pip install eliza-dq[redshift]

Quick Start

Inline checks (notebook / script)

import polars as pl
from eliza import check

df = pl.read_parquet("orders.parquet")

result = check(df, checks={
    "order_id": ["not_null", "unique"],
    "amount":   ["not_null", "not_negative", {"between": {"min": 0, "max": 100000}}],
    "email":    ["is_email"],
    "status":   [{"in_set": {"values": ["pending", "shipped", "delivered"]}}],
    "name":     [{"min_length": {"min": 2}}, {"max_length": {"max": 100}}],
})

print(result.summary())
result.raise_on_fail()  # exit code 1 on failure

YAML config (production)

# eliza_checks/orders.yaml
connection:
  type: bigquery
  project: my-project-123

table: my-project-123.analytics.orders

filter: "created_at >= '2024-01-01'"

samples:
  limit: 20

checks:
  - column: order_id
    check: not_null
  - column: order_id
    check: unique
  - column: amount
    check: not_negative
  - column: email
    check: not_missing
    missing_values: ["", "N/A", "null"]
  - column: updated_at
    check: freshness
    max_age: 24h
  - check: row_count
    min: 1000

from eliza import check
result = check(config="orders")

CLI

# Initialize project
eliza init

# Auto-learn checks from data
eliza learn data/orders.parquet --name orders

# Run checks (exit code: 0=pass, 1=fail, 2=error)
eliza check --config orders --source data/orders.parquet

# JSON output for orchestrators
eliza check --config orders --format json

Alerting & Reporting

from eliza import check
from eliza.alert import send_slack
from eliza.report import generate_pdf

result = check(config="orders")

# Slack message + PDF attachment
send_slack(result, token="xoxb-...", channel="C...", pdf=True, name="orders")

# PDF report with charts (donut, failure bars, sample tables)
generate_pdf(result, name="orders")
# -> eliza_orders_2026-09-07.pdf

pip install eliza-dq[report]  # for PDF reports

CI/CD Integration

# .github/workflows/dq.yml
- run: pip install eliza-dq
- run: eliza check --config orders --source data/orders.parquet

# Airflow
@task
def dq_check():
    from eliza import check
    result = check(config="orders")
    result.raise_on_fail()
    return result.to_dict()

Architecture

Polars native (DataFrames, files): Streaming engine with per-column grouping. Files are scanned as LazyFrames — data streams through without loading into RAM. Each column group runs one collect(engine="streaming") call. Failed row samples use .filter().head(N).collect(engine="streaming") — instant, no full materialization.

SQL pushdown (warehouses): All inline checks batched into one SELECT COUNT(*), SUM(CASE WHEN ...) FROM table. Separate checks (unique, freshness) and sample queries run in parallel via ThreadPoolExecutor with thread-local connections. Sample queries use LIMIT N in SQL — never fetches all failing rows.

License

MIT