DashDQ — Data Quality for Databricks

DashDQ is a Databricks-native data quality library. It provides an interactive notebook wizard and a Python API to run 60+ production-ready checks on Unity Catalog tables — built entirely on PySpark and the Databricks SDK. No Great Expectations or external DQ frameworks required.

Part of the Dashlibs suite for Databricks — built by Darshan Shah.

Features

• 60+ native checks — completeness, accuracy, integrity, consistency — pure PySpark, zero external DQ library
• Interactive wizard (configure()) — 3-tab notebook UI powered by ipywidgets
• GE-compatible naming — check names follow Great Expectations conventions
• DQX-inspired checks — freshness, email/URL/IPv4/UUID validation, custom SQL filter
• Cross-table checks — foreign key validation, referential integrity, row count comparison across tables
• Composite key checks — multi-column PK and uniqueness validation
• Custom SQL filter — write a SQL WHERE clause; rows matching it are marked FAILED
• Flexible output — Delta table, Databricks Volume (JSON/CSV), or DataFrame
• Summary output — 1 row per check × column with pass/fail counts, %, column coverage, and metadata headers

Install

# Inside a Databricks notebook
%pip install dash-dq
dbutils.library.restartPython()

# Locally (Python 3.9+)
pip install dash-dq

Quickstart

Option 1 — 2-cell wizard

# Cell 1: open the configuration wizard
import dashdq
config = dashdq.configure()

The wizard opens below the cell. Enter your table name → Load Columns → add checks → choose output → Save Config.

# Cell 2: run checks (after clicking Save Config above)
report = dashdq.run_checks(config)

Option 2 — all-in-one

import dashdq
dashdq.launch()

Option 3 — pure Python API (no UI)

import dashdq

config = {
    "source": {"table": "catalog.schema.dim_customer"},
    "metadata": {
        "data_owner":      "Jane Smith",
        "data_steward":    "John Doe",
        "business_domain": "Finance",
        "description":     "Customer master data quality run",
    },
    "checks": [
        # Completeness
        {"check_name": "expect_column_values_to_not_be_null",
         "column": "customer_id", "threshold_pct": 100.0, "params": {}},

        # Accuracy — range
        {"check_name": "expect_column_values_to_be_between",
         "column": "age", "threshold_pct": 99.0,
         "params": {"min_value": 18, "max_value": 120}},

        # Accuracy — format
        {"check_name": "expect_column_values_to_be_valid_email",
         "column": "email_address", "threshold_pct": 98.0, "params": {}},

        # Accuracy — freshness
        {"check_name": "expect_column_values_to_be_not_older_than_n_days",
         "column": "last_updated", "threshold_pct": 100.0,
         "params": {"n_days": 7}},

        # Accuracy — custom SQL filter
        {"check_name": "expect_column_values_to_pass_custom_sql_filter",
         "column": "annual_income_aed", "threshold_pct": 100.0,
         "params": {"sql_filter": "annual_income_aed < 0 OR annual_income_aed > 100000000"}},

        # Integrity — primary key
        {"check_name": "expect_primary_key_to_be_valid",
         "column": "_COMPOUND_", "threshold_pct": 100.0,
         "params": {"columns": ["customer_id"]}},

        # Integrity — foreign key
        {"check_name": "expect_column_values_to_exist_in_reference_table",
         "column": "branch_id", "threshold_pct": 100.0,
         "params": {
             "reference_table": "catalog.schema.dim_branch",
             "reference_column": "branch_id",
         }},

        # Consistency — table level
        {"check_name": "expect_table_row_count_to_be_between",
         "column": "_TABLE_LEVEL_", "threshold_pct": 100.0,
         "params": {"min_value": 10000, "max_value": 100000}},
    ],
    "output": {
        "type": "delta",
        "delta_table": "catalog.schema.dq_results",
    },
}

report = dashdq.run_checks(config)
report.display()
print(report.summary())
# {'total_checks': 8, 'passed': 8, 'failed': 0, 'pass_rate_pct': 100.0}

Screenshots

Note: Replace these placeholders with screenshots from your Databricks workspace.

Tab 1 — Source & Metadata

Enter a fully-qualified table name (e.g. ai_innovation_gold_dev.sdh.dim_customer), click Load Columns to pull the column list from Spark, and optionally fill in Data Owner, Data Steward, Business Domain, and Description fields.

Tab 2 — Checks Builder

Select a column, pick a check from the dropdown (grouped by DQ dimension), set a pass threshold %, fill in parameters (fields appear dynamically), and click ＋ Add Check. A live table shows all configured checks with coloured dimension badges.

Tab 3 — Output

Choose between DataFrame only, Delta Table, Volume — JSON, or Volume — CSV. The filename is auto-suggested as dq_{table}_{date}.

Results

One row per check × column combination, with a summary banner showing total pass/fail counts and pass rate %.

To add screenshots: run dashdq.configure() in your Databricks workspace, take screenshots of each tab, and save them to docs/screenshots/.

Output schema

Check catalog (60+)

Completeness (5)

Accuracy — value (13)

Accuracy — string & pattern (13)

Accuracy — date & time (7)

Accuracy — aggregate (11)

Integrity (10)

Consistency — table level (10)

DQ Dimensions

Contributing

PRs require review — direct pushes to main are blocked on all Dashlibs repos.

git clone https://github.com/dash-libs/dash-dq
cd dash-dq
pip install -e ".[dev]"
pytest tests/ -v

License

Apache 2.0 — see LICENSE.