sparkdq 0.11.2

A declarative PySpark framework for row- and aggregate-level data quality validation.

SparkDQ — Data Quality Validation for Apache Spark

SparkDQ is a lightweight data quality framework built specifically for PySpark. Define checks declaratively or in Python, run them directly inside your Spark pipelines, and catch data issues before they reach production.

No wrappers. No heavy dependencies. Just Python and Spark.

Quickstart Examples

Define checks as dictionaries that can be loaded from YAML/JSON files, stored in databases, or generated by APIs — perfect for CI/CD pipelines and data contracts.

from pyspark.sql import SparkSession

from sparkdq.engine import BatchDQEngine
from sparkdq.management import CheckSet

spark = SparkSession.builder.getOrCreate()

df = spark.createDataFrame(
    [
        {"id": 1, "name": "Alice"},
        {"id": 2, "name": None},
        {"id": 3, "name": "Bob"},
    ]
)

# Declarative configuration via dictionary
# Could be loaded from YAML, JSON, or any external system
check_definitions = [
    {"check-id": "my-null-check", "check": "null-check", "columns": ["name"]},
]
check_set = CheckSet()
check_set.add_checks_from_dicts(check_definitions)

result = BatchDQEngine(check_set).run_batch(df)
print(result.summary())
# Validation Summary (2024-01-01 00:00:00)
# Total records:   3
# Passed records:  2
# Failed records:  1
# Warnings:        0
# Pass rate:       67.00%

Prefer Python-native development? Define checks directly in Python for full type safety, IDE autocompletion, and compile-time validation:

from pyspark.sql import SparkSession

from sparkdq.checks import NullCheckConfig, RowCountMinCheckConfig
from sparkdq.core import Severity
from sparkdq.engine import BatchDQEngine
from sparkdq.management import CheckSet

spark = SparkSession.builder.getOrCreate()

df = spark.createDataFrame(
    [
        {"id": 1, "name": "Alice"},
        {"id": 2, "name": None},
        {"id": 3, "name": "Bob"},
    ]
)

check_set = (
    CheckSet()
    .add_check(
        NullCheckConfig(
            check_id="null-check",
            columns=["name"],
            severity=Severity.CRITICAL,
        )
    )
    .add_check(
        RowCountMinCheckConfig(
            check_id="min-row-count",
            min_count=1,
            severity=Severity.WARNING,
        )
    )
)

result = BatchDQEngine(check_set).run_batch(df)
print(result.summary())
# Validation Summary (2024-01-01 00:00:00)
# Total records:   3
# Passed records:  2
# Failed records:  1
# Warnings:        0
# Pass rate:       67.00%

SparkDQ ships with 30+ built-in checks across null validation, numeric ranges, string patterns, date boundaries, schema enforcement, uniqueness, and referential integrity.

🚀 See the official documentation to learn more.

Installation

For Local Development / Standalone Clusters

Install with PySpark included:

pip install sparkdq[spark]

For Databricks / Managed Platforms

Install without PySpark (runtime provided by platform):

pip install sparkdq

The framework supports Python 3.11+ and is fully tested with PySpark 3.5.x. SparkDQ will automatically check for PySpark availability on import and provide clear error messages if PySpark is missing in your environment.

Why SparkDQ?

• ✅ Extensible by design: Add custom checks via a simple plugin system — no changes to the core required

• ✅ Declarative or Pythonic: Load checks from YAML/JSON, or define them in Python with full type safety and IDE autocompletion

• ✅ Severity-aware: Distinguish between hard failures (CRITICAL) and soft constraints (WARNING) — react differently to each

• ✅ Row-level and aggregate: Validate individual records and entire datasets in a single pass

• ✅ Minimal footprint: Only Pydantic required — PySpark is provided by your platform

Typical Use Cases

• Data Ingestion: Validate schema, check for nulls, enforce value ranges, and detect format violations before bad data enters your platform

• Lakehouse Quality: Assert completeness, uniqueness, and referential integrity before writing to Delta, Iceberg, or Hudi tables

• ML & Analytics: Validate feature completeness, numeric boundaries, and row counts before model training or reporting

• Pipeline Assertions: Enforce data contracts between pipeline stages — fail fast on critical violations, log warnings for soft constraints

Let’s Build Better Data Together

⭐️ Found this useful? Give it a star and help spread the word!

📣 Questions, feedback, or ideas? Open an issue or discussion — we’d love to hear from you.

🤝 Want to contribute? Check out CONTRIBUTING.md to get started.