spark-data-quality 1.0.1

SparkDQAgent — Data Quality validation package for K8s Spark pods

spark-data-quality

Data Quality validation library for Kubernetes Spark pods. Runs Great Expectations and persists results to a DQ Engine API.

Installation

pip install .            # core (GE + SQL)
pip install ".[spark]"   # includes PySpark
pip install -e ".[dev]"  # editable + dev tools

Usage

Single table

from spark_dq.quality import SparkDQAgent

agent = SparkDQAgent(
    catalog="my_catalog",
    schema="public",
    table="sales_data",
    data_quality_url="http://dq-engine:8000/api/v1/spark",
    catalog_type="unmanaged",
    table_id=13,
    test_suite_id=37,
    trino_host="sql-host:443",
    trino_user="user",
    trino_pwd="pwd",
)

cfg = agent.get_config()        # single GET /config call
results = agent.validate(df)    # hybrid SQL + GE execution

Suite run (auto-discover tables)

agent = SparkDQAgent(
    catalog="", schema="", table="",
    data_quality_url="http://dq-engine:8000/api/v1/spark",
    catalog_type="unmanaged",
    test_suite_id=37,
)
table_configs = agent.get_all_table_configs()  # returns list of per-table configs

Hybrid execution

When SQL engine credentials are provided, the agent splits expectations into two paths:

Path	Assertion types	How
SQL fast-path	not_be_null, be_unique, be_between, match_regex, row_count_to_be_between	Single batched SQL query
GE slow-path	All others	Parallel in-memory GE validators

Both paths run concurrently. Results are merged and saved via POST /save-results.

License

MIT