spark-data-quality

Data Quality validation library for Kubernetes Spark pods. Runs Great Expectations and persists results to a DQ Engine API.

Usage

Single table

from spark_dq.quality import SparkDQAgent

agent = SparkDQAgent(
    catalog="my_catalog",
    schema="public",
    table="sales_data",
    data_quality_url="http://dq-engine:8000/api/v1/spark",
    catalog_type="unmanaged",
    test_suite_id=37,
    trino_host="sql-host:443",
    trino_user="user",
    trino_pwd="pwd",
)

cfg = agent.get_config()        # single GET /config call
results = agent.validate(df)    # hybrid SQL + GE execution

Suite run (auto-discover tables)

agent = SparkDQAgent(
    catalog="", schema="", table="",
    data_quality_url="http://dq-engine:8000/api/v1/spark",
    catalog_type="unmanaged",
    test_suite_id=37,
)
table_configs = agent.get_all_table_configs()  # returns list of per-table configs

Hybrid execution

When SQL engine credentials are provided, the agent splits expectations into two paths:

Path	Assertion types	How
SQL fast-path	`not_be_null`, `be_unique`, `be_between`, `match_regex`, `row_count_to_be_between`	Single batched SQL query
GE slow-path	All others	Parallel in-memory GE validators

Both paths run concurrently. Results are merged and saved via POST /save-results.

License

MIT

Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

spark_data_quality-1.0.7.tar.gz (14.7 kB view details)

Uploaded Jul 28, 2026 Source

Built Distribution

If you're not sure about the file name format, learn more about wheel file names.

The dropdown lists show the available interpreters, ABIs, and platforms. Enable javascript to be able to filter the list of wheel files.

spark_data_quality-1.0.7-py3-none-any.whl (19.4 kB view details)

Uploaded Jul 28, 2026 Python 3

File details

Details for the file spark_data_quality-1.0.7.tar.gz.

File metadata

Download URL: spark_data_quality-1.0.7.tar.gz
Upload date: Jul 28, 2026
Size: 14.7 kB
Tags: Source
Uploaded using Trusted Publishing? Yes
Uploaded via: twine/6.1.0 CPython/3.13.14

File hashes

Hashes for spark_data_quality-1.0.7.tar.gz
Algorithm	Hash digest
SHA256	`09b9765d392f072a6b7525367c44a3ef5ae631a08a558e57cb5c3ce56fd5d1e3`
MD5	`491103448b38f71f50fbd1194088d594`
BLAKE2b-256	`8d69bed205284cd13f31e50a79b8aeeada0ad6799a17ec9390943aca4131f410`

See more details on using hashes here.

Provenance

The following attestation bundles were made for spark_data_quality-1.0.7.tar.gz:

Publisher: ci.yml on saal-core/digixt-quality-package

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

Statement:
- Statement type: https://in-toto.io/Statement/v1
- Predicate type: https://docs.pypi.org/attestations/publish/v1
- Subject name: spark_data_quality-1.0.7.tar.gz
- Subject digest: 09b9765d392f072a6b7525367c44a3ef5ae631a08a558e57cb5c3ce56fd5d1e3
- Sigstore transparency entry: 2272846906
- Sigstore integration time: Jul 28, 2026
Source repository:
- Permalink: saal-core/digixt-quality-package@333c45ad0a41f141ec40715e89ea063eebf7b084
- Branch / Tag: refs/tags/v1.0.7
- Owner: https://github.com/saal-core
- Access: private
Publication detail:
- Token Issuer: https://token.actions.githubusercontent.com
- Runner Environment: github-hosted
- Publication workflow: ci.yml@333c45ad0a41f141ec40715e89ea063eebf7b084
- Trigger Event: push