spark-data-quality

Data Quality validation library for Kubernetes Spark pods. Runs Great Expectations and Trino SQL checks against any table, then persists results to a DQ Engine API.

Installation

pip install spark-data-quality

Usage

from spark_dq.quality import SparkDQAgent

agent = SparkDQAgent(
    catalog="my_catalog",
    schema="my_schema",
    table="my_table",
    data_quality_url="http://dq-engine:8000/api/v1/spark",
    trino_host="trino:443",
    trino_user="user",
    trino_pwd="pwd",
)

cfg = agent.fetch_table_config()
results = agent.execute_data_quality(df)

`fetch_table_config()`

Fetches the table's configuration from the DQ Engine, including the SQL query to load data, active test suites with their expectations, and scan limits. The config is cached after the first call.

`execute_data_quality(df)`

Runs all test suites against the provided Spark DataFrame. Expectations are split into two concurrent paths:

Trino fast-path — null checks, uniqueness, range, regex via a single SQL query
GE slow-path — all other expectation types via parallel Great Expectations validators

Suites run in parallel (up to 8 concurrently). Results are saved to the DQ Engine and per-suite statistics are returned.

If Trino credentials are not provided, all expectations run through Great Expectations only.

License

MIT

Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

spark_data_quality-1.0.11.tar.gz (15.0 kB view details)

Uploaded Jul 31, 2026 Source

Built Distribution

If you're not sure about the file name format, learn more about wheel file names.

The dropdown lists show the available interpreters, ABIs, and platforms. Enable javascript to be able to filter the list of wheel files.

spark_data_quality-1.0.11-py3-none-any.whl (19.7 kB view details)

Uploaded Jul 31, 2026 Python 3

File details

Details for the file spark_data_quality-1.0.11.tar.gz.

File metadata

Download URL: spark_data_quality-1.0.11.tar.gz
Upload date: Jul 31, 2026
Size: 15.0 kB
Tags: Source
Uploaded using Trusted Publishing? Yes
Uploaded via: twine/7.0.0 CPython/3.13.14

File hashes

Hashes for spark_data_quality-1.0.11.tar.gz
Algorithm	Hash digest
SHA256	`4e67fc9a6774b1889e83fbfe50ed4cfc9efa695ef888ade790feabad2debc77c`
MD5	`4d88d254637f3d73040c9595a7a2829f`
BLAKE2b-256	`d61a5d45fa4a40c5e6110fc8a9738cacdf5a4fe95c1b807ea2b91a0492af95ea`

See more details on using hashes here.

Provenance

The following attestation bundles were made for spark_data_quality-1.0.11.tar.gz:

Publisher: ci.yml on saal-core/digixt-quality-package

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

Statement:
- Statement type: https://in-toto.io/Statement/v1
- Predicate type: https://docs.pypi.org/attestations/publish/v1
- Subject name: spark_data_quality-1.0.11.tar.gz
- Subject digest: 4e67fc9a6774b1889e83fbfe50ed4cfc9efa695ef888ade790feabad2debc77c
- Sigstore transparency entry: 2300770117
- Sigstore integration time: Jul 31, 2026
Source repository:
- Permalink: saal-core/digixt-quality-package@4f37f36fb5491acb0f32e357934510004fc097c2
- Branch / Tag: refs/tags/v1.0.11
- Owner: https://github.com/saal-core
- Access: private
Publication detail:
- Token Issuer: https://token.actions.githubusercontent.com
- Runner Environment: github-hosted
- Publication workflow: ci.yml@4f37f36fb5491acb0f32e357934510004fc097c2
- Trigger Event: push