spark-data-quality

Data Quality validation library for Kubernetes Spark pods. Runs Great Expectations and Trino SQL checks against any table, then persists results to the Data Quality Engine API.

Usage — By Table Name

When you only have the table's fully qualified name (catalog.schema.table), no table_id or test_suite_id is needed. The DQ Engine auto-resolves all IDs from Discovery Hub.

from spark_dq.quality import SparkDQAgent

agent = SparkDQAgent(
    catalog="my_catalog",
    schema="my_schema",
    table="my_table",
    data_quality_url="http://dq-engine:8000/api/v1/spark",
    trino_host="trino:443",
    trino_user="user",
    trino_pwd="pwd",
)

cfg = agent.fetch_table_config()
results = agent.execute_data_quality(df)

fetch_table_config()

Calls GET /config on the DQ Engine with catalog_name, schema_name, and table_name as query parameters. The DQ Engine then calls Discovery Hub's GET /ds/table/lookup/ to resolve the internal table_id, schema_id, and catalog_id. Once the table is identified, the DQ Engine fetches all active test suites and their expectations for that table and returns them as a config dict containing:

  • quality_query — the SQL query to load table data
  • table_type — TABLE or VIRTUAL_VIEW
  • catalog_type — managed or unmanaged
  • scan_limit — max rows to scan (if configured)
  • suites — list of test suites, each with its expectations
  • table_id, schema_id, catalog_id — resolved IDs

The config is cached after the first call, so subsequent calls return instantly.

execute_data_quality(df)

Takes a Spark DataFrame and runs all suites from the config against it.

  1. Reads the config (calls fetch_table_config() internally if not already cached)
  2. Backfills table_id from the config if it was not provided at init
  3. For each suite, splits expectations into two concurrent paths:
    • Trino fast-path — null checks, uniqueness, range, regex (single SQL query)
    • GE slow-path — all other expectation types (parallel Great Expectations validators)
  4. Saves results to the DQ Engine via POST /save-results
  5. Returns per-suite statistics (evaluated, passed, failed, success %)

Suites run in parallel (up to 8 concurrently). If Trino credentials are not provided, all expectations run through Great Expectations only.

License

MIT

Key dates

PyPI data
Data sourced directly from PyPI's database.
  • Released:
Newer release available (1.1.1)

1 maintainer

PyPI data
Data sourced directly from PyPI's database.
Avatar for khailasrangath from gravatar.com khailasrangath

Credits

Author: khailas

License expression

MIT
View SPDX License List

Requires

Python >=3.8

Provides Extra

spark dev

Classifiers

Report project as malware

Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

spark_data_quality-1.0.8.tar.gz (15.4 kB view details)

Uploaded Source

Built Distribution

If you're not sure about the file name format, learn more about wheel file names.

spark_data_quality-1.0.8-py3-none-any.whl (20.1 kB view details)

Uploaded Python 3

File details

Details for the file spark_data_quality-1.0.8.tar.gz.

File metadata

  • Download URL: spark_data_quality-1.0.8.tar.gz
  • Upload date:
  • Size: 15.4 kB
  • Tags: Source
  • Uploaded using Trusted Publishing? Yes
  • Uploaded via: twine/7.0.0 CPython/3.13.14

File hashes

Hashes for spark_data_quality-1.0.8.tar.gz
Algorithm Hash digest
SHA256 113dc1b233b51b0f7112629631f7e3a85b14a52bafe4415accc3b889096777af
MD5 eddf3e562ebd63f881d9ccb7493c93d9
BLAKE2b-256 02a20ed28d002043397dd108a5951846767d5cdf3d798e92999031a3af02f5db

See more details on using hashes here.

Provenance

The following attestation bundles were made for spark_data_quality-1.0.8.tar.gz:

Publisher: ci.yml on saal-core/digixt-quality-package

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

File details

Details for the file spark_data_quality-1.0.8-py3-none-any.whl.

File metadata

File hashes

Hashes for spark_data_quality-1.0.8-py3-none-any.whl
Algorithm Hash digest
SHA256 f1ffaaf337d172959e132eca97eb4c945438e3e1f31e1d18c141d7fef11ce5fe
MD5 9150a85cc4a19d254dfecce20d18f4a6
BLAKE2b-256 36d7e6d054265f550a5c376bc41e999a509f897bcde3f7c046cbcf2c55988aae

See more details on using hashes here.

Provenance

The following attestation bundles were made for spark_data_quality-1.0.8-py3-none-any.whl:

Publisher: ci.yml on saal-core/digixt-quality-package

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

Release history Release notifications | RSS feed

1.1.1

2 files

1.1.0

2 files

1.0.14

2 files

1.0.12

2 files

1.0.11

2 files

1.0.10

2 files

1.0.9

2 files

This release

1.0.8 This release

2 files

1.0.7

2 files

1.0.6

2 files

1.0.5

2 files

1.0.4

2 files

1.0.3

2 files

1.0.2

2 files

1.0.1

2 files

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page