tgedr-dataops-ext
Concrete, tested implementations of the tgedr-dataops abstract contracts — PySpark, Delta Lake, and Databricks, all in one place.
motivation
tgedr-dataops-ext builds on top of tgedr-dataops (the abstract contracts layer) and provides concrete, tested implementations for distributed data processing with PySpark and Delta Lake. It covers session management, ETL pipelines, Delta table storage, data validation, and Databricks job integration, all following consistent code quality and structural standards.
installation
pip install tgedr-dataops-ext
package contents
commons
Shared utilities and base classes used across the library.
| Class | Description | Example |
|---|---|---|
Dataset |
Immutable wrapper pairing a Spark DataFrame with its Metadata |
test |
Metadata |
Immutable dataclass describing a dataset (name, version, framing, sources) | test |
UtilsSpark |
Utility class for creating and configuring Spark sessions (local, AWS Glue, or active session) and building PySpark schemas from type dictionaries | test |
UtilsDatabricks |
Utility class for retrieving the Databricks dbutils object from the active Spark session |
test |
EtlDatabricks |
Abstract intermediate ETL class extending Etl with Databricks job integration: captures run_id, publishes outputs via dbutils.jobs.taskValues, and provides the inject_configuration decorator for auto-wiring method parameters from configuration or upstream task values |
test |
quality
Data quality validation backed by Great Expectations.
| Class | Description | Example |
|---|---|---|
PysparkValidation |
GreatExpectationsValidation implementation for validating PySpark DataFrames using the Great Expectations library |
test |
source
Implementations of the Source contract for reading data from various backends.
| Class | Description | Example |
|---|---|---|
DeltaTableSource |
Abstract Source base class for reading Delta Lake datasets, returning a pandas DataFrame |
test |
LocalDeltaTable |
Concrete Source reading Delta Lake datasets from the local filesystem using pure Python (no PySpark required) |
test |
S3DeltaTable |
Concrete Source reading Delta Lake datasets from S3 using pure Python (no PySpark required) |
test |
CatalogFileSource |
Source implementation for listing, copying, and retrieving metadata of files in a Databricks-accessible file system (DBFS, S3, ADLS) via dbutils.fs |
test |
sink
Implementations of the Sink contract for writing and managing data in various backends.
| Class | Description | Example |
|---|---|---|
CatalogFileSink |
Sink implementation for copying and deleting files or directories in a Databricks-accessible file system via dbutils.fs |
test |
store
Implementations of the Store contract for persistent, structured data storage.
| Class | Description | Example |
|---|---|---|
SparkDeltaStore |
Store implementation for PySpark distributed processing with Delta Lake format. Supports versioned reads, append/overwrite writes, upserts, partitioning, schema evolution, retention policies, metadata management, and column comments |
test |
development
Requirements:
uvbash
# clone
git clone git@github.com:tgedr/dataops-ext
cd dataops-ext
# install dependencies
./helper.sh reqs
# run tests
./helper.sh test
1 maintainer
Metadata
1 maintainer
Release files for tgedr-dataops-ext 1.1.0
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | |
|---|---|---|---|
| tgedr_dataops_ext-1.1.0.tar.gz | 23.8 kB | Details |
Built distribution (wheel)
| File | Interpreter | ABI | Platform | Reset |
|---|---|---|---|---|
| tgedr_dataops_ext-1.1.0-py3-none-any.whl | Python 3 | none | any | Details |
Total release size: 52.0 kB
Release files / tgedr_dataops_ext-1.1.0.tar.gz
| Download URL | tgedr_dataops_ext-1.1.0.tar.gz |
|---|---|
| Size | 23.8 kB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
dad342f9fd15b3e280a1b6ab2cf483953f2b6fe5edc21bda18d94cc982028441
|
|
BLAKE2b-256 checksum How to use checksums |
547169b66af183c8b7a8678d63b33e207291af3f93045d5fc7659aa4d1461393
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
uv/0.12.13 {"installer":{"name":"uv","version":"0.12.13","subcommand":["publish"]},"python":null,"implementation":{"name":null,"version":null},"distro":{"name":"Ubuntu","version":"24.04","id":"noble","libc":null},"system":{"name":null,"release":null},"cpu":null,"openssl_version":null,"setuptools_version":null,"rustc_version":null,"ci":true}
|
Release files / tgedr_dataops_ext-1.1.0-py3-none-any.whl
| Download URL | tgedr_dataops_ext-1.1.0-py3-none-any.whl |
|---|---|
| Size | 28.2 kB |
| Tags | Python 3 |
|
SHA-256 checksum How to use checksums |
057e8144ff2d6f80dbce377a394d3b6e4495eba4b7f6be59aa4535d99c0b5fc0
|
|
BLAKE2b-256 checksum How to use checksums |
7f4c9eabf9a82d1a9bf7daaa73cf6b166c065468ce059b5ef096e216d38462fc
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
uv/0.12.13 {"installer":{"name":"uv","version":"0.12.13","subcommand":["publish"]},"python":null,"implementation":{"name":null,"version":null},"distro":{"name":"Ubuntu","version":"24.04","id":"noble","libc":null},"system":{"name":null,"release":null},"cpu":null,"openssl_version":null,"setuptools_version":null,"rustc_version":null,"ci":true}
|
