tgedr-dataops-ext 0.0.5

data operations related code - extended

tgedr-dataops-ext

Data operations library — extended.

motivation

tgedr-dataops-ext builds on top of tgedr-dataops (the abstract contracts layer) and provides concrete, tested implementations for distributed data processing with PySpark and Delta Lake. It covers session management, ETL pipelines, Delta table storage, data validation, and Databricks job integration, all following consistent code quality and structural standards.

installation

pip install tgedr-dataops-ext

package contents

commons

Class	Description	Example
Dataset	Immutable wrapper pairing a Spark DataFrame with its Metadata	test
Metadata	Immutable dataclass describing a dataset (name, version, framing, sources)	test
UtilsSpark	Utility class for creating and configuring Spark sessions (local, AWS Glue, or active session) and building PySpark schemas from type dictionaries	test
UtilsDatabricks	Utility class for retrieving the Databricks dbutils object from the active Spark session	test
EtlDatabricks	Abstract intermediate ETL class extending Etl with Databricks job integration: captures run_id, publishes outputs via dbutils.jobs.taskValues, and provides the inject_configuration decorator for auto-wiring method parameters from configuration or upstream task values	test

quality

Class	Description	Example
PysparkValidation	GreatExpectationsValidation implementation for validating PySpark DataFrames using the Great Expectations library	test

source

Class	Description	Example
DeltaTableSource	Abstract Source base class for reading Delta Lake datasets, returning a pandas DataFrame	test
LocalDeltaTable	Concrete Source reading Delta Lake datasets from the local filesystem using pure Python (no PySpark required)	test
S3DeltaTable	Concrete Source reading Delta Lake datasets from S3 using pure Python (no PySpark required)	test

store

Class	Description	Example
SparkDeltaStore	Store implementation for PySpark distributed processing with Delta Lake format. Supports versioned reads, append/overwrite writes, upserts, partitioning, schema evolution, retention policies, metadata management, and column comments	test

development

Requirements:

• uv
• bash

# clone
git clone [e-mail]:tgedr/dataops-ext
cd dataops-ext

# install dependencies
./helper.sh reqs

# run tests
./helper.sh test