tgedr-dataops-ext 0.0.4

data operations related code - extended

tgedr-dataops-ext

data operations related code - extended

motivation

dataops-ext is a library with tested and used code aligning on some standards regarding code structure and quality and to avoid reinventing the wheel. It builds on top of dataops-abs and dataops providing distributed processing features based on pyspark.

installation

    `pip install tgedr-dataops-ext`

package namespaces and its contents

commons

• Dataset: immutable class to wrap up a dataframe along with metadata (example)
• Metadata: immutable class depicting dataset metadata (example)
• UtilsSpark: utility class to work with spark, mostly helping on creating a session (example)

quality

• PysparkValidation : GreatExpectationsValidation implementation to validate pyspark dataframes with Great Expectations library (example)

source

• DeltaTableSource: abstract Source class used to read delta lake format datasets returning a pandas dataframe" (example)
• LocalDeltaTable: Source class used to read delta lake format datasets from local fs with python only, pyspark not needed, returning a pandas dataframe (example)
• S3DeltaTable: Source class used to read delta lake format datasets from s3 bucket with python only, pyspark not needed, returning a pandas dataframe (example)

store

• SparkDeltaStore : Store implementation for pyspark distributed processing with delta table format (example)

development

• main requirements:

  • uv
  • bash

• Clone the repository like this:

  git clone [e-mail]:tgedr/dataops-ext
  

• cd into the folder: cd dataops-ext

• install requirements: ./helper.sh reqs