tgedr-dataops-abs

data operations related code - abstractions

motivation

abstract constructs to depict commonly use cases in data engineering context. Think about commonly accepted abstract classes and interfaces that can be implemented and extended according to different requirements and constraints.

installation

    `pip install tgedr-dataops-abs`

package namespaces and its contents

• Chain : chain-like abstract class (for sequential processing) (example)
• Etl : Extract-Transform-Load abstract class to be extended and used in data pipelines (example)
• Great_Expectations_Validation : data validation abstract class to be extended and to validate against json-defined expectations as consumend by the great expectations library (example)
• Processor : abstract class for data processing (example)
• Sink: abstract sink class defining methods (putand delete) to manage persistence of data somewhere as defined by implementing classes (example)
• Source: abstract source class defining methods (list and get) to manage retrieval of data from somewhere as defined by implementing classes (example)
• Store : abstract class used to manage persistence, defining CRUD-like (CreateReadUpdateDelete) methods (example)

development

• main requirements:

  • uv
  • bash

• Clone the repository like this:

  git clone git@github.com:tgedr/dataops-abs
  

• cd into the folder: cd dataops-abs

• install requirements: ./helper.sh reqs