tgedr-dataops 1.0.5

data operations related code

data-ops

data operations related code

motivation

data-ops is a library with tested and used code aligning on some standards regarding code structure and quality and to avoid reinventing the wheel. It builds on top of dataops-abs.

installation

    `pip install tgedr-dataops`

package namespaces and its contents

commons

• S3Connector: base class to be extended, providing a connection session with aws s3 resources
• utils_fs: utility module with file system related functions (example)

quality

• PandasValidation : GreatExpectationsValidation implementation to validate pandas dataframes with Great Expectations library (example)

sink

• LocalFsFileSink: Sink implementation class used to save/persist an object/file to a local fs location (example)
• S3FileSink: Sink implementation class used to save/persist a local object/file to an s3 bucket (example)

source

• AbstractS3FileSource: abstract Source class used to retrieve objects/files from s3 bucket to local fs location circumventing some formats download limitation
• LocalFsFileSource: Source implementation class used to retrieve local objects/files to another local fs location (example)
• PdDfS3Source: Source implementation class used to read a pandas dataframe from s3, whether a csv or an excel (xslx) file (example csv, example excel)
• S3FileCopy: Source implementation class used to copy objects/files from an s3 bucket to another s3 bucket (example)
• S3FileExtendedSource: Source implementation class used to retrieve objects/files from s3 bucket to local fs location with the extra method get_metadata providing sile metadata ("LastModified", "ContentLength", "ETag", "VersionId", "ContentType")(example)
• S3FileSource: Source implementation class used to retrieve objects/files from s3 bucket to local fs location (example)

store

• FsSinglePartitionParquetStore : abstract Store implementation defining persistence on parquet files with an optional single partition, regardless of the location it should persist
• LocalFsSinglePartitionParquetStore : FsSinglePartitionParquetStore implementation using local file system (example)
• S3FsSinglePartitionParquetStore : FsSinglePartitionParquetStore implementation using aws s3 file system (example)
• ParquetStore : Store implementation class for interacting with Parquet files using a filesystem interface (example)

known issues/further development

• update data while changing its partition value (check unit test)

development

• main requirements:

  • uv
  • bash

• Clone the repository like this:

  git clone [e-mail]:tgedr/dataops
  

• cd into the folder: cd dataops

• install requirements: ./helper.sh reqs