data-ops

Coverage

data operations related code

motivation

data-ops is a library with tested and used code aligning on some standards regarding code structure and quality and to avoid reinventing the wheel. It builds on top of dataops-abs.

installation

    `pip install tgedr-dataops`

package namespaces and its contents

commons

S3Connector: base class to be extended, providing a connection session with aws s3 resources
utils_fs: utility module with file system related functions (example)

quality

PandasValidation : GreatExpectationsValidation implementation to validate pandas dataframes with Great Expectations library (example)

sink

LocalFsFileSink: Sink implementation class used to save/persist an object/file to a local fs location (example)
S3FileSink: Sink implementation class used to save/persist a local object/file to an s3 bucket (example)

source

AbstractS3FileSource: abstract Source class used to retrieve objects/files from s3 bucket to local fs location circumventing some formats download limitation
LocalFsFileSource: Source implementation class used to retrieve local objects/files to another local fs location (example)
PdDfS3Source: Source implementation class used to read a pandas dataframe from s3, whether a csv or an excel (xslx) file (example csv, example excel)
S3FileCopy: Source implementation class used to copy objects/files from an s3 bucket to another s3 bucket (example)
S3FileExtendedSource: Source implementation class used to retrieve objects/files from s3 bucket to local fs location with the extra method get_metadata providing sile metadata ("LastModified", "ContentLength", "ETag", "VersionId", "ContentType")(example)
S3FileSource: Source implementation class used to retrieve objects/files from s3 bucket to local fs location (example)

store

DataFrameSplits : dataclass container for train, test, and optional validation pandas DataFrames, providing helpers to build, merge, and convert splits to/from Hugging Face datasets
FsSinglePartitionParquetStore : abstract Store implementation defining persistence on parquet files with an optional single partition, regardless of the location it should persist
HuggingFaceDatasetStore : Store implementation class for interacting with Hugging Face Hub datasets, supporting CRUD operations on datasets and their splits (example)
LocalFsSinglePartitionParquetStore : FsSinglePartitionParquetStore implementation using local file system (example)
S3FsSinglePartitionParquetStore : FsSinglePartitionParquetStore implementation using aws s3 file system (example)
ParquetStore : Store implementation class for interacting with Parquet files using a filesystem interface (example)

known issues/further development

update data while changing its partition value (check unit test)

development

main requirements:
- uv
- bash
Clone the repository like this:
```
git clone git@github.com:tgedr/dataops
```
cd into the folder: cd dataops
install requirements: ./helper.sh reqs

Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

tgedr_dataops-1.1.1.tar.gz (20.5 kB view details)

Uploaded Aug 2, 2026 Source

Built Distribution

If you're not sure about the file name format, learn more about wheel file names.

The dropdown lists show the available interpreters, ABIs, and platforms. Enable javascript to be able to filter the list of wheel files.

tgedr_dataops-1.1.1-py3-none-any.whl (28.0 kB view details)

Uploaded Aug 2, 2026 Python 3

File details

Details for the file tgedr_dataops-1.1.1.tar.gz.

File metadata

Download URL: tgedr_dataops-1.1.1.tar.gz
Upload date: Aug 2, 2026
Size: 20.5 kB
Tags: Source
Uploaded using Trusted Publishing? No
Uploaded via: uv/0.12.1 {"installer":{"name":"uv","version":"0.12.1","subcommand":["publish"]},"python":null,"implementation":{"name":null,"version":null},"distro":{"name":"Ubuntu","version":"24.04","id":"noble","libc":null},"system":{"name":null,"release":null},"cpu":null,"openssl_version":null,"setuptools_version":null,"rustc_version":null,"ci":true}

File hashes

Hashes for tgedr_dataops-1.1.1.tar.gz
Algorithm	Hash digest
SHA256	`c75b9154762de4965815aff8331e51a07afce57e874d8712e8386bab45f71bec`
MD5	`1507a2c2b9238e03ee6328d090162769`
BLAKE2b-256	`0e8bfebad5fbd188d4ca8ab3cf492bdf772e836b224272aa80b94dde9b62c009`

See more details on using hashes here.

File details

Details for the file tgedr_dataops-1.1.1-py3-none-any.whl.

File metadata

Download URL: tgedr_dataops-1.1.1-py3-none-any.whl
Upload date: Aug 2, 2026
Size: 28.0 kB
Tags: Python 3
Uploaded using Trusted Publishing? No
Uploaded via: uv/0.12.1 {"installer":{"name":"uv","version":"0.12.1","subcommand":["publish"]},"python":null,"implementation":{"name":null,"version":null},"distro":{"name":"Ubuntu","version":"24.04","id":"noble","libc":null},"system":{"name":null,"release":null},"cpu":null,"openssl_version":null,"setuptools_version":null,"rustc_version":null,"ci":true}

File hashes

Hashes for tgedr_dataops-1.1.1-py3-none-any.whl
Algorithm	Hash digest
SHA256	`ecc458a8e43f8a9bf4c36664be6607ad838c67901781941139cbe499794dc173`
MD5	`35bd07c8de259cb00f43ef84733a4741`
BLAKE2b-256	`e09bd30d4faf15b66922424654f499b48de5f620062630d53dba07585f5d8f01`