data-ops
data operations related code
motivation
data-ops is a library with tested and used code aligning on some standards regarding code structure and quality and to avoid reinventing the wheel. It builds on top of dataops-abs.
installation
`pip install tgedr-dataops`
package namespaces and its contents
commons
- S3Connector: base class to be extended, providing a connection session with aws s3 resources
- utils_fs: utility module with file system related functions (example)
quality
- PandasValidation : GreatExpectationsValidation implementation to validate pandas dataframes with Great Expectations library (example)
sink
- LocalFsFileSink: Sink implementation class used to save/persist an object/file to a local fs location (example)
- S3FileSink: Sink implementation class used to save/persist a local object/file to an s3 bucket (example)
source
- AbstractS3FileSource: abstract Source class used to retrieve objects/files from s3 bucket to local fs location circumventing some formats download limitation
- LocalFsFileSource: Source implementation class used to retrieve local objects/files to another local fs location (example)
- PdDfS3Source: Source implementation class used to read a pandas dataframe from s3, whether a csv or an excel (xslx) file (example csv, example excel)
- S3FileCopy: Source implementation class used to copy objects/files from an s3 bucket to another s3 bucket (example)
- S3FileExtendedSource: Source implementation class used to retrieve objects/files from s3 bucket to local fs location with the extra method
get_metadataproviding sile metadata ("LastModified", "ContentLength", "ETag", "VersionId", "ContentType")(example) - S3FileSource: Source implementation class used to retrieve objects/files from s3 bucket to local fs location (example)
store
- DataFrameSplits : dataclass container for
train,test, and optionalvalidationpandas DataFrames, providing helpers to build, merge, and convert splits to/from Hugging Face datasets - FsSinglePartitionParquetStore : abstract Store implementation defining persistence on parquet files with an optional single partition, regardless of the location it should persist
- HuggingFaceDatasetStore : Store implementation class for interacting with Hugging Face Hub datasets, supporting CRUD operations on datasets and their splits (example)
- LocalFsSinglePartitionParquetStore : FsSinglePartitionParquetStore implementation using local file system (example)
- S3FsSinglePartitionParquetStore : FsSinglePartitionParquetStore implementation using aws s3 file system (example)
- ParquetStore : Store implementation class for interacting with Parquet files using a filesystem interface (example)
known issues/further development
- update data while changing its partition value (check unit test)
development
-
main requirements:
- uv
- bash
-
Clone the repository like this:
git clone git@github.com:tgedr/dataops
-
cd into the folder:
cd dataops -
install requirements:
./helper.sh reqs
1 maintainer
Release files for tgedr-dataops 1.2.1
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | |
|---|---|---|---|
| tgedr_dataops-1.2.1.tar.gz | 20.2 kB | Details |
Built distribution (wheel)
| File | Interpreter | ABI | Platform | Reset |
|---|---|---|---|---|
| tgedr_dataops-1.2.1-py3-none-any.whl | Python 3 | none | any | Details |
Total release size: 52.5 kB
Release files / tgedr_dataops-1.2.1.tar.gz
| Download URL | tgedr_dataops-1.2.1.tar.gz |
|---|---|
| Size | 20.2 kB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
c0ff8f99c8946a8bde7bfb8a8893616c7d48d9833074e1a4a0f609a6f26f9b6e
|
|
BLAKE2b-256 checksum How to use checksums |
6954ee41a038541e039a7b422197dfe358515665c378f9e6a0717a7f170fffbe
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
uv/0.12.13 {"installer":{"name":"uv","version":"0.12.13","subcommand":["publish"]},"python":null,"implementation":{"name":null,"version":null},"distro":{"name":"Ubuntu","version":"24.04","id":"noble","libc":null},"system":{"name":null,"release":null},"cpu":null,"openssl_version":null,"setuptools_version":null,"rustc_version":null,"ci":true}
|
Release files / tgedr_dataops-1.2.1-py3-none-any.whl
| Download URL | tgedr_dataops-1.2.1-py3-none-any.whl |
|---|---|
| Size | 32.3 kB |
| Tags | Python 3 |
|
SHA-256 checksum How to use checksums |
4af15a25dd01698310b0f97012855d9d47f19f4c97cd505b4be3773b7179ad4f
|
|
BLAKE2b-256 checksum How to use checksums |
d43c527be9e2668f35d1218d9b02ee915919de7716f479dd31853c3cad7fbc45
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
uv/0.12.13 {"installer":{"name":"uv","version":"0.12.13","subcommand":["publish"]},"python":null,"implementation":{"name":null,"version":null},"distro":{"name":"Ubuntu","version":"24.04","id":"noble","libc":null},"system":{"name":null,"release":null},"cpu":null,"openssl_version":null,"setuptools_version":null,"rustc_version":null,"ci":true}
|
