data-ops
data operations related code
motivation
data-ops is a library with tested and used code aligning on some standards regarding code structure and quality and to avoid reinventing the wheel. It builds on top of dataops-abs.
installation
`pip install tgedr-dataops`
package namespaces and its contents
commons
- S3Connector: base class to be extended, providing a connection session with aws s3 resources
- utils_fs: utility module with file system related functions (example)
quality
- PandasValidation : GreatExpectationsValidation implementation to validate pandas dataframes with Great Expectations library (example)
sink
- LocalFsFileSink: Sink implementation class used to save/persist an object/file to a local fs location (example)
- S3FileSink: Sink implementation class used to save/persist a local object/file to an s3 bucket (example)
source
- AbstractS3FileSource: abstract Source class used to retrieve objects/files from s3 bucket to local fs location circumventing some formats download limitation
- LocalFsFileSource: Source implementation class used to retrieve local objects/files to another local fs location (example)
- PdDfS3Source: Source implementation class used to read a pandas dataframe from s3, whether a csv or an excel (xslx) file (example csv, example excel)
- S3FileCopy: Source implementation class used to copy objects/files from an s3 bucket to another s3 bucket (example)
- S3FileExtendedSource: Source implementation class used to retrieve objects/files from s3 bucket to local fs location with the extra method
get_metadataproviding sile metadata ("LastModified", "ContentLength", "ETag", "VersionId", "ContentType")(example) - S3FileSource: Source implementation class used to retrieve objects/files from s3 bucket to local fs location (example)
store
- FsSinglePartitionParquetStore : abstract Store implementation defining persistence on parquet files with an optional single partition, regardless of the location it should persist
- LocalFsSinglePartitionParquetStore : FsSinglePartitionParquetStore implementation using local file system (example)
- S3FsSinglePartitionParquetStore : FsSinglePartitionParquetStore implementation using aws s3 file system (example)
- ParquetStore : Store implementation class for interacting with Parquet files using a filesystem interface (example)
known issues/further development
- update data while changing its partition value (check unit test)
development
-
main requirements:
- uv
- bash
-
Clone the repository like this:
git clone git@github.com:tgedr/dataops
-
cd into the folder:
cd dataops -
install requirements:
./helper.sh reqs
1 maintainer
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
tgedr_dataops-1.1.0.tar.gz
(20.1 kB
view details)
Built Distribution
Filter files by name, interpreter, ABI, and platform.
If you're not sure about the file name format, learn more about wheel file names.
Copy a direct link to the current filters
File details
Details for the file tgedr_dataops-1.1.0.tar.gz.
File metadata
- Download URL: tgedr_dataops-1.1.0.tar.gz
- Upload date:
- Size: 20.1 kB
- Tags: Source
- Uploaded using Trusted Publishing? No
- Uploaded via:
uv/0.12.1 {"installer":{"name":"uv","version":"0.12.1","subcommand":["publish"]},"python":null,"implementation":{"name":null,"version":null},"distro":{"name":"Ubuntu","version":"24.04","id":"noble","libc":null},"system":{"name":null,"release":null},"cpu":null,"openssl_version":null,"setuptools_version":null,"rustc_version":null,"ci":true}
File hashes
| Algorithm | Hash digest | |
|---|---|---|
| SHA256 |
0a9af1b879a6c9cad012f516a7712a3854c9380e7749dfb3d5b12a28005f43c5
|
|
| MD5 |
d8eda2a2fd931ff6b14dbc1fcca0892b
|
|
| BLAKE2b-256 |
59a8b5591b7d6c8350937acf3309e755eb6216f1d63b76546903cc722602bff3
|
File details
Details for the file tgedr_dataops-1.1.0-py3-none-any.whl.
File metadata
- Download URL: tgedr_dataops-1.1.0-py3-none-any.whl
- Upload date:
- Size: 27.8 kB
- Tags: Python 3
- Uploaded using Trusted Publishing? No
- Uploaded via:
uv/0.12.1 {"installer":{"name":"uv","version":"0.12.1","subcommand":["publish"]},"python":null,"implementation":{"name":null,"version":null},"distro":{"name":"Ubuntu","version":"24.04","id":"noble","libc":null},"system":{"name":null,"release":null},"cpu":null,"openssl_version":null,"setuptools_version":null,"rustc_version":null,"ci":true}
File hashes
| Algorithm | Hash digest | |
|---|---|---|
| SHA256 |
90d36b378a7883209fb8c729b024b72bf35551eb9043f1e62de393ff00da9d22
|
|
| MD5 |
5bb61abe4729d24e22ce1d1fa4238ab2
|
|
| BLAKE2b-256 |
56938bdc726dd7cbd0351a1eb081c54f3dcc938d3c4383be12f90bf4ed17bf3c
|
