Skip to main content

Data Preparation Laboratory Library

Project description

Data Processing Library

This provides a python framework for developing transforms on data stored in files - currently parquet files are supported - and running them in a ray cluster. Data files may be stored in the local file system or COS/S3. For more details see the documentation.

Virtual Environment

The project uses pyproject.toml and a Makefile for operations. To do development you should establish the virtual environment

make venv

and then either activate

source venv/bin/activate

or set up your IDE to use the venv directory when developing in this project

Library Artifact Build and Publish

To test, build and publish the library to artifactory

make test build publish

To up the version number, edit the Makefile to change VERSION and rerun the above. This will require committing both the Makefile and the autotmatically updated pyproject.toml file.

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

data_prep_lab-0.1.0.tar.gz (228.3 kB view hashes)

Uploaded Source

Built Distribution

data_prep_lab-0.1.0-py3-none-any.whl (54.8 kB view hashes)

Uploaded Python 3

Supported by

AWS AWS Cloud computing and Security Sponsor Datadog Datadog Monitoring Fastly Fastly CDN Google Google Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Sentry Sentry Error logging StatusPage StatusPage Status page