Easy wrapper for parallelizing Python executions
Project description
distify
Wrapper araound Ray for easy distributed processing in Python.
Features
- Multiple backends: Ray, Multiprocessing, Multithreading, Sequential.
- Logging.
- Progress bar.
- Can run in local or in multiple nodes.
- Individual timeout for map applications.
- Resume from checkpointing.
- Hydra integration.
Quickstart/usage
Install:
pip install distify
and start from this example: https://github.com/jordiae/distify/tree/main/examples/basic
To run it:
python app.py
To resume the execution (if interrupted):
python app.py hydra.run.dir=outputs/2021-09-14/08-54-10
(where hydra.run.dir is the output of the execution to be resumed)
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
distify-0.1.tar.gz
(6.8 kB
view hashes)
Built Distribution
distify-0.1-py3-none-any.whl
(6.7 kB
view hashes)