Easy wrapper for parallelizing Python executions
Project description
distify
Wrapper around Ray for easy distributed processing in Python.
Disclaimer: I use it for some of my projects, but there is no guarantee that the code will work as expected or that future versions will be backward compatible.
Install
pip install distify --upgrade
Features
- Multiple backends: Ray, Multiprocessing, Multithreading, Sequential.
- Logging.
- Progress bar.
- Can run in local or in multiple nodes.
- Individual timeout for map applications.
- Resume from checkpointing.
- Hydra integration.
Quickstart/usage
Please refer to this example: https://github.com/jordiae/distify/tree/main/examples/basic
Documentation
TODO
LICENSE
MIT
Author
Jordi Armengol Estapé
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
distify-0.8.5.tar.gz
(5.2 kB
view details)
Built Distribution
File details
Details for the file distify-0.8.5.tar.gz.
File metadata
- Download URL: distify-0.8.5.tar.gz
- Upload date:
- Size: 5.2 kB
- Tags: Source
- Uploaded using Trusted Publishing? No
- Uploaded via: twine/3.4.2 importlib_metadata/4.8.1 pkginfo/1.7.1 requests/2.26.0 requests-toolbelt/0.9.1 tqdm/4.62.2 CPython/3.9.5
File hashes
| Algorithm | Hash digest | |
|---|---|---|
| SHA256 | e94795b993dd1e91653c634a1fc4dac9d78a241ad44f09643912eb8f843232de |
|
| MD5 | 9dec3169ef8866fe1b3cbcbb6f37fff8 |
|
| BLAKE2b-256 | a8a703cc26131d11d4ef0596dc628d96c808718372a67f75a94de077fe9c1223 |
File details
Details for the file distify-0.8.5-py3-none-any.whl.
File metadata
- Download URL: distify-0.8.5-py3-none-any.whl
- Upload date:
- Size: 6.0 kB
- Tags: Python 3
- Uploaded using Trusted Publishing? No
- Uploaded via: twine/3.4.2 importlib_metadata/4.8.1 pkginfo/1.7.1 requests/2.26.0 requests-toolbelt/0.9.1 tqdm/4.62.2 CPython/3.9.5
File hashes
| Algorithm | Hash digest | |
|---|---|---|
| SHA256 | 8b846b3b706f07f241caabe3ec0495f125ad921143651e327cdb80c7cb8ff2df |
|
| MD5 | 5009baf1ed1a492bd0db573681c9e1db |
|
| BLAKE2b-256 | 7d58bd285d9c7192b3f6e898131a3bd2212673c1f1d961baf74bfa97f1ee3d43 |
