distributaur
A simple way to distribute rendering tasks across multiple machines.
Description
Distributaur distributes rendering using the Celery task queue. The queued tasks are then passed to workers using Redis as a message broker. Once the worker has completed the task, the result is uploaded to Huggingface.
Installation
pip install distributaur
Development
Setup
Clone the repository and navigate to the project directory:
git clone https://github.com/RaccoonResearch/distributaur.git
cd distributaur
Install the required packages:
pip install -r requirements.txt
Install the distributaur package:
python setup.py install
Configuration
Create a .env file in the root directory of your project or set environment variables to create your desired setup:
REDIS_HOST=redis_host
REDIS_PORT=redis_port
REDIS_USER=redis_user
REDIS_PASSWORD=redis_password
VAST_API_KEY=your_vastai_api_key
HF_TOKEN=your_huggingface_token
HF_REPO_ID=your_huggingface_repo
BROKER_POOL_LIMIT=broker_pool_limit
Getting Started
Running an Example Task
To run an example task and see Distributaur in action, you can execute the example script provided in the project:
# To run the example task locally using either a Docker container or a Celery worker
python -m distributaur.example.local
# To run the example task on vast.ai ("kitchen sink" example)
python -m distributaur.example.distributed
This script configures the environment, registers a sample function, dispatches a task, and monitors its execution.
Command Options
--max_priceis the max price (in $/hour) a node can be be rented for.--max_nodesis the max number of vast.ai nodes that can be rented.--docker_imageis the name of the docker image to load to the vast.ai node.--module_nameis the name of the celery worker--number_of_tasksis the number of example tasks that will be added to the queue and done by the workers.
Contributing
Contributions are welcome! For major changes, please open an issue first to discuss what you would like to change.
License
This project is licensed under the MIT License - see the LICENSE file for details.
Citation
@misc{distributaur,
author = {Raccoon Research},
title = {distributaur: a simple way to distribute rendering tasks across mulitiple machines},
year = {2024},
publisher = {GitHub},
howpublished = {\url{https://github.com/RaccoonResearch/distributaur}}
}
Contributors
1 maintainer
Metadata
1 maintainer
Release files for distributaur 0.0.28
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | |
|---|---|---|---|
| distributaur-0.0.28.tar.gz | 17.4 kB | Details |
Built distribution (wheel)
| File | Interpreter | ABI | Platform | Reset |
|---|---|---|---|---|
| distributaur-0.0.28-py3-none-any.whl | Python 3 | none | any | Details |
Total release size: 36.0 kB
Release files / distributaur-0.0.28.tar.gz
| Download URL | distributaur-0.0.28.tar.gz |
|---|---|
| Size | 17.4 kB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
2224afb4bd6f36c420c3126c888aad11c488d1e22dde7dd6d647719887bf5551
|
|
BLAKE2b-256 checksum How to use checksums |
b6d32c911b1d73191545d0ae5f1490a1bfcd1da77ca87d9dc7c4b9f231a772d2
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/5.1.1 CPython/3.9.19
|
Release files / distributaur-0.0.28-py3-none-any.whl
| Download URL | distributaur-0.0.28-py3-none-any.whl |
|---|---|
| Size | 18.6 kB |
| Tags | Python 3 |
|
SHA-256 checksum How to use checksums |
15149a0be1b85a7d1b764f668b7a4f37d056d7a7993f11aae513076e2e13baf5
|
|
BLAKE2b-256 checksum How to use checksums |
74547532ffeb915766a4f5e1e5df6a29d95884b81c3e0dee518555599e7fa0da
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/5.1.1 CPython/3.9.19
|
