Simple task manager and job queue for distributed rendering. Built on celery and redis.
Project description
distributaur
A simple way to distribute rendering tasks across multiple machines.
Description
Distributaur distributes rendering using the Celery task queue. The queued tasks are then passed to workers using Redis as a message broker. Once the worker has completed the task, the result is uploaded to Huggingface.
Installation
pip install distributaur
Development
Setup
Clone the repository and navigate to the project directory:
git clone https://github.com/RaccoonResearch/distributaur.git
cd distributaur
Install the required packages:
pip install -r requirements.txt
Install the distributaur package:
python setup.py install
Configuration
Create a .env
file in the root directory of your project or set environment variables to create your desired setup:
REDIS_HOST=redis_host
REDIS_PORT=redis_port
REDIS_USER=redis_user
REDIS_PASSWORD=redis_password
VAST_API_KEY=your_vastai_api_key
HF_TOKEN=your_huggingface_token
HF_REPO_ID=your_huggingface_repo
BROKER_POOL_LIMIT=broker_pool_limit
Getting Started
Running an Example Task
To run an example task and see Distributaur in action, you can execute the example script provided in the project:
# To run the example task locally using either a Docker container or a Celery worker
python -m distributaur.example.local
# To run the example task on vast.ai ("kitchen sink" example)
python -m distributaur.example.distributed
This script configures the environment, registers a sample function, dispatches a task, and monitors its execution.
Command Options
--max_price
is the max price (in $/hour) a node can be be rented for.--max_nodes
is the max number of vast.ai nodes that can be rented.--docker_image
is the name of the docker image to load to the vast.ai node.--module_name
is the name of the celery worker--number_of_tasks
is the number of example tasks that will be added to the queue and done by the workers.
Contributing
Contributions are welcome! For major changes, please open an issue first to discuss what you would like to change.
License
This project is licensed under the MIT License - see the LICENSE
file for details.
Citation
@misc{distributaur,
author = {Raccoon Research},
title = {distributaur: a simple way to distribute rendering tasks across mulitiple machines},
year = {2024},
publisher = {GitHub},
howpublished = {\url{https://github.com/RaccoonResearch/distributaur}}
}
Contributors
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
File details
Details for the file distributaur-0.0.28.tar.gz
.
File metadata
- Download URL: distributaur-0.0.28.tar.gz
- Upload date:
- Size: 17.4 kB
- Tags: Source
- Uploaded using Trusted Publishing? No
- Uploaded via: twine/5.1.1 CPython/3.9.19
File hashes
Algorithm | Hash digest | |
---|---|---|
SHA256 | 2224afb4bd6f36c420c3126c888aad11c488d1e22dde7dd6d647719887bf5551 |
|
MD5 | 7433ec143bd9676bf3d6ac7037a1491c |
|
BLAKE2b-256 | b6d32c911b1d73191545d0ae5f1490a1bfcd1da77ca87d9dc7c4b9f231a772d2 |
File details
Details for the file distributaur-0.0.28-py3-none-any.whl
.
File metadata
- Download URL: distributaur-0.0.28-py3-none-any.whl
- Upload date:
- Size: 18.6 kB
- Tags: Python 3
- Uploaded using Trusted Publishing? No
- Uploaded via: twine/5.1.1 CPython/3.9.19
File hashes
Algorithm | Hash digest | |
---|---|---|
SHA256 | 15149a0be1b85a7d1b764f668b7a4f37d056d7a7993f11aae513076e2e13baf5 |
|
MD5 | 7bba35b5c3711b324ffce6819f9f734c |
|
BLAKE2b-256 | 74547532ffeb915766a4f5e1e5df6a29d95884b81c3e0dee518555599e7fa0da |