Skip to main content

Simple task manager and job queue for distributed rendering. Built on celery and redis.

Project description

distributaur

A simple way to distribute rendering tasks across multiple machines.

Lint and Test PyPI version License forks - distributaur

Description

Distributaur distributes rendering using the Celery task queue. The queued tasks are then passed to workers using Redis as a message broker. Once the worker has completed the task, the result is uploaded to Huggingface.

Installation

pip install distributaur

Development

Setup

Clone the repository and navigate to the project directory:

git clone https://github.com/RaccoonResearch/distributaur.git
cd distributaur

Install the required packages:

pip install -r requirements.txt

Install the distributaur package:

python setup.py install

Configuration

Create a .env file in the root directory of your project or set environment variables to create your desired setup:

REDIS_HOST=redis_host
REDIS_PORT=redis_port
REDIS_USER=redis_user
REDIS_PASSWORD=redis_password
VAST_API_KEY=your_vastai_api_key
HF_TOKEN=your_huggingface_token
HF_REPO_ID=your_huggingface_repo
BROKER_POOL_LIMIT=broker_pool_limit

Getting Started

Running an Example Task

To run an example task and see Distributaur in action, you can execute the example script provided in the project:

# To run the example task locally using either a Docker container or a Celery worker
python -m distributaur.example.local

# To run the example task on vast.ai ("kitchen sink" example)
python -m distributaur.example.distributed

This script configures the environment, registers a sample function, dispatches a task, and monitors its execution.

Command Options

  • --max_price is the max price (in $/hour) a node can be be rented for.
  • --max_nodes is the max number of vast.ai nodes that can be rented.
  • --docker_image is the name of the docker image to load to the vast.ai node.
  • --module_name is the name of the celery worker
  • --number_of_tasks is the number of example tasks that will be added to the queue and done by the workers.

Contributing

Contributions are welcome! For major changes, please open an issue first to discuss what you would like to change.

License

This project is licensed under the MIT License - see the LICENSE file for details.

Citation

@misc{distributaur,
  author = {Raccoon Research},
  title = {distributaur: a simple way to distribute rendering tasks across mulitiple machines},
  year = {2024},
  publisher = {GitHub},
  howpublished = {\url{https://github.com/RaccoonResearch/distributaur}}
}

Contributors

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

distributaur-0.0.28.tar.gz (17.4 kB view details)

Uploaded Source

Built Distribution

distributaur-0.0.28-py3-none-any.whl (18.6 kB view details)

Uploaded Python 3

File details

Details for the file distributaur-0.0.28.tar.gz.

File metadata

  • Download URL: distributaur-0.0.28.tar.gz
  • Upload date:
  • Size: 17.4 kB
  • Tags: Source
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/5.1.1 CPython/3.9.19

File hashes

Hashes for distributaur-0.0.28.tar.gz
Algorithm Hash digest
SHA256 2224afb4bd6f36c420c3126c888aad11c488d1e22dde7dd6d647719887bf5551
MD5 7433ec143bd9676bf3d6ac7037a1491c
BLAKE2b-256 b6d32c911b1d73191545d0ae5f1490a1bfcd1da77ca87d9dc7c4b9f231a772d2

See more details on using hashes here.

File details

Details for the file distributaur-0.0.28-py3-none-any.whl.

File metadata

File hashes

Hashes for distributaur-0.0.28-py3-none-any.whl
Algorithm Hash digest
SHA256 15149a0be1b85a7d1b764f668b7a4f37d056d7a7993f11aae513076e2e13baf5
MD5 7bba35b5c3711b324ffce6819f9f734c
BLAKE2b-256 74547532ffeb915766a4f5e1e5df6a29d95884b81c3e0dee518555599e7fa0da

See more details on using hashes here.

Supported by

AWS AWS Cloud computing and Security Sponsor Datadog Datadog Monitoring Fastly Fastly CDN Google Google Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Sentry Sentry Error logging StatusPage StatusPage Status page