distributaur 0.0.3

Simple task manager and job queue for distributed rendering. Built on celery and redis.

A super simple way to distribute rendering tasks across multiple machines.

Installation

pip install distributaur

Development

Prerequisites

• Python 3.8 or newer (tested on 3.10)
• Redis server
• Celery

Setup

Clone the repository and navigate to the project directory:

git clone https://github.com/RaccoonResearch/distributaur.git
cd distributaur

Install the required packages:

pip install -r requirements.txt

Install the distributaur package:

python setup.py install

Configuration

Create a .env file in the root directory of your project or set environment variables to match your setup:

REDIS_HOST=localhost
REDIS_PORT=6379
REDIS_USER=user
REDIS_PASSWORD=password
VAST_API_KEY=your_vast_api_key

Getting Started

Starting the Worker

To start processing tasks, you need to run a worker. You can start a worker using the provided script:

sh scripts/kill_redis_connections.sh  # Optional: to clear previous Redis connections
celery -A distributaur.core worker --loglevel=info

Running an Example Task

To run an example task and see Distributaur in action, you can execute the example script provided in the project:

python example.py

This script configures the environment, registers a sample function, dispatches a task, and monitors its execution.

API Reference

Core Functionality

• register_function(func): Decorator to register a function that can be called as a task.
• execute_function(func_name, args): Dispatch a registered function as a task with specified arguments.

Task Management

• update_function_status(task_id, status): Update the status of a task in Redis.
• monitor_job_status(job_id): Monitor the status of a job and output updates.

VAST.ai Integration

• rent_nodes(max_price, max_nodes, image, api_key): Rent nodes from VAST.ai based on specified criteria.
• terminate_nodes(nodes): Terminate rented nodes on VAST.ai.

Contributing

Contributions are welcome! For major changes, please open an issue first to discuss what you would like to change.

License

This project is licensed under the MIT License - see the LICENSE file for details.