django-huey

This package is an extension of huey contrib djhuey package that allows users to manage multiple queues.

Compatible versions

Installation

Using pip package manager run:

pip install django-huey

Then, in your settings.py file add django_huey to the INSTALLED_APPS:

INSTALLED_APPS = [
	...
    'django_huey',
]

Configuration

In settings.py you must add the DJANGO_HUEY setting:

DJANGO_HUEY = {
    'default': 'first', #this name must match with any of the queues defined below.
    'queues': {
        'first': {#this name will be used in decorators below
            'huey_class': 'huey.RedisHuey',  
            'name': 'first_tasks',  
            'consumer': {
                'workers': 2,
                'worker_type': 'thread',
            },
        },
        'emails': {#this name will be used in decorators below
            'huey_class': 'huey.RedisHuey',  
            'name': 'emails_tasks',  
            'consumer': {
                'workers': 5,
                'worker_type': 'thread',
            },
        }
    }
}

Including queues from files

new in 1.1.0

You can also include a queue configuration from another file, located in one of your apps. Use django_huey.utils.include to do so:

In settings.py you may have:

DJANGO_HUEY = {
    'default': 'first', #this name must match with any of the queues defined below.
    'queues': {
        # Your current queues definitions
    }
}

# This is new
from django_huey.utils import include
DJANGO_HUEY["queues"].update(include("example_app.queues"))

And in your example_app.queues:

queues = {
    "test": {
        "huey_class": "huey.MemoryHuey",
        "results": True,
        "store_none": False,
        "immediate": False,
        "utc": True,
        "blocking": True,
        "consumer": {
            "workers": 1,
            "worker_type": "thread",
            "initial_delay": 0.1,
            "backoff": 1.15,
            "max_delay": 10.0,
            "scheduler_interval": 60,
            "periodic": True,
            "check_worker_health": True,
            "health_check_interval": 300,
        },
    },
}

Note: in your queues file, you should declare a variable called queues, so they can be included. If the variable doesn't exist, an AttributeError will be raised.

Usage

Now you will be able to run multiple queues using:

python manage.py djangohuey --queue first
python manage.py djangohuey --queue emails

Each queue must be run in a different terminal.

If you defined a default queue, you can just run:

python manage.py djangohuey

And the default queue will be used.

Configuring tasks

You can use usual huey decorators to register tasks, but they must be imported from django_huey as shown below:

from django_huey import db_task, task

@task() #Use the default queue 'first'
def some_func_that_uses_default_queue():
    # perform some db task
    pass

@db_task(queue='first')
def some_func():
    # perform some db task
    pass

@task(queue='emails')
def send_mails():
	# send some emails
    pass

All the args and kwargs defined in huey decorators should work in the same way, if not, let us know.

Importing a huey instance

Sometimes you'll need to import a huey instance in order to do some advanced configuration, for example, when using huey pipelines.

You can do that by using the get_queue function from django_huey:

from django_huey import get_queue

first_q = get_queue('first')

@first_q.task()
def some_func():
    pass

Integration with huey monitor

You can use django-huey with huey monitor.