Skip to main content

An extension for django and huey that supports multi queue management

Project description

License: MIT Version

django-huey


This package is an extension of huey contrib djhuey package that allows users to manage multiple queues.

Compatible versions

Package Version
Django 5.0
Django 4.2
Django 3.2
huey 2.5
huey 2.4

Installation

Using pip package manager run:

pip install django-huey

Then, in your settings.py file add django_huey to the INSTALLED_APPS:

INSTALLED_APPS = [
	...
    'django_huey',
]

Configuration

In settings.py you must add the DJANGO_HUEY setting:

DJANGO_HUEY = {
    'default': 'first', #this name must match with any of the queues defined below.
    'queues': {
        'first': {#this name will be used in decorators below
            'huey_class': 'huey.RedisHuey',  
            'name': 'first_tasks',  
            'consumer': {
                'workers': 2,
                'worker_type': 'thread',
            },
        },
        'emails': {#this name will be used in decorators below
            'huey_class': 'huey.RedisHuey',  
            'name': 'emails_tasks',  
            'consumer': {
                'workers': 5,
                'worker_type': 'thread',
            },
        }
    }
}

Including queues from files

new in 1.1.0

You can also include a queue configuration from another file, located in one of your apps. Use django_huey.utils.include to do so:

In settings.py you may have:

DJANGO_HUEY = {
    'default': 'first', #this name must match with any of the queues defined below.
    'queues': {
        # Your current queues definitions
    }
}

# This is new
from django_huey.utils import include
DJANGO_HUEY["queues"].update(include("example_app.queues"))

And in your example_app.queues:

queues = {
    "test": {
        "huey_class": "huey.MemoryHuey",
        "results": True,
        "store_none": False,
        "immediate": False,
        "utc": True,
        "blocking": True,
        "consumer": {
            "workers": 1,
            "worker_type": "thread",
            "initial_delay": 0.1,
            "backoff": 1.15,
            "max_delay": 10.0,
            "scheduler_interval": 60,
            "periodic": True,
            "check_worker_health": True,
            "health_check_interval": 300,
        },
    },
}

Note: in your queues file, you should declare a variable called queues, so they can be included. If the variable doesn't exist, an AttributeError will be raised.

Usage

Now you will be able to run multiple queues using:

python manage.py djangohuey --queue first
python manage.py djangohuey --queue emails

Each queue must be run in a different terminal.

If you defined a default queue, you can just run:

python manage.py djangohuey

And the default queue will be used.

Configuring tasks

You can use usual huey decorators to register tasks, but they must be imported from django_huey as shown below:

from django_huey import db_task, task

@task() #Use the default queue 'first'
def some_func_that_uses_default_queue():
    # perform some db task
    pass

@db_task(queue='first')
def some_func():
    # perform some db task
    pass

@task(queue='emails')
def send_mails():
	# send some emails
    pass

All the args and kwargs defined in huey decorators should work in the same way, if not, let us know.

Importing a huey instance

Sometimes you'll need to import a huey instance in order to do some advanced configuration, for example, when using huey pipelines.

You can do that by using the get_queue function from django_huey:

from django_huey import get_queue

first_q = get_queue('first')

@first_q.task()
def some_func():
    pass

Integration with huey monitor

You can use django-huey with huey monitor.

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

django_huey-1.3.0.tar.gz (11.1 kB view details)

Uploaded Source

Built Distribution

django_huey-1.3.0-py3-none-any.whl (12.6 kB view details)

Uploaded Python 3

File details

Details for the file django_huey-1.3.0.tar.gz.

File metadata

  • Download URL: django_huey-1.3.0.tar.gz
  • Upload date:
  • Size: 11.1 kB
  • Tags: Source
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/6.1.0 CPython/3.10.12

File hashes

Hashes for django_huey-1.3.0.tar.gz
Algorithm Hash digest
SHA256 011102430e949fe5807df2c44ccbfd812671a77e5dda552907c07ded554ab92f
MD5 570abc6a26201e3f33e232513db7434a
BLAKE2b-256 f2a3ca67a0c6b469f72a73e8e9e41dce6b918e35404c72994b78df5a5cea0a89

See more details on using hashes here.

File details

Details for the file django_huey-1.3.0-py3-none-any.whl.

File metadata

  • Download URL: django_huey-1.3.0-py3-none-any.whl
  • Upload date:
  • Size: 12.6 kB
  • Tags: Python 3
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/6.1.0 CPython/3.10.12

File hashes

Hashes for django_huey-1.3.0-py3-none-any.whl
Algorithm Hash digest
SHA256 b2db87248638e8b242cfd3a80a708c8940a23c4b23738e1bc00d26b1760be996
MD5 2d8a0d3b30e466b123817c5d37ba58ca
BLAKE2b-256 ab2b7203e18a1b8c6086ca5ab5899105dc877484d974532da9346ca2c87bd964

See more details on using hashes here.

Supported by

AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Pingdom Monitoring Sentry Error logging StatusPage Status page