Skip to main content

A multiprocessing task queue for Django

Project description

A multiprocessing task queue for Django

image0

Features

  • Multiprocessing worker pool

  • Encrypted and compressed task packages

  • Scheduled tasks

  • Result hooks

  • Result and Failure database

  • PaaS compatible with multiple pools

  • Django Admin

Requirements

Tested with: Python 2.7, 3.4. Django 1.7.8, 1.8.2*

*Django Q is currently in Alpha and as such not safe for production, yet.

Installation

  • Install the latest version with pip: pip install django-q

  • Add django_q to INSTALLED_APPS in your settings.py:

    INSTALLED_APPS = (
        # other apps
        'django_q',
    )
  • Run python manage.py migrate to create the database tables

  • Make sure you have a Redis server running somewhere

Configuration

All configuration settings are optional. e.g:

# settings.py
Q_CLUSTER = {
    'name': 'myproject',
    'workers': 8,
    'recycle': 500,
    'compress': True,
    'save_limit': 250,
    'label': 'Django Q',
    'redis': {
        'host': '127.0.0.1',
        'port': 6379,
        'db': 0, }
}
  • name Used to differentiatie between projects using the same Redis server* default: ‘default’

  • workers The number of workers to use in the cluster default: CPU count of host

  • recycle The number of tasks a worker will process before respawing. Used to release resources. default: 500

  • compress Compress task packages to Redis. Useful for large payloads. default: False

  • save_limit Limits the amount of successful tasks saved to Django. Set to 0 for unlimited. Set to -1 for no success storage at all. Failures are always saved. default: 250

  • label The label used for the Django Admin page default: ‘Django Q’

  • redis Connection settings for Redis. Follows standard Redis-Py syntax. default: localhost

*Django Q uses your SECRET_KEY to encrypt task packages and prevent task crossover

Managment Commands

qcluster

Start a cluster with: python manage.py qcluster

qmonitor

Monitor your clusters with python manage.py qmonitor

Creating Tasks

Async

async(func,*args,hook=None,**kwargs)
from django_q import async, result

# create the task
async('math.copysign', 2, -2)

# or with import and storing the id
import math.copysign

task_id = async(copysign, 2, -2)

# get the result
task_result = result(task_id)

# result returns None if the task has not been executed yet
# so in most cases you will want to use a hook:

async('math.modf', 2.5, hook='hooks.print_result')

# hooks.py
def print_result(task):
    print(task.result)

Schedule

Schedules are regular Django models. You can manage them through the Admin page or in your code:

from django_q import Schedule
from django.utils import timezone

Schedule.create(func='math.copysign',
                hook='hooks.print_result',
                args='2,-2',
                schedule_type=Schedule.DAILY,
                next_run=timezone.now())

Todo

  • Write sphinx documentation

  • Better tests and coverage

  • Get out of Alpha

  • Less dependencies?

Project details


Release history Release notifications | RSS feed

Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

django-q-0.1.0.tar.gz (15.7 kB view hashes)

Uploaded Source

Supported by

AWS AWS Cloud computing and Security Sponsor Datadog Datadog Monitoring Fastly Fastly CDN Google Google Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Sentry Sentry Error logging StatusPage StatusPage Status page