Skip to main content

A multiprocessing task queue for Django

Project description

Q logo

A multiprocessing task queue for Django

image0 image1 Documentation Status image2

Features

  • Multiprocessing worker pool

  • Asynchronous tasks

  • Scheduled and repeated tasks

  • Encrypted and compressed packages

  • Failure and success database

  • Result hooks

  • Django Admin integration

  • PaaS compatible with multiple instances

  • Multi cluster monitor

  • Redis

  • Python 2 and 3

Requirements

Tested with: Python 2.7 & 3.4. Django 1.7.9 & 1.8.3

Installation

  • Install the latest version with pip:

    $ pip install django-q
  • Add django_q to your INSTALLED_APPS in your projects settings.py:

    INSTALLED_APPS = (
        # other apps
        'django_q',
    )
  • Run Django migrations to create the database tables:

    $ python manage.py migrate
  • Make sure you have a Redis server running somewhere

Read the more complete documentation at http://django-q.readthedocs.org

Configuration

All configuration settings are optional. e.g:

# settings.py
Q_CLUSTER = {
    'name': 'myproject',
    'workers': 8,
    'recycle': 500,
    'timeout': 60,
    'compress': True,
    'save_limit': 250,
    'label': 'Django Q',
    'redis': {
        'host': '127.0.0.1',
        'port': 6379,
        'db': 0, }
}

For full configuration options, see the configuration documentation.

If you are using django-redis , you can configure Django Q to use its connection pool.

Management Commands

Start a cluster with:

$ python manage.py qcluster

Monitor your clusters with:

$ python manage.py qmonitor

Creating Tasks

Use async from your code to quickly offload tasks:

from django_q import async, result

# create the task
async('math.copysign', 2, -2)

# or with a reference
import math.copysign

task_id = async(copysign, 2, -2)

# get the result
task_result = result(task_id)

# result returns None if the task has not been executed yet
# so in most cases you will want to use a hook:

async('math.modf', 2.5, hook='hooks.print_result')

# hooks.py
def print_result(task):
    print(task.result)

Schedule

Schedules are regular Django models. You can manage them through the Admin page or directly from your code:

from django_q import Schedule, schedule

# Use the schedule function

schedule('math.copysign',
         2, -2,
         hook='hooks.print_result',
         schedule_type=Schedule.DAILY)

# Or create the object directly

Schedule.objects.create(func='math.copysign',
                        hook='hooks.print_result',
                        args='2,-2',
                        schedule_type=Schedule.DAILY
                        )

Testing

To run the tests you will need py.test and pytest-django

Todo

  • Write more sphinx documentation

  • Better tests and coverage

  • Get out of Alpha

  • Less dependencies?

Acknowledgements

Project details


Release history Release notifications | RSS feed

Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

django-q-0.3.3.tar.gz (18.0 kB view hashes)

Uploaded Source

Supported by

AWS AWS Cloud computing and Security Sponsor Datadog Datadog Monitoring Fastly Fastly CDN Google Google Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Sentry Sentry Error logging StatusPage StatusPage Status page