Skip to main content

Queuing and cron system for Django

Project description

# dqueue

dqueue is an _experimental_ fork of
targeting greater flexibility.

The rest of this document may be incorrect as we experiment with alternative

## Basic Usage

import time
from dqueue import task

# Define a task
def long_running_task(first_arg, second_arg):
time.sleep(first_arg * second_arg)

# Request that the task be executed at some point
long_running_task(4, second_arg=9)

See the docstring on the [`task`](dqueue/ decorator
for more details.

## Configuration

All automatically picked up configuration options begin with `DQUEUE_` and can
be found in ``. They should be placed in the usual Django
settings files, for example:

DQUEUE_BACKEND = 'dqueue.backends.redis.RedisBackend'

If desired, specific configuration overrides can be placed in a standalone
python file which passed on the command line.

For example, given a `` containing:


and then running:

$ python queue_runner

will result in the runner to use the settings from the specified configuration
file in preference to settings from the Django environment. Any settings not
present in the specified file are inherited from the global configuration.

### Backends

There are three built-in backends:

- Synchronous (the default): executes the task inline, without any actual queuing
- Redis: executes tasks at-most-once using [Redis][redis] for storage of the
enqueued tasks
- Reliable Redis: executes tasks at-least-once using [Redis][redis] for storage
of the enqueued tasks


## Running Workers

The queue runner is implemented as a Django management command:

$ python queue_runner

Workers can be distributed over multiple hosts by telling each runner that it
is part of a pool:

$ python queue_runner --machine 2 --of 4

Alternatively a runner can be told explicitly which configuration to use:

$ python queue_runner --exact-configuration

When using `--exact-configuration` the number of workers is configured
exactly, rather than being treated as the configuration for a pool.
Additionally, exactly-configured runners will _not_ run any cron workers.

### Example

Given a Django configuration containing:

'queue1': 3,

and a `` containing:

'queue1': 2,

Running any of:

$ python queue_runner --machine 1 --of 3 # or,
$ python queue_runner --machine 2 --of 3 # or,
$ python queue_runner --machine 3 --of 3

will result in one worker for `queue1` on the current machine, while:

$ python queue_runner --exact-configuration

will result in two workers on the current machine.

## Cron Tasks

DLQ supports the use of a cron-like specification of Django management
to be run at certain times.

To specify that a management command should be run at a given time, place a
`` file in the root folder of the Django app which defines the command
and which contains a `CONFIG` variable:

'command': 'my_cron_command',
# Day values 1-7 to match datetime.datetime.utcnow().isoweekday()
'days': '*',
'hours': '*',
'minutes': '*',
# Equivalent behaviour to the kwarg to `task` of the same name
'sigkill_on_stop': True,

Project details

Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Files for dqueue, version 0.0.2
Filename, size File type Python version Upload date Hashes
Filename, size dqueue-0.0.2-py3-none-any.whl (25.5 kB) File type Wheel Python version py3 Upload date Hashes View
Filename, size dqueue-0.0.2.tar.gz (18.4 kB) File type Source Python version None Upload date Hashes View

Supported by

AWS AWS Cloud computing Datadog Datadog Monitoring DigiCert DigiCert EV certificate Facebook / Instagram Facebook / Instagram PSF Sponsor Fastly Fastly CDN Google Google Object Storage and Download Analytics Pingdom Pingdom Monitoring Salesforce Salesforce PSF Sponsor Sentry Sentry Error logging StatusPage StatusPage Status page