Skip to main content

Queuing and cron system for Django

Project description

# dqueue

dqueue is an _experimental_ fork of
[django-lightweight-queue](https://github.com/thread/django-lightweight-queue),
targeting greater flexibility.

The rest of this document may be incorrect as we experiment with alternative
configurations.

## Basic Usage

```python
import time
from dqueue import task

# Define a task
@task()
def long_running_task(first_arg, second_arg):
time.sleep(first_arg * second_arg)

# Request that the task be executed at some point
long_running_task(4, second_arg=9)
```

See the docstring on the [`task`](dqueue/task.py) decorator
for more details.

## Configuration

All automatically picked up configuration options begin with `DQUEUE_` and can
be found in `app_settings.py`. They should be placed in the usual Django
settings files, for example:

```python
DQUEUE_BACKEND = 'dqueue.backends.redis.RedisBackend'
```

If desired, specific configuration overrides can be placed in a standalone
python file which passed on the command line.

For example, given a `special.py` containing:

```python
DQUEUE_REDIS_PORT = 12345
```

and then running:

```
$ python manage.py queue_runner --config=special.py
```

will result in the runner to use the settings from the specified configuration
file in preference to settings from the Django environment. Any settings not
present in the specified file are inherited from the global configuration.

### Backends

There are three built-in backends:

- Synchronous (the default): executes the task inline, without any actual queuing
- Redis: executes tasks at-most-once using [Redis][redis] for storage of the
enqueued tasks
- Reliable Redis: executes tasks at-least-once using [Redis][redis] for storage
of the enqueued tasks

[redis]: https://redis.io/

## Running Workers

The queue runner is implemented as a Django management command:

```
$ python manage.py queue_runner
```

Workers can be distributed over multiple hosts by telling each runner that it
is part of a pool:

```
$ python manage.py queue_runner --machine 2 --of 4
```

Alternatively a runner can be told explicitly which configuration to use:

```
$ python manage.py queue_runner --exact-configuration --config=special.py
```

When using `--exact-configuration` the number of workers is configured
exactly, rather than being treated as the configuration for a pool.
Additionally, exactly-configured runners will _not_ run any cron workers.

### Example

Given a Django configuration containing:

```python
DQUEUE_WORKERS = {
'queue1': 3,
}
```

and a `special.py` containing:

```python
DQUEUE_WORKERS = {
'queue1': 2,
}
```

Running any of:

```
$ python manage.py queue_runner --machine 1 --of 3 # or,
$ python manage.py queue_runner --machine 2 --of 3 # or,
$ python manage.py queue_runner --machine 3 --of 3
```

will result in one worker for `queue1` on the current machine, while:

```
$ python manage.py queue_runner --exact-configuration --config=special.py
```

will result in two workers on the current machine.

## Cron Tasks

DLQ supports the use of a cron-like specification of Django management
commands
to be run at certain times.

To specify that a management command should be run at a given time, place a
`cron.py` file in the root folder of the Django app which defines the command
and which contains a `CONFIG` variable:

```python
CONFIG = (
{
'command': 'my_cron_command',
# Day values 1-7 to match datetime.datetime.utcnow().isoweekday()
'days': '*',
'hours': '*',
'minutes': '*',
# Equivalent behaviour to the kwarg to `task` of the same name
'sigkill_on_stop': True,
},
)
```


Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Files for dqueue, version 0.0.2
Filename, size & hash File type Python version Upload date
dqueue-0.0.2-py3-none-any.whl (25.5 kB) View hashes Wheel py3
dqueue-0.0.2.tar.gz (18.4 kB) View hashes Source None

Supported by

Elastic Elastic Search Pingdom Pingdom Monitoring Google Google BigQuery Sentry Sentry Error logging AWS AWS Cloud computing DataDog DataDog Monitoring Fastly Fastly CDN SignalFx SignalFx Supporter DigiCert DigiCert EV certificate StatusPage StatusPage Status page