Skip to main content

Dead simple task queue using redis

Project description

https://travis-ci.org/baverman/dsq.svg?branch=master https://readthedocs.org/projects/dsq/badge/?version=latest

Dead simple task queue using redis.

# tasks.py
import dsq
manager = dsq.create_manager()

@manager.task(queue='normal')
def add(a, b):
    print a + b

if __name__ == '__main__':
    add.push(1, 2)
$ python tasks.py
$ dsq worker -bt tasks normal

See full DSQ documentation.

Features

  • Low latency.

  • Expiring tasks (TTL).

  • Delayed tasks (ETA).

  • Retries (forever or particular amount).

  • Periodic tasks.

  • Crontab.

  • Dead letters.

  • Queue priorities.

  • Worker lifetime.

  • Task execution timeout.

  • Task forwarder from one redis instance to another.

  • HTTP interface.

  • Inspect tools.

  • Supports 2.7, >3.4 and PyPy.

  • 100% test coverage.

The goal is a simple design. There is no worker manager, one can use supervisord/circus/whatever to spawn N workers. Simple storage model. Queue is a list and scheduled tasks are a sorted set. There are no task keys. Tasks are items of list and sorted set. There is no any registry to manage workers, basic requirements (die after some lifetime and do not hang) can be handled by workers themselves. Worker do not store result by default.

Queue overhead benchmarks

DSQ has a little overhead in compare with RQ and Celery (https://gist.github.com/baverman/5303506cd89200cf246af7bafd569b2e)

Pushing and processing 10k trivial add tasks:

=== DSQ ===
Push
real        0m0.906s
user        0m0.790s
sys         0m0.107s

Process
real        0m1.949s
user        0m0.763s
sys         0m0.103s


=== RQ ===
Push
real        0m3.617s
user        0m3.177s
sys         0m0.293s

Process
real        0m57.706s
user        0m29.807s
sys         0m20.070s


=== Celery ===
Push
real        0m5.753s
user        0m5.237s
sys         0m0.327s

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

dsq-0.9.tar.gz (14.5 kB view details)

Uploaded Source

File details

Details for the file dsq-0.9.tar.gz.

File metadata

  • Download URL: dsq-0.9.tar.gz
  • Upload date:
  • Size: 14.5 kB
  • Tags: Source
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/4.0.2 CPython/3.10.9

File hashes

Hashes for dsq-0.9.tar.gz
Algorithm Hash digest
SHA256 dc3dd7d7663457625762411c4fd301b9ab110bc5c2266d5ecd37741f3c0c14df
MD5 e95a6b470977ebf48401c9bd8a4ed291
BLAKE2b-256 26753ad5c2ce947a00520cb6c8df7fe1d9681eded95f5a5c5a3de2e75fa38c24

See more details on using hashes here.

Supported by

AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Pingdom Monitoring Sentry Error logging StatusPage Status page