Dead simple task queue using redis.
# tasks.py
import dsq
manager = dsq.create_manager()
@manager.task(queue='normal')
def add(a, b):
print a + b
if __name__ == '__main__':
add.push(1, 2)$ python tasks.py
$ dsq worker -bt tasks normalSee full DSQ documentation.
Features
Low latency.
Expiring tasks (TTL).
Delayed tasks (ETA).
Retries (forever or particular amount).
Periodic tasks.
Crontab.
Dead letters.
Queue priorities.
Worker lifetime.
Task execution timeout.
Task forwarder from one redis instance to another.
HTTP interface.
Inspect tools.
Supports 2.7, >3.4 and PyPy.
100% test coverage.
The goal is a simple design. There is no worker manager, one can use supervisord/circus/whatever to spawn N workers. Simple storage model. Queue is a list and scheduled tasks are a sorted set. There are no task keys. Tasks are items of list and sorted set. There is no any registry to manage workers, basic requirements (die after some lifetime and do not hang) can be handled by workers themselves. Worker do not store result by default.
Queue overhead benchmarks
DSQ has a little overhead in compare with RQ and Celery (https://gist.github.com/baverman/5303506cd89200cf246af7bafd569b2e)
Pushing and processing 10k trivial add tasks:
=== DSQ === Push real 0m0.906s user 0m0.790s sys 0m0.107s Process real 0m1.949s user 0m0.763s sys 0m0.103s === RQ === Push real 0m3.617s user 0m3.177s sys 0m0.293s Process real 0m57.706s user 0m29.807s sys 0m20.070s === Celery === Push real 0m5.753s user 0m5.237s sys 0m0.327s
1 maintainer
Release files for dsq 0.9
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | |
|---|---|---|---|
| dsq-0.9.tar.gz | 14.5 kB | Details |
Release files / dsq-0.9.tar.gz
| Download URL | dsq-0.9.tar.gz |
|---|---|
| Size | 14.5 kB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
dc3dd7d7663457625762411c4fd301b9ab110bc5c2266d5ecd37741f3c0c14df
|
|
BLAKE2b-256 checksum How to use checksums |
26753ad5c2ce947a00520cb6c8df7fe1d9681eded95f5a5c5a3de2e75fa38c24
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/4.0.2 CPython/3.10.9
|
