Torrelque
Torrelque is a Python package that provides minimal asynchronous reliable distributed Redis-backed (or a protocol-compatible alternative) work queues. It is built:
Lock-free. It relies on Redis transactions and its single-threaded execution model.
Poll-free. Waiting subset of the Python API relies either on blocking Redis commands or notifications.
Bulk-friendly. Tasks can be produced and consumed in bulk.
Introspectable. Task stats, task status transition watching API, and the data model comprehensible directly in Redis.
Supported Redis server implementations: Redis, KeyDB, Valkey.
Install
pip install Torrelque
Quickstart
Producer:
import redis.asyncio
import torrelque
client = redis.asyncio.Redis()
queue = torrelque.Torrelque(client, queue='email')
queue.schedule_sweep() # to make due requeued tasks available again
task_data = {'addr': 'joe@doe.com', 'subj': 'hello', 'body': '...'}
task_id = await queue.enqueue(task_data)
print('Email task enqueued', task_id)Consumer:
import redis.asyncio
import torrelque
client = redis.asyncio.Redis()
queue = torrelque.Torrelque(client, queue='email')
while True:
task_id, task_data = await queue.dequeue()
try:
await some_email_client.send(**task_data)
except Exception:
print('Email sending error, retrying in 30s', task_id)
await queue.requeue(task_id, delay=30)
else:
print('Email sent', task_id)
await queue.release(task_id)Example list
Producer-consumer. Infinite producing and consuming loops.
Batch processing. Finite number of tasks, consumers stop with a poison pill, bulk enqueue. This example can be used as a synthetic benchmark. Because there’s no IO-bound workload, it’ll be CPU-bound which isn’t normal mode of operation for an asynchronous application. But it can be used to compare between CPython, PyPy and concurrency parameters.
Web application background task. This tornado application allows to start a task and push server-sent events (SSE) to UI about its status. UI starts a task and waits for it to complete. When a task fails it’s re-queued with exponential back-off.
1 maintainer
Metadata
1 maintainer
Release files for Torrelque 0.9.0
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | Provenance | |
|---|---|---|---|---|
| torrelque-0.9.0.tar.gz | 15.2 kB | Without provenance | Details |
Release files / torrelque-0.9.0.tar.gz
| Download URL | torrelque-0.9.0.tar.gz |
|---|---|
| Size | 15.2 kB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
40e8830445dbc83c446251d54727f3d0034fec4f6a384161ac5010362eabf201
|
|
BLAKE2b-256 checksum How to use checksums |
761245eaf22e48cb12c39d7d274a4a842155231069716aa259fb44907838d8b8
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/6.2.0 CPython/3.10.19
|
Security for Torrelque 0.9.0
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
No provenance found
None of the files in this release were uploaded with provenance metadata.
A note on package security
PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.
