Asynchronous Redis-backed reliable queue package
Project description
Torrelque
Torrelque is a Python package that provides minimal asynchronous reliable distributed Redis-backed work queues. It is built:
Lock-free. It relies on Redis transactions and its single-threaded execution model.
Poll-free. Waiting subset of the Python API relies either on blocking Redis commands or notifications.
Bulk-friendly. Tasks can be produced and consumed in bulk.
Introspectable. Task stats, task status transition watching API, and the data model comprehensible directly in Redis.
Install
pip install Torrelque
Quickstart
Producer:
redis = yaaredis.StrictRedis()
queue = torrelque.Torrelque(redis, queue='email')
queue.schedule_sweep()
task_data = {'addr': 'joe@doe.com', 'subj': 'hello', 'body': '...'}
task_id = await queue.enqueue(task)
logger.info('Email task enqueued %s', task_id)
Consumer:
redis = yaaredis.StrictRedis()
queue = torrelque.Torrelque(redis, queue='email')
while True:
task_id, task_data = await queue.dequeue()
try:
await some_email_client.send(**task_data)
except SomeEmailError:
logger.exception('Sending error, retrying in 30 seconds')
await queue.requeue(task_id, delay=30)
else:
await queue.release(task_id)
Example list
Producer-consumer. Infinite producing and consuming loops.
Batch processing. Finite number of tasks, consumers stop with a poison pill, bulk enqueue. This example can be used as a synthetic benchmark. Because there’s no IO-bound workload, it’ll be CPU-bound which isn’t normal mode of operation for an asynchronous application. But it can be used to compare between CPython, PyPy and concurrency parameters.
Web application background task. This tornado application allows to start a task and push server-sent events (SSE) to UI about its status. UI starts a task and waits for it to complete. When a task fails it’s requeued with exponential back-off.
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.