Torrelque 0.4.0

Asynchronous Redis-backed reliable queue package

Torrelque

Torrelque is a Python package that provides asynchronous reliable Redis-backed work queues.

Note

Prior to version 0.2 Torrelque was Tornado-specific. Since version 5 Tornado runs on asyncio event loop by default, hence newer Torrelque is still compatible with (newer) Tornado, even though it’s based on asyncio now.

Install

pip install Torrelque

Quickstart

Producer:

redis = aredis.StrictRedis()
queue = torrelque.Torrelque(redis, queue='email')
queue.schedule_sweep()

task_data = {'addr': '[e-mail]', 'subj': 'hello', 'body': '...'}
task_id = await queue.enqueue(task)
logger.info('Email task enqueued %s', task_id)

Consumer:

redis = aredis.StrictRedis()
queue = torrelque.Torrelque(redis, queue='email')

while True:
    task_id, task_data = await queue.dequeue()
    try:
        await some_email_client.send(**task_data)
    except SomeEmailError:
        logger.exception('Sending error, retrying in 30 seconds')
        await queue.requeue(task_id, delay=30)
    else:
        await queue.release(task_id)

Example list

• Producer-consumer. Infinite producing and consuming loops.
• Batch processing. Finite number of tasks, consumers stop with a poison pill, bulk enqueue. This example can be used as a synthetic benchmark. Because there’s no IO-bound workload, it’ll be CPU-bound which isn’t normal mode of operation for an asynchronous application. But it can be used to compare between CPython, PyPy and concurrency parameters.
• Web application background task. This tornado application allows to start a task and push server-sent events (SSE) to UI about its status. UI starts a task and waits for it to complete. When a task fails it’s requeued with exponential back-off.