Skip to main content
Python Software Foundation 20th Year Anniversary Fundraiser  Donate today!

Asynchronous Redis-backed reliable queue package

Project description

Pipeline status Test code coverage PyPI RTFD

Torrelque

Torrelque is a Python package that provides asynchronous reliable Redis-backed work queues.

Note

Prior to version 0.2 Torrelque was Tornado-specific. Since version 5 Tornado runs on asyncio event loop by default, hence newer Torrelque is still compatible with (newer) Tornado, even though it’s based on asyncio now.

Install

pip install Torrelque

Quickstart

Producer:

redis = aredis.StrictRedis()
queue = torrelque.Torrelque(redis, queue='email')
queue.schedule_sweep()

task_data = {'addr': 'joe@doe.com', 'subj': 'hello', 'body': '...'}
task_id = await queue.enqueue(task)
logger.info('Email task enqueued %s', task_id)

Consumer:

redis = aredis.StrictRedis()
queue = torrelque.Torrelque(redis, queue='email')

while True:
    task_id, task_data = await queue.dequeue()
    try:
        await some_email_client.send(**task_data)
    except SomeEmailError:
        logger.exception('Sending error, retrying in 30 seconds')
        await queue.requeue(task_id, delay=30)
    else:
        await queue.release(task_id)

Example list

  • Producer-consumer. Infinite producing and consuming loops.
  • Batch processing. Finite number of tasks, consumers stop with a poison pill, bulk enqueue. This example can be used as a synthetic benchmark. Because there’s no IO-bound workload, it’ll be CPU-bound which isn’t normal mode of operation for an asynchronous application. But it can be used to compare between CPython, PyPy and concurrency parameters.
  • Web application background task. This tornado application allows to start a task and push server-sent events (SSE) to UI about its status. UI starts a task and waits for it to complete. When a task fails it’s requeued with exponential back-off.

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Files for Torrelque, version 0.4.0
Filename, size File type Python version Upload date Hashes
Filename, size Torrelque-0.4.0.tar.gz (11.9 kB) File type Source Python version None Upload date Hashes View

Supported by

AWS AWS Cloud computing Datadog Datadog Monitoring DigiCert DigiCert EV certificate Facebook / Instagram Facebook / Instagram PSF Sponsor Fastly Fastly CDN Google Google Object Storage and Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Salesforce Salesforce PSF Sponsor Sentry Sentry Error logging StatusPage StatusPage Status page