Skip to main content

Asynchronous Redis-backed reliable queue package

Project description

PyPI - License Pipeline status Test code coverage PyPI RTFD

Torrelque

Torrelque is a Python package that provides minimal asynchronous reliable distributed Redis-backed (or a protocol-compatible alternative) work queues. It is built:

  1. Lock-free. It relies on Redis transactions and its single-threaded execution model.

  2. Poll-free. Waiting subset of the Python API relies either on blocking Redis commands or notifications.

  3. Bulk-friendly. Tasks can be produced and consumed in bulk.

  4. Introspectable. Task stats, task status transition watching API, and the data model comprehensible directly in Redis.

Supported Redis server implementations: Redis, KeyDB.

Install

pip install Torrelque

Quickstart

Producer:

import redis.asyncio
import torrelque

client = redis.asyncio.Redis()
queue = torrelque.Torrelque(client, queue='email')
queue.schedule_sweep()  # to make due requeued tasks available again

task_data = {'addr': 'joe@doe.com', 'subj': 'hello', 'body': '...'}
task_id = await queue.enqueue(task_data)
print('Email task enqueued', task_id)

Consumer:

import redis.asyncio
import torrelque

client = redis.asyncio.Redis()
queue = torrelque.Torrelque(client, queue='email')

while True:
    task_id, task_data = await queue.dequeue()
    try:
        await some_email_client.send(**task_data)
    except Exception:
        print('Email sending error, retrying in 30s', task_id)
        await queue.requeue(task_id, delay=30)
    else:
        print('Email sent', task_id)
        await queue.release(task_id)

Example list

  • Producer-consumer. Infinite producing and consuming loops.

  • Batch processing. Finite number of tasks, consumers stop with a poison pill, bulk enqueue. This example can be used as a synthetic benchmark. Because there’s no IO-bound workload, it’ll be CPU-bound which isn’t normal mode of operation for an asynchronous application. But it can be used to compare between CPython, PyPy and concurrency parameters.

  • Web application background task. This tornado application allows to start a task and push server-sent events (SSE) to UI about its status. UI starts a task and waits for it to complete. When a task fails it’s re-queued with exponential back-off.

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

Torrelque-0.7.1.tar.gz (14.8 kB view details)

Uploaded Source

File details

Details for the file Torrelque-0.7.1.tar.gz.

File metadata

  • Download URL: Torrelque-0.7.1.tar.gz
  • Upload date:
  • Size: 14.8 kB
  • Tags: Source
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/5.0.0 CPython/3.10.13

File hashes

Hashes for Torrelque-0.7.1.tar.gz
Algorithm Hash digest
SHA256 43f6957f4bf087b6dc6bdc133bcf1bdce5c68fe854e8700c885cc24edca7fd97
MD5 ada76d5441ca560b1f32794318a238f6
BLAKE2b-256 2d55a693144625632521b8faebc4545d39f6bfe3a0cd7cba31c3aeb110cdc2a3

See more details on using hashes here.

Supported by

AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Pingdom Monitoring Sentry Error logging StatusPage Status page