Skip to main content

TQueues

TQueues (reThinkdb Queues) is a simple python3.5+ library for queueing jobs and processing them in workers using rethinkdb.

Features

  • Accepts coroutines as jobs

  • Distributed

  • Handles everything in a rethinkdb database object that can be reused at the worker

  • Exposes changes via websockets

  • Uses rethinkdb streaming “changes” method, wich is pretty efficient

  • Uses asyncio, implements an async context manager for jobs and async iterator for the workers

                        +--> Worker
                        |
Rethindb --> Dispatcher ---> Worker
                        |
                        +--> Worker

Usage

TQueues provides two entry points, tqueues_dispatcher and tqueues_worker.

TQueues dispatcher must be accesible from all workers, and you need to start one worker per parallel task you need. They may be distributed (as long as they can reach the dispatcher)

TQueues worker

Usage:
    tqueues_worker -h | --help
    tqueues_worker --version
    tqueues_worker --endpoint_url <endpoint_url>
    tqueues_worker --queue <queue>

Options:
    -h --help                        Show this screen
    -v --version                     Show version
    --endpoint_url <ENDPOINT_URL>    TQueues dispatcher endpoint
    --queue        <QUEUE>           Endpoint queue to listen on

Examples:
    tqueues_worker --endpoint_url http://127.0.0.1:800/ --queue testqueue

TQueues job dispatcher

Usage:
    tqueues_dispatcher --db "db" --host "127.0.0.1" --port 28015
    tqueues_dispatcher --db "db" --host "127.0.0.1"
    tqueues_dispatcher --db "db" --port 28015
    tqueues_dispatcher --db "db" --user 'user'
    tqueues_dispatcher --db "db" --password 'password'
    tqueues_dispatcher -h | --help
    tqueues_dispatcher --version

Options:
    --host "127.0.0.1"                          Rethinkdb host
    --db "db"                                   Rethinkdb databaes
    --port 28015                                Rethinkdb port
    --user 'user'                               Rethinkdb user
    --password 'password'                       Rethinkdb password
    --allowed_domains 'foo.com,bar.com'         Allowed domains
    --loglevel (DEBUG|INFO)                     Loglevel
    -h   --help                                 Show this screen
    --version                                   Show version

Examples:
    tqueues_dispatcher --host localhost --db foo --port 28015 --user foo --password bar --loglevel INFO --allowed_domains 'foo.com,bar.com'

Metadata

Release files for tqueues 0.1.7

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for tqueues 0.1.7
File Size Uploaded
tqueues-0.1.7.tar.gz 15.4 kB Details

Release files / tqueues-0.1.7.tar.gz

Download URL tqueues-0.1.7.tar.gz
Size 15.4 kB
Tags Source
SHA-256 checksum
How to use checksums
e01aa651686adbc4de8c0f45bb73e7ece339b234c08594d1e34832688071639c
BLAKE2b-256 checksum
How to use checksums
ea2e6c32f4d9e927ddd607b0648f67e76cc276b3e16003811592a38cca9a1fe5
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page