Skip to main content

Rethinkdb job queues

Project description


TQueues (reThinkdb Queues) is a simple python3.5+ library for queueing jobs and processing them in workers using rethinkdb.


  • Accepts coroutines as jobs
  • Distributed
  • Handles everything in a rethinkdb database object that can be reused at the worker
  • Exposes changes via websockets
  • Uses rethinkdb streaming “changes” method, wich is pretty efficient
  • Uses asyncio, implements an async context manager for jobs and async iterator for the workers
                        +--> Worker
Rethindb --> Dispatcher ---> Worker
                        +--> Worker


TQueues provides two entry points, tqueues_dispatcher and tqueues_worker.

TQueues dispatcher must be accesible from all workers, and you need to start one worker per parallel task you need. They may be distributed (as long as they can reach the dispatcher)

TQueues worker

    tqueues_worker -h | --help
    tqueues_worker --version
    tqueues_worker --endpoint_url <endpoint_url>
    tqueues_worker --queue <queue>

    -h --help                        Show this screen
    -v --version                     Show version
    --endpoint_url <ENDPOINT_URL>    TQueues dispatcher endpoint
    --queue        <QUEUE>           Endpoint queue to listen on

    tqueues_worker --endpoint_url --queue testqueue

TQueues job dispatcher

    tqueues_dispatcher --db "db" --host "" --port 28015
    tqueues_dispatcher --db "db" --host ""
    tqueues_dispatcher --db "db" --port 28015
    tqueues_dispatcher --db "db" --user 'user'
    tqueues_dispatcher --db "db" --password 'password'
    tqueues_dispatcher -h | --help
    tqueues_dispatcher --version

    --host ""                          Rethinkdb host
    --db "db"                                   Rethinkdb databaes
    --port 28015                                Rethinkdb port
    --user 'user'                               Rethinkdb user
    --password 'password'                       Rethinkdb password
    --allowed_domains ','         Allowed domains
    --loglevel (DEBUG|INFO)                     Loglevel
    -h   --help                                 Show this screen
    --version                                   Show version

    tqueues_dispatcher --host localhost --db foo --port 28015 --user foo --password bar --loglevel INFO --allowed_domains ','

Project details

Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Filename, size & hash SHA256 hash help File type Python version Upload date
tqueues-0.1.7.tar.gz (15.4 kB) Copy SHA256 hash SHA256 Source None

Supported by

Elastic Elastic Search Pingdom Pingdom Monitoring Google Google BigQuery Sentry Sentry Error logging AWS AWS Cloud computing DataDog DataDog Monitoring Fastly Fastly CDN SignalFx SignalFx Supporter DigiCert DigiCert EV certificate StatusPage StatusPage Status page