Skip to main content

Postgres Broker for Dramatiq Task Queue

Project description

dramatiq-pg -- Postgres Broker for Dramatiq

dramatiq is a simple task queue implementation for Python3. dramatiq-pg provides a Postgres-based implementation of a dramatiq broker.

The project is not feature complete yet.

Features

  • Super simple deployment.
  • Message payload stored as native JSONb.
  • All messages in a single table.
  • All data are wrapped in a dedicated schema.
  • Uses LISTEN/NOTIFY to keep worker sync. No polling.
  • Reliable thanks to Postgres MVCC.
  • Using plain psycopg2. No ORM.
  • Requeueing of failed tasks.

Installation

  • Install dramatiq-pg package from PyPI:
    $ pip install dramatiq-pg
    
  • Apply dramatiq_pg/schema.sql file in your database:
    $ psql -f dramatiq_pg/schema.sql
    
  • Before importing actors, define global broker with a connection pool:
    import dramatiq
    import dramatiq_pg
    import psycopg2.pool
    
    pool = psycopg2.pool.ThreadedConnectionPool(0, 4, conninfo)
    dramatiq.set_broker(dramatiq_pg.PostgresBroker(pool=pool))
    

Now declare/import actors and manage worker just like any dramatiq setup.

Roadmap

  • Rejecting message.
  • Process missed notifies while resuming worker.
  • Result storage as JSONb.
  • Delayed task.

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Files for dramatiq-pg, version 0.2.0
Filename, size File type Python version Upload date Hashes
Filename, size dramatiq_pg-0.2.0-py3-none-any.whl (7.9 kB) File type Wheel Python version py3 Upload date Hashes View hashes
Filename, size dramatiq-pg-0.2.0.tar.gz (4.0 kB) File type Source Python version None Upload date Hashes View hashes

Supported by

Elastic Elastic Search Pingdom Pingdom Monitoring Google Google BigQuery Sentry Sentry Error logging AWS AWS Cloud computing DataDog DataDog Monitoring Fastly Fastly CDN DigiCert DigiCert EV certificate StatusPage StatusPage Status page