Postgres Broker for Dramatiq Task Queue
dramatiq-pg − Postgres Broker for Dramatiq
Dramatiq is a simple task queue implementation for Python3. dramatiq-pg provides a Postgres-based implementation of a dramatiq broker.
The project is not feature complete yet.
- Super simple deployment.
- Uses plain psycopg2. No ORM.
- Stores message payload as native JSONb.
- Stores all messages in a single table, in a dedicated schema.
- Uses LISTEN/NOTIFY to keep worker sync. No polling.
- Replay pending messages on worker startup.
- Requeues failed tasks.
- Delayed task.
- Reliable thanks to Postgres MVCC.
- Self-healing. Old messages are purge from time to time.
Note that dramatiq assumes tasks are idempotent. This broker makes the same assumptions for recovering after a crash.
- Install dramatiq-pg package from PyPI:
$ pip install dramatiq-pg
- Apply dramatiq_pg/schema.sql file in your database:
$ psql -f dramatiq_pg/schema.sql
- Before importing actors, define global broker with a connection
import dramatiq import psycopg2.pool from dramatiq_pg import PostgresBroker dramatiq.set_broker(PostgresBroker(url="postgresql:///?minconn=0&maxconn=10")) @dramatiq.actor def myactor(): ...
The CLI tool
dramatiq-pg allows you to requeue messages, purge old messages
and show stats on the queue. See
--help for details.
Postgres does not replicate notifications to standby instances. Thus the broker connection pool must point to the master instance. Actor can connect to hot standby for its work.
If you use pgbouncer, you must configure session pooling method to keep notify.
Each dramatiq process opens one persistent connection per queue and one
connection to ack messages. Thus, to be save, you should provision pool size
num_processes x num_queues x 2. A best practice is to only add process as
needed and reduce the number of queues.
- Result storage as JSONb.
Feel free to suggest feature through support channels.
If you encounter a bug or miss a feature, please open an issue on GitLab with as much information as possible.
dramatiq_pg is available under the PostgreSQL licence.
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
|Filename, size||File type||Python version||Upload date||Hashes|
|Filename, size dramatiq_pg-0.4.1-py3-none-any.whl (18.2 kB)||File type Wheel||Python version py3||Upload date||Hashes View hashes|
|Filename, size dramatiq-pg-0.4.1.tar.gz (7.6 kB)||File type Source||Python version None||Upload date||Hashes View hashes|
Hashes for dramatiq_pg-0.4.1-py3-none-any.whl