pgbg

PostgreSQL-orchestrated background threads for Python

pgbg builds on bgt (a great way to reliably run background threads) and adds PostgreSQL integration:

• A NOTIFY dispatcher that takes one database connection per process and wakes up an arbitrary number of background threads.

• A table-based leader election with automatic failover. Make sure only one process runs work at a time.

Even with database backing, background tasks are not a job queue like Celery or RQ¹. Common use cases include:

• Periodic cleanup duties for expired caches or sessions.
• Maintenance of eventually consistent read models.
• Lightweight background tasks with the transactional outbox pattern.

The core needs and supports only Psycopg 3 for database access. pgbg comes with optional support for SQLAlchemy in the pgbg.sqlalchemy module that connects everything to a SQLAlchemy Engine. If you use psycopg-pool, you don't need an adapter at all: pass your pool's connection method wherever pgbg asks for a connection provider.

Check out our step-by-step tutorial to get an instant feel for the features!

Installation

The package is available on PyPI under the pgbg name. It comes with two optional extras:

• sqlalchemy (uv pip install 'pgbg[sqlalchemy]') currently only adds a SQLAlchemy>=2 dependency.
• pool (uv pip install 'pgbg[pool]') installs psycopg-pool.

Documentation

Full documentation lives at https://pgbg.hynek.me/.

Release Information

Added

• pgbg.NotifyDispatcher.wake() that makes a dispatch loop you run yourself notice its stop event at once. #3

Changed

• Stopping a dispatcher, subscribing, and closing a subscription take effect at once instead of within one interval. A LISTEN connection that PostgreSQL closes is replaced at once, too. interval now only sets how often an idle dispatcher checks that its connection is alive. #3

• As such, the default interval of pgbg.NotifyDispatcher, pgbg.SupervisedDispatcher.start(), and pgbg.sqlalchemy.start_dispatcher() went from 1 to 30 seconds. #3

Full Changelog →

Credits

pgbg is written by Hynek Schlawack and distributed under the terms of the MIT license.

The development is kindly supported by my employer Variomedia AG and all my fabulous GitHub Sponsors.