pgbg

PostgreSQL-orchestrated background threads for Python

pgbg builds on bgt (a great way to reliably run background threads) and adds PostgreSQL integration:

• A NOTIFY dispatcher that takes one database connection per process and wakes up an arbitrary number of background threads.

• A table-based leader election with automatic failover. Make sure only one process runs work at a time.

Even with database backing, background tasks are not a job queue like Celery or RQ¹. Common use cases include:

• Periodic cleanup duties for expired caches or sessions.
• Maintenance of eventually consistent read models.
• Lightweight background tasks with the transactional outbox pattern.

The core needs and supports only Psycopg 3 for database access. pgbg comes with optional support for SQLAlchemy in the pgbg.sqlalchemy module that connects everything to a SQLAlchemy Engine. If you use psycopg-pool, you don't need an adapter at all: pass your pool's connection method wherever pgbg asks for a connection provider.

Check out our step-by-step tutorial to get an instant feel for the features!

Installation

The package is available on PyPI under the pgbg name. It comes with two optional extras:

• sqlalchemy (uv pip install 'pgbg[sqlalchemy]') currently only adds a SQLAlchemy>=2 dependency.
• pool (uv pip install 'pgbg[pool]') installs psycopg-pool.

Documentation

Full documentation lives at https://pgbg.hynek.me/.

Release Information

Removed

• The supervision layer moved to the new bgt package, which pgbg now depends on. pgbg.Supervisor, pgbg.Service, pgbg.SupervisedService, pgbg.IntervalOnlyWakeup, pgbg.as_work_factory, pgbg.exceptions.SuppressedCrashError, and the pgbg.typing.Loop, pgbg.typing.Wakeup, pgbg.typing.DoWork, and pgbg.typing.WorkFactory protocols are gone. Import them from bgt instead; their behavior is unchanged.

Added

• pgbg.sqlalchemy.init_db() that takes a SQLAlchemy Engine or Connection. #1

Changed

• The service.notified log event is now called service.woken. It fires on any wakeup, not only on notifications.

• The restart counter is now called bgt_supervisor_restarts_total, because bgt's supervisor drives every loop. The metrics of elected services keep their pgbg_ prefix. Supervision log events now go to the bgt logger; everything else stays on pgbg.

Full Changelog →

Credits

pgbg is written by Hynek Schlawack and distributed under the terms of the MIT license.

The development is kindly supported by my employer Variomedia AG and all my fabulous GitHub Sponsors.