pgbg: PostgreSQL-orchestrated background threads for Python

POV: you want a framework-agnostic way to reliably run a plain¹ function or method in the background, repeatedly, but not all the time.

pgbg comes to the rescue with:

• A NOTIFY dispatcher that takes one database connection per process and wakes up an arbitrary number of subscribers.

• A supervisor that runs your code as a service: in a loop, in a background thread. If your code crashes, the supervisor restarts the loop. Write crash-only code, pgbg takes care of the rest.

  Your services can wake up on NOTIFYs, fixed time intervals, or both.

• PostgreSQL-based leader election with automatic failover. Make sure only one process runs work at a time.

• Framework and platform independence.

Background tasks are not a traditional worker queue². Common use cases include:

• Periodic cleanup duties for expired caches or sessions.
• Maintenance of eventually consistent read models.
• Lightweight transactional background tasks with the outbox pattern.

The core needs and supports only Psycopg 3 for database access. pgbg comes with optional support for SQLAlchemy in the pgbg.sqlalchemy module that connects everything to a SQLAlchemy Engine.

Check out our step-by-step tutorial to get an instant feel for the features!

Installation

The package is available on PyPI under the pgbg name. It comes with an optional extra sqlalchemy (uv pip install 'pgbg[sqlalchemy]'), but that currently only adds a SQLAlchemy>2 lower pin.

Documentation

Full documentation lives at https://pgbg.hynek.me/.

Credits

pgbg is written by Hynek Schlawack and distributed under the terms of the MIT license.

The development is kindly supported by my employer Variomedia AG and all my fabulous GitHub Sponsors.