walbox

Async Python runtime for consuming PostgreSQL logical replication as a stream of committed transactions.

📖 Documentation · ⚡ Quickstart

Some reasons you might want to use walbox:

• You want to react to database changes in real time without polling tables or adding fragile trigger-based workarounds.
• You care about reliable delivery: walbox checkpoints progress durably so it can resume cleanly after restarts or connection drops.
• You’re building around the transactional outbox pattern and want a consumer that preserves commit ordering and handles recovery sensibly.
• You want backpressure built in, so a slow handler does not turn into unbounded memory growth or silent data loss.
• You need a Python async consumer that plugs directly into PostgreSQL logical replication instead of shoving data through a separate broker.
• You want to stream changes from published tables without writing your own replication client, reconnect logic, or checkpoint tracking.
• You want operational visibility: retries, lag, queue depth, and checkpoint timing are all surfaced so production behavior is easier to reason about.

Guarantees

• At-least-once delivery: a durable local checkpoint ensures committed events are not silently skipped
• Transactional consistency: outbox events are committed atomically with your business data
• No polling: consumes changes directly from PostgreSQL logical replication
• Bounded backpressure: queue limits prevent a slow handler from accumulating unbounded memory
• Automatic recovery: resumes from the last durable checkpoint on restart
• Graceful shutdown: lets in-flight work complete before exiting
• Asyncio-native: built for Python's async runtime

Install

pip install walbox

Psycopg 3 is the only Python dependency. By default it uses your system's libpq; if you don't have it installed, use the binary distribution: pip install walbox psycopg[binary].

Example

PostgreSQL setup (one-time):

CREATE TABLE outbox (
    id BIGSERIAL PRIMARY KEY, entity_type TEXT, entity_id TEXT,
    event_type TEXT, payload JSONB, created_at TIMESTAMPTZ DEFAULT now()
);
CREATE PUBLICATION walbox_pub FOR TABLE outbox;

Handler (handler.py):

import asyncio
from walbox import (
    ChangeKind,
    CheckpointHandle,
    Transaction,
    Walbox,
    WalboxOptions,
)


async def handle(tx: Transaction, checkpoint: CheckpointHandle) -> None:
    for change in tx.changes:
        if change.table == "public.outbox" and change.kind == ChangeKind.INSERT:
            print(f"Event: {change.new}")
    await checkpoint.save(tx.commit_lsn)


async def main():
    options = WalboxOptions(
        consumer_name="app",
        dsn="postgresql://user:password@localhost/db",
        slot_name="slot",
        publication_name="walbox_pub",
    )
    client = Walbox.build(options)
    await client.run(handle)


asyncio.run(main())

Run it, then insert a row:

python handler.py
# In another terminal:
INSERT INTO outbox (entity_type, entity_id, event_type, payload)
VALUES ('user', '42', 'created', '{"name":"Alice"}'::jsonb);

The handler receives the row and saves a durable checkpoint. On restart, it resumes from that checkpoint. No data loss.

See Examples for working patterns: publishing to a message broker, writing to a PostgreSQL sink with exactly-once effects, sharded concurrent handling, and metrics.

Next steps

• Quickstart (5 minutes): step-by-step setup
• Getting Started: concepts and guarantees
• Production Guide: deployment, monitoring, configuration

Status

walbox is at v1.0.0. It has 100% branch coverage, integration tests against real PostgreSQL, and went through three pre-release cycles (beta.1, beta.2, rc.1) to settle the API.

See CHANGELOG.md for what's changed release to release, and the Upgrading guide if you're on a pre-1.0.0 version.

API Stability

As of v1.0.0, the following are stable and won't break within the 1.0.x series:

• Walbox (build, build_with_pool)
• Client
• WalboxOptions
• ConnectionPool
• Transaction
• ChangeEvent, ChangeKind
• CheckpointHandle
• Metrics, MetricsCallback
• WalboxError and its subclasses

walbox.abc and walbox.checkpoint expose the underlying protocols and the PostgreSQL checkpoint store, for advanced use like manual Client construction or a custom CheckpointStore. These are lower-level and may change shape in a minor release.

1.0.x releases can add fields or methods, but won't remove or change the meaning of anything listed above.

Development: see CONTRIBUTING.md for setup and LICENSE for terms.