Skip to main content

xtr-orm-messenger

A database transport for xtr-messenger: messages kept in a table on an xtr-orm connection, claimed by workers, delayed, redelivered and woken by notifications.

python 3.14+ typed license MIT

Why?

A queue needs a broker, and a broker is one more server to run, back up and watch. An application that already has a database can keep its messages there: in a table, on the connection xtr-orm already configures, sent in the same transaction as the changes they announce if it wants.

  • 🗄️ No broker — orm://default is the application's own default connection.
  • 🔒 Workers never collide — each claims the oldest due message FOR UPDATE SKIP LOCKED.
  • ⏰ Delays — a DelayStamp makes a message due later.
  • ♻️ Nothing is lost to a dead worker — a message never settled is handed out again.
  • 🔔 No polling on PostgreSQL — a waiting worker is woken the moment a message is sent.
  • 🤝 Sent with the handler's changes — transactional=true writes the message in the unit of work's session, so it commits or rolls back with them.

Install

uv add xtr-orm-messenger
uv add "xtr-messenger[orm]"            # the same, from the message bus's side

It needs xtr-messenger and xtr-orm, which come with it. It brings no database driver: a transport writes through the connection its DSN names, with the driver the application installed for that connection — xtr-orm's postgres, mysql or sqlite extra. On PostgreSQL through asyncpg, a waiting worker is woken by notifications; with any other driver it polls. It ships no bundle: with a container, the orm bundle registers the transport once this package is installed.

Quick start

Without a container, hand the factory the connections it builds on and the envelope serializer the bus and its workers share:

from xtr_messenger import (
    JsonSerializer,
    MessageBusConfig,
    MessageBusFactory,
    TransportConfig,
    WorkerFactory,
)
from xtr_orm import ConnectionRegistry
from xtr_orm_messenger import OrmTransportFactory
from xtr_serializer import JsonEncoder, PydanticNormalizer, Serializer

registry = ConnectionRegistry()
registry.register("default", engine=engine, migrator=migrator, database=database)

config = MessageBusConfig(
    transports={"jobs": TransportConfig("orm://default", queue="jobs")},
    routing={IngestDocument: "jobs"},
)
wire = JsonSerializer(Serializer([PydanticNormalizer()], [JsonEncoder()]))
factory = OrmTransportFactory(registry, wire)

await MessageBusFactory(config, [factory]).bus().dispatch(IngestDocument(document_id=1))
await WorkerFactory(config, [factory]).worker(["jobs"]).run()

The bus writes the message into messenger_messages; the worker claims it, hands it to its handlers and deletes it. With a container all of that is wired for you.

How it works

Step What happens
Send One row: the encoded envelope, its queue, created_at, and available_at — later by the delay, rounded up to a whole second
Claim In one transaction, the oldest row of the queue that is due and not handed out is selected FOR UPDATE SKIP LOCKED and stamped delivered_at
Handle The worker dispatches it; the envelope carries OrmReceivedStamp and TransportMessageIdStamp, both the row's id
Keep alive While the worker holds it, the row's delivered_at is renewed every half of redeliver_timeout, so a long handler keeps its message
Settle Acknowledged, the row is deleted. Rejected, it is retried or moved to the failure queue (see below)
Redeliver A row not renewed for redeliver_timeout seconds is claimable again: its worker is taken for dead. A handler must be idempotent: a worker frozen that long has its message handled twice

Due times and the redelivery window are measured on each worker's clock, not the database's: keep the workers' clocks synchronized. A clock running ahead claims delayed messages early and takes a live worker's message for abandoned.

A rejected message gets max_attempts attempts in all, the first included. While it has attempts left its row is put back in its queue, due base_delay_seconds later, the delay doubling with each attempt; the envelope carries a RedeliveryStamp with the attempt count and an ErrorDetailsStamp with why the last one failed. Once its attempts run out the row moves to the failure_queue of the same table, failed unless named otherwise, where count(), all() and find() on a transport reading that queue show it, and a worker consuming it tries it again, with a fresh count. A message failing for good in the failure queue itself is deleted. The worker asks the transport before it announces a failure, so WorkerMessageFailedEvent carries will_retry=True for an attempt that will be made again.

A worker that finds nothing due waits poll_interval seconds before asking again. On PostgreSQL through asyncpg it listens instead: every send announces itself on the table's channel, naming its queue, and the worker asks again at once — or after check_delayed_interval, for a delayed message falling due. Listening holds one connection of the pool for as long as the worker runs. If the database drops that connection, the wait under way ends and the next one listens on a fresh connection; while listening again fails, the worker polls every poll_interval and keeps trying. A claim whose worker is cancelled before it hears of it is handed back at once, not left until redeliver_timeout.

SQLite has no row locks, so the lock clause is left out there, and SQLite is for development and tests. With the driver's default transaction mode two workers can claim one row; run one worker on SQLite, or pass connect_args={"autocommit": False}, under which the second gets "database is locked", which counts as a conflict and is tried again.

Conflicts between workers — a deadlock, a lock wait that timed out — say nothing about the message. A poll sits through 3 in a row before it raises; an acknowledgement or a rejection tries again with a pause doubling from 100 ms. One whose database went away waits up to 5 seconds for it and is replayed once: losing an acknowledgement to a restart would hand the message out again.

A message that does not decode can never be handled, so it is rejected at once, its reason logged on the messenger logger — never its body.

The DSN

# .env
MESSENGER_DSN=orm://default

The format is orm://<connection_name>: the name of a connection the application configures in xtr-orm, for when it has several and the messages go on one other than default; orm:// alone is the default connection. The URL, the driver and the pool are the connection's. The transport creates a table named messenger_messages the first time it runs. To name the table otherwise, give table_name in the DSN:

# .env
MESSENGER_DSN=orm://default?table_name=your_custom_table_name

The queue a transport reads and writes is queue — ?queue=jobs, or TransportConfig(dsn, queue="jobs") — the setting every xtr-messenger transport takes for it.

Options

Every option may go in the DSN's query string or in TransportConfig.options; options wins. An unknown one is refused with UnknownTransportOptionError, a bad value with InvalidTransportOptionError.

Option Default What it sets
queue default the queue this transport reads and writes; also TransportConfig(queue=...)
table_name messenger_messages the table, a plain identifier
redeliver_timeout 3600 seconds after which a message handed out and never settled is handed out again
auto_setup true create the table the first time an operation finds it missing
poll_interval 1.0 seconds a worker waits before asking again when nothing was due
transactional false write the message in the session of the unit of work under way
use_notify true on PostgreSQL through asyncpg, wait for notifications instead of polling
check_delayed_interval 60000 milliseconds a listening worker waits at most before asking anyway
get_notify_timeout 60000 milliseconds one wait for a notification lasts at most
max_attempts 3 attempts a message gets, the first included, before it moves to the failure queue
base_delay_seconds 1.0 seconds before the second attempt; each later one waits twice as long
failure_queue failed the queue of the same table a message is moved to once its attempts run out

orm:// with no name uses the default connection. A name the application did not configure is refused with InvalidOrmDsnError when the transport is built.

Sending with the handler's changes

With transactional=true, a message dispatched while a message is handled — or a request, or a command: any unit of work — is written in that unit's session and sent only when the unit commits. Run the orm_transaction middleware, or commit yourself: a session never commits on its own. Outside a unit of work the message is written in a transaction of its own, as it is without the option. A follow-up held back with DispatchAfterCurrentBusStamp in a worker goes out after the handled message's unit is over; with transactional=true it is written in the worker's session only when orm_transaction runs for it too, so keep that middleware on the bus the worker uses.

Looking into a queue

OrmTransport is a TransportInterface with more to ask:

transport = factory.transport("jobs", config.transports["jobs"])

await transport.count()  # what a worker could claim now
async for envelope in transport.all(limit=10):
    ...  # the same, oldest first
await transport.find(message_id)  # one message, handed out or not
await transport.keepalive(envelope, seconds=600)  # a long handler keeps its message
await transport.setup()  # create the table ahead of time

keepalive restarts a message's redelivery clock; seconds is how long until the next call, refused when longer than redeliver_timeout. A worker already renews what it holds on its own; call it for a message held some other way.

Use in an application

Everything adding this package to an application takes — and, read backwards, what removing it undoes. It ships no bundle: the orm bundle registers the transport, built on its connections, when the messenger bundle is active too.

  • Install — uv add xtr-orm-messenger, or uv add "xtr-messenger[orm]", beside xtr-messenger and xtr-orm set up as their Use in an application says. The database driver is the connection's own.
  • Recipe — uv run xtr-recipes recipes:sync writes nothing: there is no bundle to list, no configuration file, environment variable or ignore line. It prints the step naming a transport, which a recipe cannot make for you.
  • Configure — name a transport orm://default in <app>/config/packages/messenger.py and route messages to it, as below.
  • Run — <script> orm:migrations:diff writes the messages table of every orm:// transport into its connection's next revision; until it is applied, the transport creates the table the first time it runs, unless auto_setup=false. A worker is <script> messenger:consume <transport>.
  • Remove — drop the orm:// transports and the routing to them, and a migration dropping the table. Then uv remove xtr-orm-messenger.
  • Check — debug:config messenger lists the orm:// transports.

Everything else — the bundles, the connections, the bus — is xtr-orm's and xtr-messenger's.

# app/config/packages/messenger.py
from xtr_dependency_injection import configure
from xtr_messenger.bundle import MessageBusConfig, TransportConfig


@configure
def messenger() -> MessageBusConfig:
    return MessageBusConfig(
        transports={"jobs": TransportConfig("orm://default?auto_setup=false", queue="jobs")},
        routing={IngestDocument: "jobs"},
    )

The orm bundle registers OrmTransportFactory with the messenger, built on its ConnectionRegistry and the messenger's envelope serializer, and consulted ahead of the factories found by DSN scheme. A bus and every worker of the container share it, and so share each transport. When the kernel shuts down, the connections workers listened on are given back.

Without a container, messages_tables(bus_config, connection, default_connection) names the tables the transports of a bus configuration keep on a connection, and MessagesTableDriver adds one to the MetaData a connection's migrations compare with.

Errors

Everything this library raises derives from OrmMessengerError, itself a MessageBusError of xtr-messenger:

Error Raised when
TransportError the database refuses what a transport asked — sending, claiming, settling, listing — or an envelope settled here was not received here; the driver's error is its cause
InvalidOrmDsnError an orm:// DSN names a connection the application does not configure; also a LookupError

Upgrading

What changes for an application moving to 7.0, and how to keep the old behaviour where that is still possible: Upgrading to 7.0.

Development

Developed in the python-xtr monorepo, under packages/xtr-orm-messenger; run the commands below from there. The python-xtr-orm-messenger repository is a read-only copy, so send issues and pull requests to the monorepo.

uv sync
uv run ruff check && uv run ruff format --check && uv run basedpyright && uv run ty check && uv run pytest

Every test runs on SQLite files. The PostgreSQL suite under tests/postgresql/ runs when XTR_TEST_POSTGRES_URL names a database, as an asyncpg URL:

XTR_TEST_POSTGRES_URL=postgresql+asyncpg://user:password@127.0.0.1/test uv run pytest tests/postgresql

License

MIT — see LICENSE.

Metadata

Release files for xtr-orm-messenger 7.0.0

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for xtr-orm-messenger 7.0.0
File Size Uploaded Provenance Actions
xtr_orm_messenger-7.0.0.tar.gz 37.4 kB With provenance Details for xtr_orm_messenger-7.0.0.tar.gz

Built distribution (wheel)

Table of built distributions (wheels) for xtr-orm-messenger 7.0.0
File Interpreter ABI Platform Actions
xtr_orm_messenger-7.0.0-py3-none-any.whl Python 3 none any Details for xtr_orm_messenger-7.0.0-py3-none-any.whl

Total release size: 80.0 kB

Release files / xtr_orm_messenger-7.0.0.tar.gz

Download URL xtr_orm_messenger-7.0.0.tar.gz
Size 37.4 kB
Tags Source
SHA-256 checksum
How to use checksums
051d872c61187f91f529b0b318cf78788499bea337b913575a0caabc5ee84f0f
BLAKE2b-256 checksum
How to use checksums
6f8caa4083b3674e1ea3cff20124f88d55b298be2dcf8ab1376f6e861783c60b
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
Yes
Uploaded via twine/7.0.0 CPython/3.13.16

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Nov 19, 2026.

Transparency log

Release files / xtr_orm_messenger-7.0.0-py3-none-any.whl

Download URL xtr_orm_messenger-7.0.0-py3-none-any.whl
Size 42.6 kB
Tags Python 3
SHA-256 checksum
How to use checksums
5998f8d131b9c7e52b978b81cb7dbb63c5e57c4ab99ca6ea8cd73118bdd1983b
BLAKE2b-256 checksum
How to use checksums
21338ad7eb2a7d40f516c33640547caf4331cdac48da4b56d6d5056ec4e80800
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
Yes
Uploaded via twine/7.0.0 CPython/3.13.16

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Nov 19, 2026.

Transparency log

Security for xtr-orm-messenger 7.0.0

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

Full provenance

Both files in this release have been uploaded with provenance metadata.

Select from 2 files

Attestation for xtr_orm_messenger-7.0.0.tar.gz

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Nov 19, 2026.

Transparency log

A note on package security

PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.

To learn more about protecting yourself from malicious actors, visit our package security documentation.

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page