simplebroker-redis

Valkey/Redis backend extension for SimpleBroker.

This package exposes the public SimpleBroker backend name redis. It targets Valkey 7.x and Redis 7.x and the test suite runs against Valkey.

Requirements

• Python 3.11+
• Valkey 7.x or Redis 7.x

Durability depends on the server configuration. A Valkey or Redis deployment without AOF/RDB persistence can lose messages on restart. Use SQLite or Postgres when you need storage durability from the broker stack by default.

Regular broker commands use a redis-py BlockingConnectionPool owned by the process-local Redis runner. Pub/Sub wake hints use a separate dedicated connection because subscribed Redis connections cannot serve normal commands.

Pool defaults:

• max_connections = 50
• pool_timeout = BROKER_BUSY_TIMEOUT / 1000

The defaults can be overridden in project backend options:

[backend_options]
namespace = "simplebroker_redis_v1"
max_connections = 50
pool_timeout = 5.0

Pool exhaustion is bounded by pool_timeout and surfaces as an operational error from broker operations.

Alias creation validates the live alias map and publishes the alias plus its version in one Lua operation. New aliases must remain flat in either creation order. A canonical target may have no messages or existing rows; it only needs valid queue-name syntax. Legacy invalid rows remain available for one-hop lookup and removal and are not rewritten automatically.

Concurrency semantics

Stale at-least-once batches are recovered after stale_batch_seconds (default 300; negative disables recovery). Recovery qualifies age using exact integer nanoseconds, then atomically rechecks that token's source and creation timestamp, reads its live IDs, releases reservations, and removes its token keys. A token that committed, rolled back, disappeared, or changed after the scan is skipped. This prevents an old recovery scan from releasing a newer batch's reservations for the same message. The count is the number of live token IDs processed.

This repair changes no storage keys or backend API version. Quiesce clients for each affected namespace and upgrade all of them: an older client can still run the unsafe recovery sequence. The fix prevents new corruption; it does not repair previously duplicated IDs or missing bodies. Rolling back restores the race and is not a safe service fallback.

Queue deletion is atomic per queue. delete() first snapshots the queue registry, then one Lua invocation per selected queue rechecks active at-least-once reservations and removes that queue's pending, claimed, body, and global-ID state together. A queue created after the registry snapshot is outside the operation and is not deleted. If a reservation starts between per-queue invocations, deletion stops with an error; queues already processed remain deleted, while that reserved queue and later queues remain intact.

Patternless broadcast() selects the current queue registry and inserts every copy in one Lua invocation. A queue cannot be missed or resurrected by a concurrent write or deletion that commits before that invocation. Activity notifications and maintenance accounting run after the atomic insert commits.

Exact-target broadcast(..., queue_names=...) also intersects the requested literal names with the registry and inserts all copies in one Lua invocation. Missing names are ignored and not created. A requested queue deleted before the script selects targets is not resurrected; an all-missing request returns zero without advancing persisted last_ts, publishing wakeups, or scheduling maintenance.

Python create_missing=True changes exact selection to the complete requested set. The script validates all anticipated failures before its first mutation, then adds missing names to the registry and inserts every message in one non-interleaved Lua phase. A queue deleted before that phase is intentionally recreated.

Patterned broadcasts deliberately keep a client-side queue snapshot so their matching stays exactly Python fnmatchcase syntax. A queue created after the snapshot can miss that broadcast; a queue deleted after the snapshot can be recreated by it. Use a patternless or exact-target broadcast when atomic registry selection is required.

Writes and broadcasts retry only explicit timestamp and message-ID conflicts that Redis reports before mutation. ID collisions use bounded exponential backoff for up to 30 seconds, with each sleep capped at 250 ms. A stale high-water fence refreshes and retries immediately so another writer cannot make the refreshed value stale during a backoff sleep. Retry sleeps observe the core stop event; interruption raises the same repeated-conflict error as budget exhaustion. A write holds the process-local write lock for this retry interval, so sustained ID collisions can delay sibling writers. Transport errors and lost responses are not retried because the server may already have committed the operation.

Exact-target broadcast requires backend API v5: SimpleBroker 5.6.1 or newer and simplebroker-redis 3.3.1 or newer.

SimpleBroker 7.1.0 and simplebroker-redis 3.6.0 are the first coordinated backend API v6 set, which adds terminal activity-waiter close. Package dependency floors are minimums; the exact runtime handshake remains authoritative for every installed pair.

SimpleBroker 7.3.0 and simplebroker-redis 3.8.0 are the first coordinated backend API v7 set. It adds the required monotone durable high-water advance used by persistence restore. Package dependency floors remain minimums; the exact runtime handshake remains authoritative for every installed pair.

SimpleBroker 8.0.0 and simplebroker-redis 4.0.0 are the first coordinated backend API v8 set. It adds bounded public-ID selection order. Package dependency floors remain minimums; the exact runtime handshake remains authoritative for every installed pair.

SimpleBroker 8.1.0 and simplebroker-redis 4.1.0 are the first coordinated backend API v9 set. It adds the atomic write-time keep_newest pending window in one Lua script. The script validates key types, displaced bodies, and active reservations before mutation. A displaced active reservation produces a retryable failure with no write or claim; a reservation in the retained newest-N set does not conflict. The server is blocked for the script; measured 100k/220k displaced rows took about 166/377 ms on Valkey 7.2. Cost is linear in displaced rows and has no bounded-time guarantee.

Core Compatibility

This first-party extension declares a minimum supported SimpleBroker core version and its backend API version independently. SimpleBroker checks the exact API handshake when it resolves the plugin. An incompatible pair fails at backend resolution with upgrade-or-pin guidance instead of running against an unknown interface. Core and extension package version numbers do not match. The backend API version is separate from the Redis storage schema version and is not stored in Redis. A breaking private-seam change requires a backend API version bump.

The package dependency is an install-time minimum; the runtime handshake is the authoritative interface check. Install the extension through the core release's redis extra. See the backend authoring guide for the handshake boundary.

Multi-Queue Activity Waiters

Redis/Valkey supports simplebroker.create_activity_waiter_for_queues(...) with queue-scoped Pub/Sub registrations. Wakeups are hints; callers still drain queues through normal SimpleBroker operations.

Close the waiter explicitly when its watcher lifecycle ends. The first close() marks the composite terminal before closing its children, attempts every independently safe ordinary cleanup, and raises the first failure with later failures retained as ordered exception notes. Every later close is a no-op, including when the first call raised. The waiter owns registrations, not the runner or shared Pub/Sub listener, and does not expose shutdown().