localqueue

A persistent, multiprocess-safe local queue for Python, backed by SQLite and Rust.

localqueue gives scripts and workers durable jobs with ACK/NACK, leases, bounded retries, and dead-letter handling—without a server, daemon, or external service.

• Durable: jobs survive process restarts in a local SQLite database.
• Safe under concurrency: multiple Python threads and processes can share a queue on the same machine.
• Failure-aware: expired leases and transient failures are retried; exhausted jobs move to a dead-letter state.
• Fenced: every delivery has a unique receipt, so stale workers cannot acknowledge a newer delivery.
• Flexible: use multiple named queues, optional job deduplication, automatic worker heartbeats, and custom serializers.

Installation · Quick start · Worker · Guarantees · API · Development

Installation

Using uv:

uv add localqueue

Using pip:

python -m pip install localqueue

localqueue requires Python 3.10 or newer.

Quick start

from localqueue import SimpleQueue

with SimpleQueue("./data", lease_seconds=30, max_retries=3) as queue:
    queue.put(
        {"task": "send-email", "to": "hello@example.com"},
        job_id="welcome-email-42",
    )

    job = queue.get()

    try:
        send_email(job.data)
    except Exception as error:
        queue.nack(job, last_error=str(error))
    else:
        queue.ack(job)

The path passed to SimpleQueue is a directory. The queue creates and manages localqueue.db inside it. Payloads are JSON-serialized by default.

Worker

Worker handles the ACK/NACK lifecycle for you. A successful handler is acknowledged, an unexpected exception is retried, and an exception listed in permanent_errors is sent directly to the dead-letter state.

from localqueue import SimpleQueue, Worker


class InvalidDeployment(Exception):
    pass


def deploy(job):
    print(f"Deploying {job.data['app']}@{job.data['revision']}")


with SimpleQueue("./data", lease_seconds=30, max_retries=3) as queue:
    worker = Worker(
        queue,
        deploy,
        permanent_errors=(InvalidDeployment,),
        heartbeat_interval=10,
    )
    worker.run()

For long-running handlers, heartbeat_interval renews the lease in the background. A handler can also renew it explicitly with job.extend_lease(seconds).

Delivery guarantees

put() ──> ready ──> leased ──> acked
                     │
                     ├── nack() or expired lease ──> ready
                     │
                     └── fail() or retry limit ────> failed
                                                        │
                                           retry_failed() ──> ready

By default, SQLite uses synchronous=NORMAL, which protects against normal process crashes but may lose recent transactions after an operating-system or power failure. Pass fsync=True to use synchronous=FULL for stronger durability.

See Delivery guarantees for the complete durability, lease, retry, fencing, and deduplication contract.

Configuration

queue = SimpleQueue(
    "./data",
    name="emails",
    lease_seconds=60,
    max_retries=3,
    fsync=False,
    serializer=None,
)

API overview

Architecture

The public API is a small Python facade over a native PyO3 extension. Rust owns the transactional state machine, while SQLite WAL mode provides local persistence and coordinates competing writers.

Python application
       │
       ▼
SimpleQueue / Worker       Python API and serialization
       │
       ▼
Rust native extension      leases, receipts, retries, transactions
       │
       ▼
localqueue.db              SQLite WAL, one machine

All queue state lives in one SQLite table. Claims and multi-step transitions use immediate transactions so concurrent processes cannot reserve the same delivery. This is local infrastructure for a shared database on one machine, not a distributed message broker.

Development

Build the extension in a local virtual environment and run the test suite:

uv sync --extra dev
uv run maturin develop
uv run pytest

Rust quality checks:

cargo fmt --all --check
cargo clippy --locked --all-targets --all-features -- -D warnings

Build a release wheel:

uv run maturin build --release --locked