smallage

Background tasks, cron and a dead letter queue on Redis Streams — typed tasks, retries with backoff, delayed jobs, and consumption of streams somebody else writes. The core reaches for no web framework; the Litestar layer adds real dependency injection and a CLI through the native plugin protocol.

Smallage is the old name for wild celery.

📖 Documentation

Features

• Tasks are ordinary functions: arguments are serialised, dependencies injected
• Retries with backoff, a delivery ceiling, and a dead letter queue that keeps the payload, the traceback and the attempt history
• Delayed jobs and cron with no scheduler process, correct across DST
• Priority queues with a bounded starvation window, and shards for fairness
• Broker mode: consume streams somebody else writes, in the same worker
• Optional results, a deduplication gate, and trace context carried into the task
• Health endpoint served identically by the web process and the worker
• Eager mode, assert_enqueued and a real-worker fixture for your own tests

Installation

uv add smallage

Redis 7 or newer. Standalone, Sentinel and Cluster are all covered by the test suite.

Quick start

from dataclasses import dataclass
from uuid import UUID, uuid4

from litestar import Litestar, post
from litestar.di import Provide

from smallage.litestar import QueueConfig, QueuePlugin, TaskRegistry

tasks = TaskRegistry()


@dataclass
class Settings:
    index_name: str = "documents"


def settings() -> Settings:
    return Settings()


@tasks.task
async def reindex(doc_id: UUID, settings: Settings) -> None:
    """`doc_id` is serialised; `settings` comes from the application."""


@post("/documents")
async def create() -> str:
    await reindex.enqueue(doc_id=uuid4())
    return "queued"


app = Litestar(
    route_handlers=[create],
    dependencies={"settings": Provide(settings, sync_to_thread=False)},
    plugins=[QueuePlugin(QueueConfig(registry=tasks, redis_url="redis://localhost"))],
)

doc_id travels in the payload; settings is injected in the worker from the application's own dependency graph, and a real one would be a database session or a client. There is no context dictionary.

Workers

A worker is the same application, started differently:

litestar workers run --queue high --concurrency 20

The lifecycle comes with it: a worker enters the application's lifespan, so on_startup hooks and lifespan managers run there too and a dependency closing over what they opened is usable in a task. run_app_lifespan=False declines that, for a lifespan whose work belongs to a web process alone.

Anything that can be settled at startup is — a task registered twice, an argument with no annotation, a dependency the application does not provide, a cycle, a provider only a request could satisfy — rather than on the first job in production.

Delivery guarantee

At-least-once. A worker that completed its side effect and died before XACK will be reclaimed and the work repeated. No amount of protocol work removes that; only an idempotent handler does. A dedup key is provided for the cases where that is not naturally true.

Documentation

Tasks, scheduling, retries, priorities, broker mode, results, testing and operations are covered in the full documentation. The rules the library is built to are in Design and invariants. Runnable examples are included.

Development

make install   # dependencies and git hooks
make check     # lint, types, import contracts, unit tests
make test-int  # integration suite, needs Docker

License

MIT