Polyqueue
Async-first Python job queue that keeps Postgres as the authoritative job state while supporting Redis, SQS, Azure Service Bus, PGMQ, and an in-process broker. Handlers are typed via Pydantic. Return values are persisted to the same Postgres DB — no separate result backend required.
| Dimension | Polyqueue | Celery | Taskiq |
|---|---|---|---|
| Primary model | Small queue runtime with Postgres as source of truth plus pluggable broker adapters | Mature distributed task queue with broad ecosystem/tooling | Async-first distributed task runner with modular brokers/extensions |
| Async support | Yes, async-first. Worker, handlers, DB access, and adapters are designed around asyncio | Not primarily async-first in overall design | Yes, async-first |
| Built-in broker/backend support | Redis, SQS, Azure Service Bus, PGMQ, none (in-process) | Commonly used with Redis/RabbitMQ and other broker/result-backend combinations | Broker-based, typically via pluggable integrations |
| Redis support | Yes | Yes | Yes, depending on broker plugin/config |
| SQS support | Yes | Yes | Possible via broker integration/plugin, but not the default mental model |
| Azure Service Bus support | Yes | Not a standard first-class default choice in the same way as Redis/RabbitMQ | Possible via ecosystem extension/custom broker, but not a core default pairing |
| No-broker / in-process mode | Yes via queue_backend="none" |
Not the usual model | Not the main model |
| Source of truth for job state | Postgres | Broker/backend-centric task system, optionally paired with result backends | Broker/message-centric |
| Pydantic support | Yes — enqueue() accepts BaseModel, dispatch() validates annotated payload models via model_validate() |
Available via explicit task-side support | Works naturally with typed task signatures and model parsing patterns |
| Pydantic behavior | Strict: model serialised to dict on enqueue, model_validate() on dispatch, ValidationError => TerminalError (no retry) |
Usually task-side validation/conversion | More ergonomic/best-effort style than strict contract-first behavior |
| Result persistence | Postgres jobs.result JSONB — same DB already tracking job state, no extra infrastructure |
Requires a separate result backend (Redis, DB, RPC, etc.) configured explicitly; off by default | Requires a separate result backend plugin (e.g. taskiq-redis); pluggable but extra infrastructure |
| Multi-worker distribution | Yes. Workers compete on claim; one worker owns the in-flight delivery | Yes | Yes |
| Worker identity tracking | Yes — polyqueue_workers registry: worker_id, hostname, pid, current job, heartbeat, status (running/stopped/dead); stale workers reaped by maintenance loop |
Stronger built-in worker identity/inspection model | Workers exist, but less built-in cluster observability than Celery |
| In-flight ownership model | Redis lease / SQS receipt handle + visibility timeout / Azure SB lock token + lock renewal | Broker/worker-managed | Broker/worker-managed |
| Recovery when worker crashes | Yes, via lease expiry / visibility timeout / lock expiry and redelivery/reclaim | Yes | Yes |
| Detecting "alive but wedged" jobs | Basic — progress_heartbeat_at + timeout_at in DB; maintenance backstop reconciles timed-out jobs |
Better operational story; mature controls exist | Typically needs app/broker-level design |
| Time limits | Yes — per-job max_run_seconds, cooperative timeout via asyncio.wait_for, DB backstop, three strategies (retry/fail/ignore) |
More mature built-in support | Less central than in Celery's ops model |
| Operational maturity | Focused, understandable, smaller surface area | Most mature | Lighter-weight, async-friendly |
| Best fit | Internal service jobs where you want simplicity, explicit semantics, and control | Large-scale general background job platform | Async Python apps that want typed/ergonomic task execution |
Summary
- Brokers: Redis, SQS, Azure Service Bus, PGMQ, or in-process (
none). Postgres always holds the authoritative job state regardless of broker. - Pydantic-first: handlers declare typed payloads via
@register;ValidationErroron dispatch becomesTerminalError(no retry). - Result persistence in Postgres: handler return values (
BaseModel, dict, or primitive) are saved tojobs.resultJSONB — no separate result backend needed. Fetch viaget_job_result,get_job_result_typed, orget_job_result_typed_via_registry. Not available with thenone/in-process backend. - Time limits: per-job
max_run_secondsenforced viaasyncio.wait_for, with three strategies (retry/fail/ignore) and a maintenance-loop DB backstop. - Worker registry:
polyqueue_workerstable tracks id, hostname, pid, current job, and heartbeat. Stale workers are reaped todeadby the maintenance loop. - Metrics hook: pluggable
MetricsHookfor Prometheus, CloudWatch, or OpenTelemetry. Pass tomain(metrics=...). - Admin CLI:
polyqueue inspect stats|workers|jobsandpolyqueue admin requeue|fail. - Smaller surface area than Celery — no admin UI or auto-scaling built-in.
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
polyqueue-0.0.1.tar.gz
(143.1 kB
view details)
Built Distribution
Filter files by name, interpreter, ABI, and platform.
If you're not sure about the file name format, learn more about wheel file names.
Copy a direct link to the current filters
polyqueue-0.0.1-py3-none-any.whl
(70.0 kB
view details)
File details
Details for the file polyqueue-0.0.1.tar.gz.
File metadata
- Download URL: polyqueue-0.0.1.tar.gz
- Upload date:
- Size: 143.1 kB
- Tags: Source
- Uploaded using Trusted Publishing? No
- Uploaded via:
uv/0.9.26 {"installer":{"name":"uv","version":"0.9.26","subcommand":["publish"]},"python":null,"implementation":{"name":null,"version":null},"distro":{"name":"macOS","version":null,"id":null,"libc":null},"system":{"name":null,"release":null},"cpu":null,"openssl_version":null,"setuptools_version":null,"rustc_version":null,"ci":null}
File hashes
| Algorithm | Hash digest | |
|---|---|---|
| SHA256 |
6e2ff82a4705fbd894e279403e931d1b3d11864668524fe1ddd700c6236cbc23
|
|
| MD5 |
39d634569f1e769b82e8a01400f03ca8
|
|
| BLAKE2b-256 |
54090159ec5d7ccbd6eb535c17a813790d78eaefc6faccf9341bfcc1a31e105d
|
File details
Details for the file polyqueue-0.0.1-py3-none-any.whl.
File metadata
- Download URL: polyqueue-0.0.1-py3-none-any.whl
- Upload date:
- Size: 70.0 kB
- Tags: Python 3
- Uploaded using Trusted Publishing? No
- Uploaded via:
uv/0.9.26 {"installer":{"name":"uv","version":"0.9.26","subcommand":["publish"]},"python":null,"implementation":{"name":null,"version":null},"distro":{"name":"macOS","version":null,"id":null,"libc":null},"system":{"name":null,"release":null},"cpu":null,"openssl_version":null,"setuptools_version":null,"rustc_version":null,"ci":null}
File hashes
| Algorithm | Hash digest | |
|---|---|---|
| SHA256 |
a9e6a05cef193d0541d70df675442a75577e3d7e0d31bad7680ce686eedddab3
|
|
| MD5 |
ef8e8eb4fbec005f3fc44ec850fc4c96
|
|
| BLAKE2b-256 |
e29e7ac05612ef150cef1196e8364c3784adeca47bc97fa8c2b624467ad2eb3e
|