Skip to main content

tarsk

A Python task queue whose workers hold a memory ceiling you set — without losing a task.

task with rust through the middle. The scheduler, retry state machine, lease tracking and child supervision are Rust; the only Python in the hot path is your handler.

PyPI Python License

pip install tarsk            # memory broker only
pip install tarsk-redis      # + Redis Streams
pip install tarsk-postgres   # + Postgres
pip install tarsk-amqp       # + RabbitMQ

Pick one. All four are the same tarsk with a different broker compiled in — same import, same API, same tarsk package — so installing two of them leaves you running whichever one pip unpacked last. tarsk refuses to import when it finds more than one, and a build without the backend your URL asks for names the package you want.

Upgrading from 0.1.x: this is a breaking change. pip install tarsk has shipped Redis, Postgres and AMQP since 0.1. From 0.2.0 it ships none of them — it is the memory-broker build. If your broker URL is anything but memory://, pip uninstall tarsk and install the package for your broker. Nothing in your code changes. See CHANGELOG.md.

Upgrading from 0.3: a cancelled job is now dropped rather than dead-lettered, and a Postgres worker holds one more connection. From 0.2, broker failures raise tarsk.BrokerError instead of ValueError. Both lists are at the top of their sections in CHANGELOG.md.

Status: early. The version badge above reads live from PyPI — a number written here went stale twice before this sentence replaced it. Wheels for Linux (glibc and musl, x86_64 and aarch64), macOS universal2 and Windows x64, on Python 3.11 through 3.14 including the free-threaded build. See What is missing.

📖 Documentation — everything below the fold lives there: how it works, writing tasks, routing and scheduling, operating, benchmarks.

# pip install tarsk-redis
from tarsk import App

app = App(broker="redis://localhost:6379/0")

@app.task(retries=3, timeout=30, queue="heavy")
def embed_document(doc_id: str) -> dict:
    ...

task_id = embed_document.send("abc")
tarsk worker --app myapp:app --broker redis://localhost:6379/0 \
             --queues heavy --children 4 --max-rss 400MB --metrics 0.0.0.0:9090

Handlers must be idempotent. Delivery is at-least-once. This is a contract, not a footnote: a worker killed mid-task will run that task again.

What it does that others do not

child RSS over one hour under a leaky handler, and the supervisor holding it

One hour, 72,001 tasks, a handler that never frees anything. 66 recycles, peak 400 MB against a 400 MB ceiling, no sample over it, nothing killed, nothing lost. The flat line beneath the sawtooth, on the same axis, is the supervisor doing the holding: 28.32 to 28.40 MB across the hour, a drift of 82 KB. A ceiling enforced from inside a process with the same problem would only defer it.

Every Python task queue leaks, because leaks come from the code they run rather than from the queue. The difference is what the runtime does about it.

--max-rss is a byte budget. Celery's --max-tasks-per-child is a task count, which only bounds bytes if you already know how many bytes a task costs — and stays wrong once that changes. Same configuration, different workloads:

leak 20MB/task leak 40MB/task payload-dependent 2–80MB
celery --max-tasks-per-child=6 163 MB 283 MB 328 MB
tarsk --max-rss=200MB 183 MB 183 MB 187 MB

Celery wins the first column, and that is the honest result: when the leak per task is known and constant, dividing the budget by it works — someone divided 200 by 20 and typed 6. It is the other two columns tarsk was built for: nothing was reconfigured between them, the workload moved, and the guess encoded in that 6 went with it.

The worker that runs your code is 28 MB against Celery's 42 MB and taskiq's 47 MB, because it imports your tasks and nothing else — no broker driver, no scheduler. Recycling costs 7 ms at the 99th percentile against Celery's 144 ms, because the replacement child starts before the trigger fires and the slot never goes empty.

What it does not claim

Not faster. With a 50 ms handler tarsk, Celery and taskiq all reach 20 tasks/s, and that is the row that matters. Draining 10,000 no-op tasks across four processes, measured side by side on one machine: Celery 10.8s, taskiq 2.6s, tarsk 1.7s. That lead is repeatable there, and it is still a no-op handler on one kind of machine — on a 16-core laptop the same pair are 10% apart. It starts in half the time and runs in half the memory; that is the claim, and speed is not.

This file claimed a 1.45× lead until the numbers were checked on more than one machine. The full retraction, and every column where tarsk loses, is in the benchmarks.

Not a hard ceiling regardless of task size. The ceiling is read at the dispatch decision, and a handler that allocates 300 MB will allocate it. Overshoot is bounded by the peak of whatever is in flight when the limit is crossed — with --slots 1 that is one task, because the child is idle at the moment it is read; at the default of 100 slots it is up to 100. The budget divides the remaining headroom by the measured per-task cost, so that number is usually far below the slot count, but the only way to bound it at one task is --slots 1.

Not durable execution. That is Temporal's category, and it is much heavier.

What is missing

  • RabbitMQ is the basic backend: jobs, retries, delays, chains and results, and none of the things that need a store every worker can read — cancel, pause, dedup, chords, shared limits
  • Windows runs the suites that need no broker, since neither Redis nor Postgres ships for it. Everything else — the channel, recycling, the memory ceiling — is tested there

Running the tests

The toolchain is pinned in mise.toml — Rust 1.100.0 and Python 3.14.3. With mise installed, mise install fetches both and entering the directory creates .venv from the pinned Python. Without it, any Python 3.11+ and a recent stable Rust will build: the wheel is abi3-py311.

python -m venv .venv && .venv/bin/pip install maturin msgpack
# Every backend in one build: the published wheels take one each, the source
# tree is where all four are tested.
.venv/bin/maturin develop --release -F redis,postgres,amqp,test-hooks

.venv/bin/python tests/test_ipc.py       # protocol, timeouts, retries
.venv/bin/python tests/test_recycle.py   # the ceiling, soft timeouts, middleware
.venv/bin/python tests/test_brokers.py   # all four brokers end to end; name one to run it alone
cargo test --lib                         # cron, console, socket permissions

The broker tests start their own redis-server and Postgres cluster, use any RabbitMQ answering on the conventional ports, and skip whichever is missing — loudly, so a skip cannot pass for a pass. TARSK_REDIS_URL, TARSK_PG_URL and TARSK_AMQP_URL point them at servers you already have instead, which is how a machine without the server binaries still tests those backends. Give each one a database of its own. python demo/run.py --minutes 1 --ceiling 150MB --rate 30 is the definition of done in miniature: it exits non-zero if a task goes missing or the supervisor drifts.

License

MIT — see LICENSE.

Metadata

Release files for tarsk-amqp 0.4.1

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for tarsk-amqp 0.4.1
File Size Uploaded Provenance Actions
tarsk_amqp-0.4.1.tar.gz 176.1 kB Without provenance Details for tarsk_amqp-0.4.1.tar.gz

Built distributions (wheels)

Table of built distributions (wheels) for tarsk-amqp 0.4.1
File Actions
tarsk_amqp-0.4.1-cp314-cp314t-win_amd64.whl CPython 3.14 CPython 3.14 free-threading Windows x86-64 Details for tarsk_amqp-0.4.1-cp314-cp314t-win_amd64.whl
tarsk_amqp-0.4.1-cp314-cp314t-musllinux_1_2_x86_64.whl CPython 3.14 CPython 3.14 free-threading Linux musl 1.2+ x86-64 Details for tarsk_amqp-0.4.1-cp314-cp314t-musllinux_1_2_x86_64.whl
tarsk_amqp-0.4.1-cp314-cp314t-manylinux2014_x86_64.manylinux_2_17_x86_64.whl CPython 3.14 CPython 3.14 free-threading Linux glibc 2.17+ x86-64 Details for tarsk_amqp-0.4.1-cp314-cp314t-manylinux2014_x86_64.manylinux_2_17_x86_64.whl
tarsk_amqp-0.4.1-cp314-cp314t-manylinux2014_aarch64.manylinux_2_17_aarch64.whl CPython 3.14 CPython 3.14 free-threading Linux glibc 2.17+ ARM64 Details for tarsk_amqp-0.4.1-cp314-cp314t-manylinux2014_aarch64.manylinux_2_17_aarch64.whl
tarsk_amqp-0.4.1-cp314-cp314t-macosx_10_12_universal2.macosx_10_12_x86_64.macosx_11_0_arm64.whl CPython 3.14 CPython 3.14 free-threading macOS 10.12+ universal2 (ARM64, x86-64), macOS 10.12+ x86-64, macOS 11.0+ ARM64 Details for tarsk_amqp-0.4.1-cp314-cp314t-macosx_10_12_universal2.macosx_10_12_x86_64.macosx_11_0_arm64.whl
tarsk_amqp-0.4.1-cp311-abi3-win_amd64.whl CPython 3.11 abi3 Windows x86-64 Details for tarsk_amqp-0.4.1-cp311-abi3-win_amd64.whl
tarsk_amqp-0.4.1-cp311-abi3-musllinux_1_2_x86_64.whl CPython 3.11 abi3 Linux musl 1.2+ x86-64 Details for tarsk_amqp-0.4.1-cp311-abi3-musllinux_1_2_x86_64.whl
tarsk_amqp-0.4.1-cp311-abi3-manylinux2014_x86_64.manylinux_2_17_x86_64.whl CPython 3.11 abi3 Linux glibc 2.17+ x86-64 Details for tarsk_amqp-0.4.1-cp311-abi3-manylinux2014_x86_64.manylinux_2_17_x86_64.whl
tarsk_amqp-0.4.1-cp311-abi3-manylinux2014_aarch64.manylinux_2_17_aarch64.whl CPython 3.11 abi3 Linux glibc 2.17+ ARM64 Details for tarsk_amqp-0.4.1-cp311-abi3-manylinux2014_aarch64.manylinux_2_17_aarch64.whl
tarsk_amqp-0.4.1-cp311-abi3-macosx_10_12_universal2.macosx_10_12_x86_64.macosx_11_0_arm64.whl CPython 3.11 abi3 macOS 10.12+ x86-64, macOS 11.0+ ARM64, macOS 10.12+ universal2 (ARM64, x86-64) Details for tarsk_amqp-0.4.1-cp311-abi3-macosx_10_12_universal2.macosx_10_12_x86_64.macosx_11_0_arm64.whl

Total release size: 40.6 MB

Release files / tarsk_amqp-0.4.1.tar.gz

Download URL tarsk_amqp-0.4.1.tar.gz
Size 176.1 kB
Tags Source
SHA-256 checksum
How to use checksums
62584b5e1af3c80b2e4f4535174601553120bf1d9e1d636a0353235ca00f7317
BLAKE2b-256 checksum
How to use checksums
6ba4be73f0249348ea22e2743b81ba0af264a61eddf3da145a579e18a566b61c
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
Yes
Uploaded via uv/0.13.8 {"installer":{"name":"uv","version":"0.13.8","subcommand":["publish"]},"python":null,"implementation":{"name":null,"version":null},"distro":{"name":"Ubuntu","version":"24.04","id":"noble","libc":null},"system":{"name":null,"release":null},"cpu":null,"openssl_version":null,"setuptools_version":null,"rustc_version":null,"ci":true}

Release files / tarsk_amqp-0.4.1-cp314-cp314t-win_amd64.whl

Download URL tarsk_amqp-0.4.1-cp314-cp314t-win_amd64.whl
Size 3.2 MB
Tags CPython 3.14 CPython 3.14 free-threading Windows x86-64
SHA-256 checksum
How to use checksums
4422bd2b580fba461db85b559a1887f412f7840efb7f1d4f51469c2757cc3848
BLAKE2b-256 checksum
How to use checksums
b8f6bec5fc94c1b87c6aaba77d1e6748d9ac0a4c4f724c475bcae9c3e1c19b36
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
Yes
Uploaded via uv/0.13.8 {"installer":{"name":"uv","version":"0.13.8","subcommand":["publish"]},"python":null,"implementation":{"name":null,"version":null},"distro":{"name":"Ubuntu","version":"24.04","id":"noble","libc":null},"system":{"name":null,"release":null},"cpu":null,"openssl_version":null,"setuptools_version":null,"rustc_version":null,"ci":true}

Release files / tarsk_amqp-0.4.1-cp314-cp314t-musllinux_1_2_x86_64.whl

Download URL tarsk_amqp-0.4.1-cp314-cp314t-musllinux_1_2_x86_64.whl
Size 3.9 MB
Tags CPython 3.14 CPython 3.14 free-threading Linux musl 1.2+ x86-64
SHA-256 checksum
How to use checksums
15347945dcba74cd3c41a2a372ece1fb076e41dd60cdb4b4f334af6be0689ea6
BLAKE2b-256 checksum
How to use checksums
8cb89898907caeb3a0018ef1185ae6727c8a777ef27d307371ad785d55b88273
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
Yes
Uploaded via uv/0.13.8 {"installer":{"name":"uv","version":"0.13.8","subcommand":["publish"]},"python":null,"implementation":{"name":null,"version":null},"distro":{"name":"Ubuntu","version":"24.04","id":"noble","libc":null},"system":{"name":null,"release":null},"cpu":null,"openssl_version":null,"setuptools_version":null,"rustc_version":null,"ci":true}

Release files / tarsk_amqp-0.4.1-cp314-cp314t-manylinux2014_x86_64.manylinux_2_17_x86_64.whl

Download URL tarsk_amqp-0.4.1-cp314-cp314t-manylinux2014_x86_64.manylinux_2_17_x86_64.whl
Size 3.6 MB
Tags CPython 3.14 CPython 3.14 free-threading Linux glibc 2.17+ x86-64
SHA-256 checksum
How to use checksums
346def4e4f21c7ac16f0cb615195ad03f97a5ceeaefc86c5fe0458797071d5d1
BLAKE2b-256 checksum
How to use checksums
9795a3827a7e24ac91b3d09b137c77c9e21c36133f747035fa951fe334524f4e
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
Yes
Uploaded via uv/0.13.8 {"installer":{"name":"uv","version":"0.13.8","subcommand":["publish"]},"python":null,"implementation":{"name":null,"version":null},"distro":{"name":"Ubuntu","version":"24.04","id":"noble","libc":null},"system":{"name":null,"release":null},"cpu":null,"openssl_version":null,"setuptools_version":null,"rustc_version":null,"ci":true}

Release files / tarsk_amqp-0.4.1-cp314-cp314t-manylinux2014_aarch64.manylinux_2_17_aarch64.whl

Download URL tarsk_amqp-0.4.1-cp314-cp314t-manylinux2014_aarch64.manylinux_2_17_aarch64.whl
Size 3.1 MB
Tags CPython 3.14 CPython 3.14 free-threading Linux glibc 2.17+ ARM64
SHA-256 checksum
How to use checksums
e695ffbded4e48ff592aa4a3a432477ad5d70fa0ae50853e649d524705f7d782
BLAKE2b-256 checksum
How to use checksums
a22a0e6ddab7dc7747e546a5d665448aec2ed1cf7fffd21331b6ae2a38d49f28
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
Yes
Uploaded via uv/0.13.8 {"installer":{"name":"uv","version":"0.13.8","subcommand":["publish"]},"python":null,"implementation":{"name":null,"version":null},"distro":{"name":"Ubuntu","version":"24.04","id":"noble","libc":null},"system":{"name":null,"release":null},"cpu":null,"openssl_version":null,"setuptools_version":null,"rustc_version":null,"ci":true}

Release files / tarsk_amqp-0.4.1-cp314-cp314t-macosx_10_12_universal2.macosx_10_12_x86_64.macosx_11_0_arm64.whl

Download URL tarsk_amqp-0.4.1-cp314-cp314t-macosx_10_12_universal2.macosx_10_12_x86_64.macosx_11_0_arm64.whl
Size 6.5 MB
Tags CPython 3.14 CPython 3.14 free-threading macOS 10.12+ universal2 (ARM64, x86-64) macOS 10.12+ x86-64 macOS 11.0+ ARM64
SHA-256 checksum
How to use checksums
c0adf3ddaf21ee7b5e96a8b97f2b1a75c05a26ec05acf7ff35623edb94756dd1
BLAKE2b-256 checksum
How to use checksums
aaf60a1d5365cdef6ce2b34b90fa945373b35835eb3202a10291012b19b386f9
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
Yes
Uploaded via uv/0.13.8 {"installer":{"name":"uv","version":"0.13.8","subcommand":["publish"]},"python":null,"implementation":{"name":null,"version":null},"distro":{"name":"Ubuntu","version":"24.04","id":"noble","libc":null},"system":{"name":null,"release":null},"cpu":null,"openssl_version":null,"setuptools_version":null,"rustc_version":null,"ci":true}

Release files / tarsk_amqp-0.4.1-cp311-abi3-win_amd64.whl

Download URL tarsk_amqp-0.4.1-cp311-abi3-win_amd64.whl
Size 3.2 MB
Tags CPython 3.11 Windows x86-64 abi3
SHA-256 checksum
How to use checksums
36d9bae04979327f82c03f72dc369dc0cf5baa729411ae1bcf776631c6fa8a3f
BLAKE2b-256 checksum
How to use checksums
404719644bda220aaf335d95d7717850fa358accef12a7d8e7aea1bb462b5f49
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
Yes
Uploaded via uv/0.13.8 {"installer":{"name":"uv","version":"0.13.8","subcommand":["publish"]},"python":null,"implementation":{"name":null,"version":null},"distro":{"name":"Ubuntu","version":"24.04","id":"noble","libc":null},"system":{"name":null,"release":null},"cpu":null,"openssl_version":null,"setuptools_version":null,"rustc_version":null,"ci":true}

Release files / tarsk_amqp-0.4.1-cp311-abi3-musllinux_1_2_x86_64.whl

Download URL tarsk_amqp-0.4.1-cp311-abi3-musllinux_1_2_x86_64.whl
Size 3.9 MB
Tags CPython 3.11 Linux musl 1.2+ x86-64 abi3
SHA-256 checksum
How to use checksums
e382df4a027e64d71ee6ea30d5751cd859d2ddc33121facdc9597d9ed6619d61
BLAKE2b-256 checksum
How to use checksums
3ba556ed22f408023ffe8890174a98fa88bd78579d79e2f02e9267cad9ba4742
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
Yes
Uploaded via uv/0.13.8 {"installer":{"name":"uv","version":"0.13.8","subcommand":["publish"]},"python":null,"implementation":{"name":null,"version":null},"distro":{"name":"Ubuntu","version":"24.04","id":"noble","libc":null},"system":{"name":null,"release":null},"cpu":null,"openssl_version":null,"setuptools_version":null,"rustc_version":null,"ci":true}

Release files / tarsk_amqp-0.4.1-cp311-abi3-manylinux2014_x86_64.manylinux_2_17_x86_64.whl

Download URL tarsk_amqp-0.4.1-cp311-abi3-manylinux2014_x86_64.manylinux_2_17_x86_64.whl
Size 3.6 MB
Tags CPython 3.11 Linux glibc 2.17+ x86-64 abi3
SHA-256 checksum
How to use checksums
a3c1b1266b04cd9fe2d3872c78b28ccc1e66b7cb3454e316c4f62189e7299aea
BLAKE2b-256 checksum
How to use checksums
fdd6e3f431187ab7fd4b50c58c950a25d60201e06bcab6c44f41825fea2ee50c
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
Yes
Uploaded via uv/0.13.8 {"installer":{"name":"uv","version":"0.13.8","subcommand":["publish"]},"python":null,"implementation":{"name":null,"version":null},"distro":{"name":"Ubuntu","version":"24.04","id":"noble","libc":null},"system":{"name":null,"release":null},"cpu":null,"openssl_version":null,"setuptools_version":null,"rustc_version":null,"ci":true}

Release files / tarsk_amqp-0.4.1-cp311-abi3-manylinux2014_aarch64.manylinux_2_17_aarch64.whl

Download URL tarsk_amqp-0.4.1-cp311-abi3-manylinux2014_aarch64.manylinux_2_17_aarch64.whl
Size 3.1 MB
Tags CPython 3.11 Linux glibc 2.17+ ARM64 abi3
SHA-256 checksum
How to use checksums
760d9382485780bf1f0c61b62d3d1ba6e58bb4764634b7dcca6a7f6dd75d668a
BLAKE2b-256 checksum
How to use checksums
e737854b99134243c9c793e00211f12307e68db80749d73923778cdbd16d5c57
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
Yes
Uploaded via uv/0.13.8 {"installer":{"name":"uv","version":"0.13.8","subcommand":["publish"]},"python":null,"implementation":{"name":null,"version":null},"distro":{"name":"Ubuntu","version":"24.04","id":"noble","libc":null},"system":{"name":null,"release":null},"cpu":null,"openssl_version":null,"setuptools_version":null,"rustc_version":null,"ci":true}

Release files / tarsk_amqp-0.4.1-cp311-abi3-macosx_10_12_universal2.macosx_10_12_x86_64.macosx_11_0_arm64.whl

Download URL tarsk_amqp-0.4.1-cp311-abi3-macosx_10_12_universal2.macosx_10_12_x86_64.macosx_11_0_arm64.whl
Size 6.5 MB
Tags CPython 3.11 abi3 macOS 10.12+ universal2 (ARM64, x86-64) macOS 10.12+ x86-64 macOS 11.0+ ARM64
SHA-256 checksum
How to use checksums
b538b017c8b1ca5552f084f5e4cfa02f8b35416636c3a7e0c9e00d921a78b533
BLAKE2b-256 checksum
How to use checksums
c3260222c01115ffd3cb8c801cb1e3c7557c62b6c02287a9ab8e25b0563d90a6
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
Yes
Uploaded via uv/0.13.8 {"installer":{"name":"uv","version":"0.13.8","subcommand":["publish"]},"python":null,"implementation":{"name":null,"version":null},"distro":{"name":"Ubuntu","version":"24.04","id":"noble","libc":null},"system":{"name":null,"release":null},"cpu":null,"openssl_version":null,"setuptools_version":null,"rustc_version":null,"ci":true}

Security for tarsk-amqp 0.4.1

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

No provenance found

None of the files in this release were uploaded with provenance metadata.

A note on package security

PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.

To learn more about protecting yourself from malicious actors, visit our package security documentation.

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page