Skip to main content

tarsk

A Python task queue whose workers hold a memory ceiling you set — without losing a task.

task with rust through the middle. The scheduler, retry state machine, lease tracking and child supervision are Rust; the only Python in the hot path is your handler.

PyPI Python License

pip install tarsk            # memory broker only
pip install tarsk-redis      # + Redis Streams
pip install tarsk-postgres   # + Postgres
pip install tarsk-amqp       # + RabbitMQ

Pick one. All four are the same tarsk with a different broker compiled in — same import, same API, same tarsk package — so installing two of them leaves you running whichever one pip unpacked last. tarsk refuses to import when it finds more than one, and a build without the backend your URL asks for names the package you want.

Upgrading from 0.1.x: this is a breaking change. pip install tarsk has shipped Redis, Postgres and AMQP since 0.1. From 0.2.0 it ships none of them — it is the memory-broker build. If your broker URL is anything but memory://, pip uninstall tarsk and install the package for your broker. Nothing in your code changes. See CHANGELOG.md.

Upgrading from 0.2: broker failures now raise tarsk.BrokerError instead of ValueError, and the workers of a fleet are best upgraded together. The four things worth knowing are at the top of CHANGELOG.md.

Status: early. The version badge above reads live from PyPI — a number written here went stale twice before this sentence replaced it. Wheels for Linux (glibc and musl, x86_64 and aarch64), macOS universal2 and Windows x64, on Python 3.11 through 3.14 including the free-threaded build. See What is missing.

📖 Documentation — everything below the fold lives there: how it works, writing tasks, routing and scheduling, operating, benchmarks.

# pip install tarsk-redis
from tarsk import App

app = App(broker="redis://localhost:6379/0")

@app.task(retries=3, timeout=30, queue="heavy")
def embed_document(doc_id: str) -> dict:
    ...

task_id = embed_document.send("abc")
tarsk worker --app myapp:app --broker redis://localhost:6379/0 \
             --queues heavy --children 4 --max-rss 400MB --metrics 0.0.0.0:9090

Handlers must be idempotent. Delivery is at-least-once. This is a contract, not a footnote: a worker killed mid-task will run that task again.

What it does that others do not

child RSS over one hour under a leaky handler, and the supervisor holding it

One hour, 72,001 tasks, a handler that never frees anything. 66 recycles, peak 400 MB against a 400 MB ceiling, no sample over it, nothing killed, nothing lost. The flat line beneath the sawtooth, on the same axis, is the supervisor doing the holding: 28.32 to 28.40 MB across the hour, a drift of 82 KB. A ceiling enforced from inside a process with the same problem would only defer it.

Every Python task queue leaks, because leaks come from the code they run rather than from the queue. The difference is what the runtime does about it.

--max-rss is a byte budget. Celery's --max-tasks-per-child is a task count, which only bounds bytes if you already know how many bytes a task costs — and stays wrong once that changes. Same configuration, different workloads:

leak 20MB/task leak 40MB/task payload-dependent 2–80MB
celery --max-tasks-per-child=6 162 MB 282 MB 327 MB
tarsk --max-rss=200MB 183 MB 183 MB 187 MB

Celery wins the first column, and that is the honest result: when the leak per task is known and constant, dividing the budget by it works — someone divided 200 by 20 and typed 6. It is the other two columns tarsk was built for: nothing was reconfigured between them, the workload moved, and the guess encoded in that 6 went with it.

The worker that runs your code is 27 MB against Celery's 41 MB and taskiq's 44 MB, because it imports your tasks and nothing else — no broker driver, no scheduler. Recycling costs 7 ms at the 99th percentile against Celery's 139 ms, because the replacement child starts before the trigger fires and the slot never goes empty.

What it does not claim

Not faster. With a 50 ms handler tarsk, Celery and taskiq all reach 19–20 tasks/s. Draining 10,000 no-op tasks across four processes: Celery 10.0s, taskiq 2.3s, tarsk 2.2s — ahead in every run and by 4%, which this project's own benchmark rules call a tie. It starts in half the time and runs in half the memory; that is the claim, and speed is not.

This file claimed a 1.45× lead until the numbers were checked on more than one machine. The full retraction, and every column where tarsk loses, is in the benchmarks.

Not a hard ceiling regardless of task size. The ceiling is read at the dispatch decision, and a handler that allocates 300 MB will allocate it. Overshoot is bounded by the peak of whatever is in flight when the limit is crossed — with --slots 1 that is one task, because the child is idle at the moment it is read; at the default of 100 slots it is up to 100. The budget divides the remaining headroom by the measured per-task cost, so that number is usually far below the slot count, but the only way to bound it at one task is --slots 1.

Not durable execution. That is Temporal's category, and it is much heavier.

What is missing

  • No chord. chain and group are here; fanning back in to a callback is not
  • Windows runs the suites that need no broker, since neither Redis nor Postgres ships for it. Everything else — the channel, recycling, the memory ceiling — is tested there

Running the tests

The toolchain is pinned in mise.toml — Rust 1.94.0 and Python 3.14.3, the versions the published numbers were produced with. With mise installed, mise install fetches both and entering the directory creates .venv from the pinned Python. Without it, any Python 3.11+ and a recent stable Rust will build: the wheel is abi3-py311.

python -m venv .venv && .venv/bin/pip install maturin msgpack
# Every backend in one build: the published wheels take one each, the source
# tree is where all four are tested.
.venv/bin/maturin develop --release -F redis,postgres,amqp

.venv/bin/python tests/test_ipc.py       # protocol, timeouts, retries
.venv/bin/python tests/test_recycle.py   # the ceiling, soft timeouts, middleware
.venv/bin/python tests/test_brokers.py   # all four brokers end to end
cargo test --lib                         # cron, console, socket permissions

The broker tests start their own redis-server and Postgres cluster, use any RabbitMQ answering on the conventional ports, and skip whichever is missing — loudly, so a skip cannot pass for a pass. TARSK_REDIS_URL, TARSK_PG_URL and TARSK_AMQP_URL point them at servers you already have instead, which is how a machine without the server binaries still tests those backends. Give each one a database of its own. python demo/run.py --minutes 1 --ceiling 150MB --rate 30 is the definition of done in miniature: it exits non-zero if a task goes missing or the supervisor drifts.

License

MIT — see LICENSE.

Metadata

Release files for tarsk-amqp 0.3.0

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for tarsk-amqp 0.3.0
File Size Uploaded Provenance Actions
tarsk_amqp-0.3.0.tar.gz 154.2 kB Without provenance Details for tarsk_amqp-0.3.0.tar.gz

Built distributions (wheels)

Table of built distributions (wheels) for tarsk-amqp 0.3.0
File Actions
tarsk_amqp-0.3.0-cp314-cp314t-win_amd64.whl CPython 3.14 CPython 3.14 free-threading Windows x86-64 Details for tarsk_amqp-0.3.0-cp314-cp314t-win_amd64.whl
tarsk_amqp-0.3.0-cp314-cp314t-musllinux_1_2_x86_64.whl CPython 3.14 CPython 3.14 free-threading Linux musl 1.2+ x86-64 Details for tarsk_amqp-0.3.0-cp314-cp314t-musllinux_1_2_x86_64.whl
tarsk_amqp-0.3.0-cp314-cp314t-manylinux2014_x86_64.manylinux_2_17_x86_64.whl CPython 3.14 CPython 3.14 free-threading Linux glibc 2.17+ x86-64 Details for tarsk_amqp-0.3.0-cp314-cp314t-manylinux2014_x86_64.manylinux_2_17_x86_64.whl
tarsk_amqp-0.3.0-cp314-cp314t-manylinux2014_aarch64.manylinux_2_17_aarch64.whl CPython 3.14 CPython 3.14 free-threading Linux glibc 2.17+ ARM64 Details for tarsk_amqp-0.3.0-cp314-cp314t-manylinux2014_aarch64.manylinux_2_17_aarch64.whl
tarsk_amqp-0.3.0-cp314-cp314t-macosx_10_12_universal2.macosx_10_12_x86_64.macosx_11_0_arm64.whl CPython 3.14 CPython 3.14 free-threading macOS 11.0+ ARM64, macOS 10.12+ x86-64, macOS 10.12+ universal2 (ARM64, x86-64) Details for tarsk_amqp-0.3.0-cp314-cp314t-macosx_10_12_universal2.macosx_10_12_x86_64.macosx_11_0_arm64.whl
tarsk_amqp-0.3.0-cp311-abi3-win_amd64.whl CPython 3.11 abi3 Windows x86-64 Details for tarsk_amqp-0.3.0-cp311-abi3-win_amd64.whl
tarsk_amqp-0.3.0-cp311-abi3-musllinux_1_2_x86_64.whl CPython 3.11 abi3 Linux musl 1.2+ x86-64 Details for tarsk_amqp-0.3.0-cp311-abi3-musllinux_1_2_x86_64.whl
tarsk_amqp-0.3.0-cp311-abi3-manylinux2014_x86_64.manylinux_2_17_x86_64.whl CPython 3.11 abi3 Linux glibc 2.17+ x86-64 Details for tarsk_amqp-0.3.0-cp311-abi3-manylinux2014_x86_64.manylinux_2_17_x86_64.whl
tarsk_amqp-0.3.0-cp311-abi3-manylinux2014_aarch64.manylinux_2_17_aarch64.whl CPython 3.11 abi3 Linux glibc 2.17+ ARM64 Details for tarsk_amqp-0.3.0-cp311-abi3-manylinux2014_aarch64.manylinux_2_17_aarch64.whl
tarsk_amqp-0.3.0-cp311-abi3-macosx_10_12_universal2.macosx_10_12_x86_64.macosx_11_0_arm64.whl CPython 3.11 abi3 macOS 11.0+ ARM64, macOS 10.12+ x86-64, macOS 10.12+ universal2 (ARM64, x86-64) Details for tarsk_amqp-0.3.0-cp311-abi3-macosx_10_12_universal2.macosx_10_12_x86_64.macosx_11_0_arm64.whl

Total release size: 39.6 MB

Release files / tarsk_amqp-0.3.0.tar.gz

Download URL tarsk_amqp-0.3.0.tar.gz
Size 154.2 kB
Tags Source
SHA-256 checksum
How to use checksums
ae68cdec197a238f3568b1a106bf3822332a9c7e87d2c215fb3bd7ee15a0960b
BLAKE2b-256 checksum
How to use checksums
8463305019ec9cc4a867a2f7d21b97869ed2b885e4e2f859c813df5ec8905b35
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
Yes
Uploaded via uv/0.13.8 {"installer":{"name":"uv","version":"0.13.8","subcommand":["publish"]},"python":null,"implementation":{"name":null,"version":null},"distro":{"name":"Ubuntu","version":"24.04","id":"noble","libc":null},"system":{"name":null,"release":null},"cpu":null,"openssl_version":null,"setuptools_version":null,"rustc_version":null,"ci":true}

Release files / tarsk_amqp-0.3.0-cp314-cp314t-win_amd64.whl

Download URL tarsk_amqp-0.3.0-cp314-cp314t-win_amd64.whl
Size 3.1 MB
Tags CPython 3.14 CPython 3.14 free-threading Windows x86-64
SHA-256 checksum
How to use checksums
edcfd3a605a87b9c8f00a034aa3a5561aab1e90170f05bf4640a5c9a6a05501b
BLAKE2b-256 checksum
How to use checksums
5defe7c2beac5b8def03b583f09d1fa35cc8dcd4827771a6b90b3a5224dad186
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
Yes
Uploaded via uv/0.13.8 {"installer":{"name":"uv","version":"0.13.8","subcommand":["publish"]},"python":null,"implementation":{"name":null,"version":null},"distro":{"name":"Ubuntu","version":"24.04","id":"noble","libc":null},"system":{"name":null,"release":null},"cpu":null,"openssl_version":null,"setuptools_version":null,"rustc_version":null,"ci":true}

Release files / tarsk_amqp-0.3.0-cp314-cp314t-musllinux_1_2_x86_64.whl

Download URL tarsk_amqp-0.3.0-cp314-cp314t-musllinux_1_2_x86_64.whl
Size 3.8 MB
Tags CPython 3.14 CPython 3.14 free-threading Linux musl 1.2+ x86-64
SHA-256 checksum
How to use checksums
f3e7bc31da2c450aa246f8f6ec562f7406b9aa565d6b6cb0c7b184c84b2ad3c8
BLAKE2b-256 checksum
How to use checksums
f8da09a6d260538a3da3bd606e06e2fca67509af8e02ea6e3bc921fb730e7b8d
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
Yes
Uploaded via uv/0.13.8 {"installer":{"name":"uv","version":"0.13.8","subcommand":["publish"]},"python":null,"implementation":{"name":null,"version":null},"distro":{"name":"Ubuntu","version":"24.04","id":"noble","libc":null},"system":{"name":null,"release":null},"cpu":null,"openssl_version":null,"setuptools_version":null,"rustc_version":null,"ci":true}

Release files / tarsk_amqp-0.3.0-cp314-cp314t-manylinux2014_x86_64.manylinux_2_17_x86_64.whl

Download URL tarsk_amqp-0.3.0-cp314-cp314t-manylinux2014_x86_64.manylinux_2_17_x86_64.whl
Size 3.5 MB
Tags CPython 3.14 CPython 3.14 free-threading Linux glibc 2.17+ x86-64
SHA-256 checksum
How to use checksums
9c7f6146d7d2a953bd7ba09d6659fb1cc85c13d2c8601a3dbfd44203cba767c7
BLAKE2b-256 checksum
How to use checksums
4aeb3965e304d54122748da258f09b9d8305c0d7064940f0b7bc21fe4e060b6b
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
Yes
Uploaded via uv/0.13.8 {"installer":{"name":"uv","version":"0.13.8","subcommand":["publish"]},"python":null,"implementation":{"name":null,"version":null},"distro":{"name":"Ubuntu","version":"24.04","id":"noble","libc":null},"system":{"name":null,"release":null},"cpu":null,"openssl_version":null,"setuptools_version":null,"rustc_version":null,"ci":true}

Release files / tarsk_amqp-0.3.0-cp314-cp314t-manylinux2014_aarch64.manylinux_2_17_aarch64.whl

Download URL tarsk_amqp-0.3.0-cp314-cp314t-manylinux2014_aarch64.manylinux_2_17_aarch64.whl
Size 3.0 MB
Tags CPython 3.14 CPython 3.14 free-threading Linux glibc 2.17+ ARM64
SHA-256 checksum
How to use checksums
70834833aa5faa79e919948e8842d1516b6ce10bbe68a06667f0be57c1ab4fe6
BLAKE2b-256 checksum
How to use checksums
490ef190141315caad9a0f016712f3bb37096b8ee20d825c47eaae04f0011ad5
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
Yes
Uploaded via uv/0.13.8 {"installer":{"name":"uv","version":"0.13.8","subcommand":["publish"]},"python":null,"implementation":{"name":null,"version":null},"distro":{"name":"Ubuntu","version":"24.04","id":"noble","libc":null},"system":{"name":null,"release":null},"cpu":null,"openssl_version":null,"setuptools_version":null,"rustc_version":null,"ci":true}

Release files / tarsk_amqp-0.3.0-cp314-cp314t-macosx_10_12_universal2.macosx_10_12_x86_64.macosx_11_0_arm64.whl

Download URL tarsk_amqp-0.3.0-cp314-cp314t-macosx_10_12_universal2.macosx_10_12_x86_64.macosx_11_0_arm64.whl
Size 6.3 MB
Tags CPython 3.14 CPython 3.14 free-threading macOS 10.12+ universal2 (ARM64, x86-64) macOS 10.12+ x86-64 macOS 11.0+ ARM64
SHA-256 checksum
How to use checksums
598885889be476a4227407cd59b8f09d9654f57fa5c37f143d179419a0bc5ee0
BLAKE2b-256 checksum
How to use checksums
25961299e0eeb885bb1efc6910fed8f7969aeb85748ba86f39a047ef3d8854cd
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
Yes
Uploaded via uv/0.13.8 {"installer":{"name":"uv","version":"0.13.8","subcommand":["publish"]},"python":null,"implementation":{"name":null,"version":null},"distro":{"name":"Ubuntu","version":"24.04","id":"noble","libc":null},"system":{"name":null,"release":null},"cpu":null,"openssl_version":null,"setuptools_version":null,"rustc_version":null,"ci":true}

Release files / tarsk_amqp-0.3.0-cp311-abi3-win_amd64.whl

Download URL tarsk_amqp-0.3.0-cp311-abi3-win_amd64.whl
Size 3.1 MB
Tags CPython 3.11 Windows x86-64 abi3
SHA-256 checksum
How to use checksums
021b4def8a81d1032059bf70ec1c803646f5f64dc06660ccdd239c0e224ab0d6
BLAKE2b-256 checksum
How to use checksums
3c306234553a4afb0aaf122556f490c5f7bc73c5346e5ccddc45d4995ee1f882
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
Yes
Uploaded via uv/0.13.8 {"installer":{"name":"uv","version":"0.13.8","subcommand":["publish"]},"python":null,"implementation":{"name":null,"version":null},"distro":{"name":"Ubuntu","version":"24.04","id":"noble","libc":null},"system":{"name":null,"release":null},"cpu":null,"openssl_version":null,"setuptools_version":null,"rustc_version":null,"ci":true}

Release files / tarsk_amqp-0.3.0-cp311-abi3-musllinux_1_2_x86_64.whl

Download URL tarsk_amqp-0.3.0-cp311-abi3-musllinux_1_2_x86_64.whl
Size 3.8 MB
Tags CPython 3.11 Linux musl 1.2+ x86-64 abi3
SHA-256 checksum
How to use checksums
b7c8ffab8aa324c4e0251981e03168fc00ef537cf5fede86e16295ad58699df4
BLAKE2b-256 checksum
How to use checksums
c7b4745adefd4d37ab3609df60a3b85b77626d43e857471a1738373f87fe8df0
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
Yes
Uploaded via uv/0.13.8 {"installer":{"name":"uv","version":"0.13.8","subcommand":["publish"]},"python":null,"implementation":{"name":null,"version":null},"distro":{"name":"Ubuntu","version":"24.04","id":"noble","libc":null},"system":{"name":null,"release":null},"cpu":null,"openssl_version":null,"setuptools_version":null,"rustc_version":null,"ci":true}

Release files / tarsk_amqp-0.3.0-cp311-abi3-manylinux2014_x86_64.manylinux_2_17_x86_64.whl

Download URL tarsk_amqp-0.3.0-cp311-abi3-manylinux2014_x86_64.manylinux_2_17_x86_64.whl
Size 3.5 MB
Tags CPython 3.11 Linux glibc 2.17+ x86-64 abi3
SHA-256 checksum
How to use checksums
7816b74bb7e9805af97d751bf9de023f8481636b044a2cced435c0bbed36773a
BLAKE2b-256 checksum
How to use checksums
5c5e06c5c2f34b2be69abf7e1422a875cfd811518ba32acf8c5f8c4d2e03005c
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
Yes
Uploaded via uv/0.13.8 {"installer":{"name":"uv","version":"0.13.8","subcommand":["publish"]},"python":null,"implementation":{"name":null,"version":null},"distro":{"name":"Ubuntu","version":"24.04","id":"noble","libc":null},"system":{"name":null,"release":null},"cpu":null,"openssl_version":null,"setuptools_version":null,"rustc_version":null,"ci":true}

Release files / tarsk_amqp-0.3.0-cp311-abi3-manylinux2014_aarch64.manylinux_2_17_aarch64.whl

Download URL tarsk_amqp-0.3.0-cp311-abi3-manylinux2014_aarch64.manylinux_2_17_aarch64.whl
Size 3.0 MB
Tags CPython 3.11 Linux glibc 2.17+ ARM64 abi3
SHA-256 checksum
How to use checksums
7d2521df782e22011ad6e92fcfea69a4b57cb9e5299eed8be8e89d4bff2c3a68
BLAKE2b-256 checksum
How to use checksums
67f17afa83c40562b0ebd2a84bd5ee486883d5c0c15d04f93bad82dfda3ce06f
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
Yes
Uploaded via uv/0.13.8 {"installer":{"name":"uv","version":"0.13.8","subcommand":["publish"]},"python":null,"implementation":{"name":null,"version":null},"distro":{"name":"Ubuntu","version":"24.04","id":"noble","libc":null},"system":{"name":null,"release":null},"cpu":null,"openssl_version":null,"setuptools_version":null,"rustc_version":null,"ci":true}

Release files / tarsk_amqp-0.3.0-cp311-abi3-macosx_10_12_universal2.macosx_10_12_x86_64.macosx_11_0_arm64.whl

Download URL tarsk_amqp-0.3.0-cp311-abi3-macosx_10_12_universal2.macosx_10_12_x86_64.macosx_11_0_arm64.whl
Size 6.3 MB
Tags CPython 3.11 abi3 macOS 10.12+ universal2 (ARM64, x86-64) macOS 10.12+ x86-64 macOS 11.0+ ARM64
SHA-256 checksum
How to use checksums
6c4c77def6ff97f3ff03dddb4604affc5b1b7f702c49d2e70433f00be0318c23
BLAKE2b-256 checksum
How to use checksums
9e0d29c438d67c06c7ac72e3b92c2c879c11ea37a1c484bfcfc8a9cba01b167a
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
Yes
Uploaded via uv/0.13.8 {"installer":{"name":"uv","version":"0.13.8","subcommand":["publish"]},"python":null,"implementation":{"name":null,"version":null},"distro":{"name":"Ubuntu","version":"24.04","id":"noble","libc":null},"system":{"name":null,"release":null},"cpu":null,"openssl_version":null,"setuptools_version":null,"rustc_version":null,"ci":true}

Security for tarsk-amqp 0.3.0

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

No provenance found

None of the files in this release were uploaded with provenance metadata.

A note on package security

PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.

To learn more about protecting yourself from malicious actors, visit our package security documentation.

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page