Skip to main content

tarsk

A Python task queue whose workers hold a memory ceiling you set — without losing a task.

task with rust through the middle. The scheduler, retry state machine, lease tracking and child supervision are Rust; the only Python in the hot path is your handler.

PyPI Python License

pip install tarsk            # memory broker only
pip install tarsk-redis      # + Redis Streams
pip install tarsk-postgres   # + Postgres
pip install tarsk-amqp       # + RabbitMQ

Pick one. All four are the same tarsk with a different broker compiled in — same import, same API, same tarsk package — so installing two of them leaves you running whichever one pip unpacked last. tarsk refuses to import when it finds more than one, and a build without the backend your URL asks for names the package you want.

Upgrading from 0.1.x: this is a breaking change. pip install tarsk has shipped Redis, Postgres and AMQP since 0.1. From 0.2.0 it ships none of them — it is the memory-broker build. If your broker URL is anything but memory://, pip uninstall tarsk and install the package for your broker. Nothing in your code changes. See CHANGELOG.md.

Upgrading from 0.2: broker failures now raise tarsk.BrokerError instead of ValueError, and the workers of a fleet are best upgraded together. The four things worth knowing are at the top of CHANGELOG.md.

Status: early. The version badge above reads live from PyPI — a number written here went stale twice before this sentence replaced it. Wheels for Linux (glibc and musl, x86_64 and aarch64), macOS universal2 and Windows x64, on Python 3.11 through 3.14 including the free-threaded build. See What is missing.

📖 Documentation — everything below the fold lives there: how it works, writing tasks, routing and scheduling, operating, benchmarks.

# pip install tarsk-redis
from tarsk import App

app = App(broker="redis://localhost:6379/0")

@app.task(retries=3, timeout=30, queue="heavy")
def embed_document(doc_id: str) -> dict:
    ...

task_id = embed_document.send("abc")
tarsk worker --app myapp:app --broker redis://localhost:6379/0 \
             --queues heavy --children 4 --max-rss 400MB --metrics 0.0.0.0:9090

Handlers must be idempotent. Delivery is at-least-once. This is a contract, not a footnote: a worker killed mid-task will run that task again.

What it does that others do not

child RSS over one hour under a leaky handler, and the supervisor holding it

One hour, 72,001 tasks, a handler that never frees anything. 66 recycles, peak 400 MB against a 400 MB ceiling, no sample over it, nothing killed, nothing lost. The flat line beneath the sawtooth, on the same axis, is the supervisor doing the holding: 28.32 to 28.40 MB across the hour, a drift of 82 KB. A ceiling enforced from inside a process with the same problem would only defer it.

Every Python task queue leaks, because leaks come from the code they run rather than from the queue. The difference is what the runtime does about it.

--max-rss is a byte budget. Celery's --max-tasks-per-child is a task count, which only bounds bytes if you already know how many bytes a task costs — and stays wrong once that changes. Same configuration, different workloads:

leak 20MB/task leak 40MB/task payload-dependent 2–80MB
celery --max-tasks-per-child=6 163 MB 283 MB 328 MB
tarsk --max-rss=200MB 183 MB 183 MB 187 MB

Celery wins the first column, and that is the honest result: when the leak per task is known and constant, dividing the budget by it works — someone divided 200 by 20 and typed 6. It is the other two columns tarsk was built for: nothing was reconfigured between them, the workload moved, and the guess encoded in that 6 went with it.

The worker that runs your code is 28 MB against Celery's 42 MB and taskiq's 47 MB, because it imports your tasks and nothing else — no broker driver, no scheduler. Recycling costs 7 ms at the 99th percentile against Celery's 144 ms, because the replacement child starts before the trigger fires and the slot never goes empty.

What it does not claim

Not faster. With a 50 ms handler tarsk, Celery and taskiq all reach 20 tasks/s. Draining 10,000 no-op tasks across four processes: Celery 7.3s, taskiq 1.8s, tarsk 1.2s on the latest run — and 4% apart on the run before it, on the same kind of machine. Ahead in every run, by an amount that has not held still, so it is not claimed. It starts in half the time and runs in half the memory; that is the claim, and speed is not.

This file claimed a 1.45× lead until the numbers were checked on more than one machine. The full retraction, and every column where tarsk loses, is in the benchmarks.

Not a hard ceiling regardless of task size. The ceiling is read at the dispatch decision, and a handler that allocates 300 MB will allocate it. Overshoot is bounded by the peak of whatever is in flight when the limit is crossed — with --slots 1 that is one task, because the child is idle at the moment it is read; at the default of 100 slots it is up to 100. The budget divides the remaining headroom by the measured per-task cost, so that number is usually far below the slot count, but the only way to bound it at one task is --slots 1.

Not durable execution. That is Temporal's category, and it is much heavier.

What is missing

  • RabbitMQ is the basic backend: jobs, retries, delays, chains and results, and none of the things that need a store every worker can read — cancel, pause, dedup, chords, shared limits
  • Windows runs the suites that need no broker, since neither Redis nor Postgres ships for it. Everything else — the channel, recycling, the memory ceiling — is tested there

Running the tests

The toolchain is pinned in mise.toml — Rust 1.100.0 and Python 3.14.3. With mise installed, mise install fetches both and entering the directory creates .venv from the pinned Python. Without it, any Python 3.11+ and a recent stable Rust will build: the wheel is abi3-py311.

python -m venv .venv && .venv/bin/pip install maturin msgpack
# Every backend in one build: the published wheels take one each, the source
# tree is where all four are tested.
.venv/bin/maturin develop --release -F redis,postgres,amqp,test-hooks

.venv/bin/python tests/test_ipc.py       # protocol, timeouts, retries
.venv/bin/python tests/test_recycle.py   # the ceiling, soft timeouts, middleware
.venv/bin/python tests/test_brokers.py   # all four brokers end to end; name one to run it alone
cargo test --lib                         # cron, console, socket permissions

The broker tests start their own redis-server and Postgres cluster, use any RabbitMQ answering on the conventional ports, and skip whichever is missing — loudly, so a skip cannot pass for a pass. TARSK_REDIS_URL, TARSK_PG_URL and TARSK_AMQP_URL point them at servers you already have instead, which is how a machine without the server binaries still tests those backends. Give each one a database of its own. python demo/run.py --minutes 1 --ceiling 150MB --rate 30 is the definition of done in miniature: it exits non-zero if a task goes missing or the supervisor drifts.

License

MIT — see LICENSE.

Metadata

Release files for tarsk 0.4.0

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for tarsk 0.4.0
File Size Uploaded Provenance Actions
tarsk-0.4.0.tar.gz 175.5 kB Without provenance Details for tarsk-0.4.0.tar.gz

Built distributions (wheels)

Table of built distributions (wheels) for tarsk 0.4.0
File Actions
tarsk-0.4.0-cp314-cp314t-win_amd64.whl CPython 3.14 CPython 3.14 free-threading Windows x86-64 Details for tarsk-0.4.0-cp314-cp314t-win_amd64.whl
tarsk-0.4.0-cp314-cp314t-musllinux_1_2_x86_64.whl CPython 3.14 CPython 3.14 free-threading Linux musl 1.2+ x86-64 Details for tarsk-0.4.0-cp314-cp314t-musllinux_1_2_x86_64.whl
tarsk-0.4.0-cp314-cp314t-manylinux2014_x86_64.manylinux_2_17_x86_64.whl CPython 3.14 CPython 3.14 free-threading Linux glibc 2.17+ x86-64 Details for tarsk-0.4.0-cp314-cp314t-manylinux2014_x86_64.manylinux_2_17_x86_64.whl
tarsk-0.4.0-cp314-cp314t-manylinux2014_aarch64.manylinux_2_17_aarch64.whl CPython 3.14 CPython 3.14 free-threading Linux glibc 2.17+ ARM64 Details for tarsk-0.4.0-cp314-cp314t-manylinux2014_aarch64.manylinux_2_17_aarch64.whl
tarsk-0.4.0-cp314-cp314t-macosx_10_12_universal2.macosx_10_12_x86_64.macosx_11_0_arm64.whl CPython 3.14 CPython 3.14 free-threading macOS 11.0+ ARM64, macOS 10.12+ universal2 (ARM64, x86-64), macOS 10.12+ x86-64 Details for tarsk-0.4.0-cp314-cp314t-macosx_10_12_universal2.macosx_10_12_x86_64.macosx_11_0_arm64.whl
tarsk-0.4.0-cp311-abi3-win_amd64.whl CPython 3.11 abi3 Windows x86-64 Details for tarsk-0.4.0-cp311-abi3-win_amd64.whl
tarsk-0.4.0-cp311-abi3-musllinux_1_2_x86_64.whl CPython 3.11 abi3 Linux musl 1.2+ x86-64 Details for tarsk-0.4.0-cp311-abi3-musllinux_1_2_x86_64.whl
tarsk-0.4.0-cp311-abi3-manylinux2014_x86_64.manylinux_2_17_x86_64.whl CPython 3.11 abi3 Linux glibc 2.17+ x86-64 Details for tarsk-0.4.0-cp311-abi3-manylinux2014_x86_64.manylinux_2_17_x86_64.whl
tarsk-0.4.0-cp311-abi3-manylinux2014_aarch64.manylinux_2_17_aarch64.whl CPython 3.11 abi3 Linux glibc 2.17+ ARM64 Details for tarsk-0.4.0-cp311-abi3-manylinux2014_aarch64.manylinux_2_17_aarch64.whl
tarsk-0.4.0-cp311-abi3-macosx_10_12_universal2.macosx_10_12_x86_64.macosx_11_0_arm64.whl CPython 3.11 abi3 macOS 11.0+ ARM64, macOS 10.12+ x86-64, macOS 10.12+ universal2 (ARM64, x86-64) Details for tarsk-0.4.0-cp311-abi3-macosx_10_12_universal2.macosx_10_12_x86_64.macosx_11_0_arm64.whl

Total release size: 12.9 MB

Release files / tarsk-0.4.0.tar.gz

Download URL tarsk-0.4.0.tar.gz
Size 175.5 kB
Tags Source
SHA-256 checksum
How to use checksums
2ef759697ef741481a20388a8d0eb06b77553259e5fd1b7e1b3eaedaef2edf64
BLAKE2b-256 checksum
How to use checksums
8a8dd17dfdd237a1887c07bbb81a59594ca3c83c8de073312d2e26bb39cafdab
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
Yes
Uploaded via uv/0.13.8 {"installer":{"name":"uv","version":"0.13.8","subcommand":["publish"]},"python":null,"implementation":{"name":null,"version":null},"distro":{"name":"Ubuntu","version":"24.04","id":"noble","libc":null},"system":{"name":null,"release":null},"cpu":null,"openssl_version":null,"setuptools_version":null,"rustc_version":null,"ci":true}

Release files / tarsk-0.4.0-cp314-cp314t-win_amd64.whl

Download URL tarsk-0.4.0-cp314-cp314t-win_amd64.whl
Size 925.6 kB
Tags CPython 3.14 CPython 3.14 free-threading Windows x86-64
SHA-256 checksum
How to use checksums
e170c0642c910b727f8917a8bf1d6f067d41cd64d1fe53e43191b6d2c9c2c01e
BLAKE2b-256 checksum
How to use checksums
1cfe0fa0429f08e77726e1d047d4992ee07c94e889ecc7d3990d809bb0127e05
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
Yes
Uploaded via uv/0.13.8 {"installer":{"name":"uv","version":"0.13.8","subcommand":["publish"]},"python":null,"implementation":{"name":null,"version":null},"distro":{"name":"Ubuntu","version":"24.04","id":"noble","libc":null},"system":{"name":null,"release":null},"cpu":null,"openssl_version":null,"setuptools_version":null,"rustc_version":null,"ci":true}

Release files / tarsk-0.4.0-cp314-cp314t-musllinux_1_2_x86_64.whl

Download URL tarsk-0.4.0-cp314-cp314t-musllinux_1_2_x86_64.whl
Size 1.4 MB
Tags CPython 3.14 CPython 3.14 free-threading Linux musl 1.2+ x86-64
SHA-256 checksum
How to use checksums
b36c055b12f73cc2ea6caab0a574f564e5260e659f23bbb25cfeab63697bfc7b
BLAKE2b-256 checksum
How to use checksums
dc497d1f44f8a5f4a9975f2188bf3946c4e62bfe9931e327694d382ea4c7317f
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
Yes
Uploaded via uv/0.13.8 {"installer":{"name":"uv","version":"0.13.8","subcommand":["publish"]},"python":null,"implementation":{"name":null,"version":null},"distro":{"name":"Ubuntu","version":"24.04","id":"noble","libc":null},"system":{"name":null,"release":null},"cpu":null,"openssl_version":null,"setuptools_version":null,"rustc_version":null,"ci":true}

Release files / tarsk-0.4.0-cp314-cp314t-manylinux2014_x86_64.manylinux_2_17_x86_64.whl

Download URL tarsk-0.4.0-cp314-cp314t-manylinux2014_x86_64.manylinux_2_17_x86_64.whl
Size 1.1 MB
Tags CPython 3.14 CPython 3.14 free-threading Linux glibc 2.17+ x86-64
SHA-256 checksum
How to use checksums
df363b10c39dfa0097ff6ac48b6df7456d479413f958f0dad1480eeb03dcc988
BLAKE2b-256 checksum
How to use checksums
d4d1eea971249e50fce25018e544e768fe2265043e579fa593ccbfc0bce4d4d1
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
Yes
Uploaded via uv/0.13.8 {"installer":{"name":"uv","version":"0.13.8","subcommand":["publish"]},"python":null,"implementation":{"name":null,"version":null},"distro":{"name":"Ubuntu","version":"24.04","id":"noble","libc":null},"system":{"name":null,"release":null},"cpu":null,"openssl_version":null,"setuptools_version":null,"rustc_version":null,"ci":true}

Release files / tarsk-0.4.0-cp314-cp314t-manylinux2014_aarch64.manylinux_2_17_aarch64.whl

Download URL tarsk-0.4.0-cp314-cp314t-manylinux2014_aarch64.manylinux_2_17_aarch64.whl
Size 972.4 kB
Tags CPython 3.14 CPython 3.14 free-threading Linux glibc 2.17+ ARM64
SHA-256 checksum
How to use checksums
52b10f9bee99dd1b7a66f83e5919560ff985b2035b6edd93dbf16a0a0aa57db3
BLAKE2b-256 checksum
How to use checksums
c150de9693a01b32f725861c9da3042fe58f76eb957b205cecb5c7c57532c90f
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
Yes
Uploaded via uv/0.13.8 {"installer":{"name":"uv","version":"0.13.8","subcommand":["publish"]},"python":null,"implementation":{"name":null,"version":null},"distro":{"name":"Ubuntu","version":"24.04","id":"noble","libc":null},"system":{"name":null,"release":null},"cpu":null,"openssl_version":null,"setuptools_version":null,"rustc_version":null,"ci":true}

Release files / tarsk-0.4.0-cp314-cp314t-macosx_10_12_universal2.macosx_10_12_x86_64.macosx_11_0_arm64.whl

Download URL tarsk-0.4.0-cp314-cp314t-macosx_10_12_universal2.macosx_10_12_x86_64.macosx_11_0_arm64.whl
Size 2.0 MB
Tags CPython 3.14 CPython 3.14 free-threading macOS 10.12+ universal2 (ARM64, x86-64) macOS 10.12+ x86-64 macOS 11.0+ ARM64
SHA-256 checksum
How to use checksums
a0efc1afb2439fefbff0d41e4e44e34f6f2b7ff9e3b4dae046cfa9366d9eb129
BLAKE2b-256 checksum
How to use checksums
50eb9e86128d52bb62e4321d2358e1901b00519de66668e24c581e54f351ed77
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
Yes
Uploaded via uv/0.13.8 {"installer":{"name":"uv","version":"0.13.8","subcommand":["publish"]},"python":null,"implementation":{"name":null,"version":null},"distro":{"name":"Ubuntu","version":"24.04","id":"noble","libc":null},"system":{"name":null,"release":null},"cpu":null,"openssl_version":null,"setuptools_version":null,"rustc_version":null,"ci":true}

Release files / tarsk-0.4.0-cp311-abi3-win_amd64.whl

Download URL tarsk-0.4.0-cp311-abi3-win_amd64.whl
Size 930.4 kB
Tags CPython 3.11 Windows x86-64 abi3
SHA-256 checksum
How to use checksums
c845fc6ac22423fc867cf3d53f7ead367436679dc916673460cc3829673c2a67
BLAKE2b-256 checksum
How to use checksums
f94586edd3d3aba717c9c59b6a1152696b1978b39f96149e9f82aec31e2ab955
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
Yes
Uploaded via uv/0.13.8 {"installer":{"name":"uv","version":"0.13.8","subcommand":["publish"]},"python":null,"implementation":{"name":null,"version":null},"distro":{"name":"Ubuntu","version":"24.04","id":"noble","libc":null},"system":{"name":null,"release":null},"cpu":null,"openssl_version":null,"setuptools_version":null,"rustc_version":null,"ci":true}

Release files / tarsk-0.4.0-cp311-abi3-musllinux_1_2_x86_64.whl

Download URL tarsk-0.4.0-cp311-abi3-musllinux_1_2_x86_64.whl
Size 1.4 MB
Tags CPython 3.11 Linux musl 1.2+ x86-64 abi3
SHA-256 checksum
How to use checksums
eb43cd0e60ac8598ac8abc8a5ad89d0b0c56988b7fffb8f2d29a04c17d94081d
BLAKE2b-256 checksum
How to use checksums
feb23c47dcacd449171a049470fa3248aa2815ec1f0f520e76eec8d040448e7a
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
Yes
Uploaded via uv/0.13.8 {"installer":{"name":"uv","version":"0.13.8","subcommand":["publish"]},"python":null,"implementation":{"name":null,"version":null},"distro":{"name":"Ubuntu","version":"24.04","id":"noble","libc":null},"system":{"name":null,"release":null},"cpu":null,"openssl_version":null,"setuptools_version":null,"rustc_version":null,"ci":true}

Release files / tarsk-0.4.0-cp311-abi3-manylinux2014_x86_64.manylinux_2_17_x86_64.whl

Download URL tarsk-0.4.0-cp311-abi3-manylinux2014_x86_64.manylinux_2_17_x86_64.whl
Size 1.2 MB
Tags CPython 3.11 Linux glibc 2.17+ x86-64 abi3
SHA-256 checksum
How to use checksums
31fa0877bb2e41e998d2c56edd51b80f007201a4cb0f80450a3793a6d27dbb8e
BLAKE2b-256 checksum
How to use checksums
a20fd495ca121d81b62bad5077e20e8d12a3b654d043afc0f1d7ee1be3ccd7ec
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
Yes
Uploaded via uv/0.13.8 {"installer":{"name":"uv","version":"0.13.8","subcommand":["publish"]},"python":null,"implementation":{"name":null,"version":null},"distro":{"name":"Ubuntu","version":"24.04","id":"noble","libc":null},"system":{"name":null,"release":null},"cpu":null,"openssl_version":null,"setuptools_version":null,"rustc_version":null,"ci":true}

Release files / tarsk-0.4.0-cp311-abi3-manylinux2014_aarch64.manylinux_2_17_aarch64.whl

Download URL tarsk-0.4.0-cp311-abi3-manylinux2014_aarch64.manylinux_2_17_aarch64.whl
Size 978.9 kB
Tags CPython 3.11 Linux glibc 2.17+ ARM64 abi3
SHA-256 checksum
How to use checksums
d57908da33ebd8e2688920a55e9a43f95f45c2f4f02cd192773fb65cc7083e08
BLAKE2b-256 checksum
How to use checksums
2c0969715a1455b686f7208ed5967715313437f76751d4a57772ab657c28a881
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
Yes
Uploaded via uv/0.13.8 {"installer":{"name":"uv","version":"0.13.8","subcommand":["publish"]},"python":null,"implementation":{"name":null,"version":null},"distro":{"name":"Ubuntu","version":"24.04","id":"noble","libc":null},"system":{"name":null,"release":null},"cpu":null,"openssl_version":null,"setuptools_version":null,"rustc_version":null,"ci":true}

Release files / tarsk-0.4.0-cp311-abi3-macosx_10_12_universal2.macosx_10_12_x86_64.macosx_11_0_arm64.whl

Download URL tarsk-0.4.0-cp311-abi3-macosx_10_12_universal2.macosx_10_12_x86_64.macosx_11_0_arm64.whl
Size 2.0 MB
Tags CPython 3.11 abi3 macOS 10.12+ universal2 (ARM64, x86-64) macOS 10.12+ x86-64 macOS 11.0+ ARM64
SHA-256 checksum
How to use checksums
5e09aab413d80e38f5aa814250e576303281910f92e3f9d3004b20f633692e02
BLAKE2b-256 checksum
How to use checksums
ea07e16122737d6dc2d0fb2a4a5b0953de855e134f4653c1794ffd428a7d4a3d
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
Yes
Uploaded via uv/0.13.8 {"installer":{"name":"uv","version":"0.13.8","subcommand":["publish"]},"python":null,"implementation":{"name":null,"version":null},"distro":{"name":"Ubuntu","version":"24.04","id":"noble","libc":null},"system":{"name":null,"release":null},"cpu":null,"openssl_version":null,"setuptools_version":null,"rustc_version":null,"ci":true}

Security for tarsk 0.4.0

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

No provenance found

None of the files in this release were uploaded with provenance metadata.

A note on package security

PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.

To learn more about protecting yourself from malicious actors, visit our package security documentation.

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page