Skip to main content

straw

中文版

Filesystem-based durable queues and shared tensor storage for AI applications.

straw uses a shared filesystem as its storage layer: workers on multiple machines read and write packed data through the same mounted filesystem and exchange small references. Storage and queue protocols run in Rust; Python APIs accept bytes, NumPy arrays and PyTorch tensors.

The current implementation and multi-machine validation focus on JuiceFS shared mounts, accessed through Linux/POSIX filesystem operations. Local filesystems are also used for development and single-machine tests. Support for additional network filesystems, including NFS deployments, is planned; each needs validation of its locking, visibility and durability semantics.

straw currently serves the rollout and training pipelines in slime, including rollout-to-rollout and rollout-to-training queues that share tensor storage. It is an independent AI application library, also suited to preprocessing and inference pipelines. See the design rationale and application integration.

Multi-machine parallel reads and writes

Multiple producers append concurrently to their own packs; readers on other machines access committed immutable extents by reference. The protocol makes payloads durable before publishing references, journals task ownership, accepted results and consumer progress, and coordinates shared ownership and GC through cross-client catalog locks. Reader pins and explicit completion signals protect data until every consumer has finished. See the concurrency rules and protocol.

Avoid the small-file bottleneck

Creating a file for every sample or tensor puts pressure on shared-filesystem metadata operations, especially on NFS. straw packs many records and their manifests into large append-only files, batches publication, and uses fixed append logs for queue metadata. There is no file per sample or tensor. File counts grow with packs and writer lifetimes instead of individual samples; packing and GC control their lifecycle.

Status: early release, Linux/POSIX. One externally fenced coordinator per queue; no automatic leader election or journal/live-pack compaction. Applications must provide real reader-completion signals before reclamation. See filesystem requirements and protocol guarantees.

Install

Install from PyPI:

pip install straw-queue

pip selects a compatible wheel containing the Rust extension. Installing a wheel does not require Rust or maturin. NumPy and PyTorch are runtime dependencies; an existing compatible PyTorch installation can be reused. Distribution name: straw-queue; import: straw. See supported platforms, offline installation and source builds.

Write, read, keep, reclaim

This complete example uses a private temporary directory. Use a new directory on your shared mount for a real job, with the same run ID on every client.

from tempfile import TemporaryDirectory
from straw import Record, SharedFilesystemStore

with TemporaryDirectory() as root:
    with SharedFilesystemStore(root, "example", online_gc=True) as store:
        ref = store.publish([Record("message", b"hello")], submission_id="write-1")
        store.retain("application:checkpoint-1", [ref])
        store.release_publications([ref])  # The checkpoint now owns this data.
        assert next(store.read(ref)).payload == b"hello"

        store.seal()  # Stop appending to this pack before it can be collected.
        assert store.collect_garbage()["reclaimed_files"] == 0
        store.release("application:checkpoint-1")
        assert store.collect_garbage()["reclaimed_files"] == 1

A returned reference is an address, not a permanent lifetime guarantee. Publication staging initially protects it. A queue, checkpoint, explicit owner or reader pin must protect it while it is in use. Garbage collection removes only sealed packs with no remaining owners; closing a writer does not release application ownership. See data and lifetime APIs.

What can I store?

Data Write Read
Opaque bytes, encoded text, JSON, images or application formats Record + store.publish store.read / read_record, then your decoder
NumPy arrays and PyTorch tensors publish_tensors TensorRef.load() or contiguous row slices
Several records or tensors forming one result publish / publish_many One RecordSetRef addresses the ordered records
Shared immutable data used by several queues Dependencies and TensorRef.share Each queue retains the same underlying packs
A changed tensor TensorRef.updated A new tensor; the original remains unchanged

straw never unpickles stored payloads. Codec names are explicit and versioned; applications own serialization of their custom objects. Tensor helpers copy device data to contiguous CPU storage and support checked row reads. They are not GPU IPC or page-level copy-on-write. Supported types and examples.

Applications may use multiple processes/machines or threads. straw exposes synchronous Rust-backed calls and does not start an I/O process pool itself. See concurrency and writer ownership.

Keep file counts bounded by packs

Reuse a store per writer process. Its default 1 GiB pack contains many publications, including their manifests. Queue metadata uses fixed append logs, and GC uses one shared catalog. There is no file per sample or tensor.

Rotate by bytes, batch small writes with publish_many, release consumed data, and seal idle writers. A pack containing even one live record remains retained. Repeated writer restarts, per-sample close()/seal(), or retaining every checkpoint can still grow storage; the target size is not a global quota. File-count and retention planning.

Run something small

After installing the wheel, examples from this source tree run directly:

python examples/records.py
python examples/tensors.py
python examples/work_queue.py
python examples/multiprocess.py --root /tmp/new-straw-recovery-run

Run a bounded local benchmark without SSH or GPUs:

python -m straw.benchmark run --local \
  --root /tmp/new-straw-benchmark --report benchmark-results/local.json \
  --gib 0.0625 --record-bytes 1048576 262144 \
  --segment-mib 8 --online-gc --max-files 64 --max-gib 0.25

The benchmark verifies reads, measures durable publication/acceptance latency, reports bytes and file counts, and removes its temporary payload root after workers stop. Multi-host benchmarks and trace replay.

Build and test on GitHub

  • Full tests: Rust tests, formatting and Clippy, plus the complete Python suite, ownership model, crash campaign, examples and bounded I/O/GC benchmark against installed wheels on CPython 3.10–3.13.
  • Wheels: build and verify Linux x86_64 wheels for CPython 3.10–3.13 (manylinux/glibc 2.28+), plus a source archive and rebuild check. Download the outputs from the run's Artifacts section. Version-tag pushes publish the verified artifacts to PyPI through Trusted Publishing. The same-commit complete verification workflow must pass before package uploads.

Both workflows support Actions → select workflow → Run workflow. Tests also run on pushes and pull requests; wheels run on pull requests and v* tags. Hosted CI uses a local filesystem and CPU PyTorch; deployment qualification on multiple JuiceFS clients and GPU training use separate checks. See build and release instructions.

License

MIT.

Metadata

Release files for straw-queue 0.1.4

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for straw-queue 0.1.4
File Size Uploaded Provenance Actions
straw_queue-0.1.4.tar.gz 1.8 MB With provenance Details for straw_queue-0.1.4.tar.gz

Built distributions (wheels)

Table of built distributions (wheels) for straw-queue 0.1.4
File Actions
straw_queue-0.1.4-cp313-cp313-manylinux_2_28_x86_64.whl CPython 3.13 CPython 3.13 Linux glibc 2.28+ x86-64 Details for straw_queue-0.1.4-cp313-cp313-manylinux_2_28_x86_64.whl
straw_queue-0.1.4-cp312-cp312-manylinux_2_28_x86_64.whl CPython 3.12 CPython 3.12 Linux glibc 2.28+ x86-64 Details for straw_queue-0.1.4-cp312-cp312-manylinux_2_28_x86_64.whl
straw_queue-0.1.4-cp311-cp311-manylinux_2_28_x86_64.whl CPython 3.11 CPython 3.11 Linux glibc 2.28+ x86-64 Details for straw_queue-0.1.4-cp311-cp311-manylinux_2_28_x86_64.whl
straw_queue-0.1.4-cp310-cp310-manylinux_2_28_x86_64.whl CPython 3.10 CPython 3.10 Linux glibc 2.28+ x86-64 Details for straw_queue-0.1.4-cp310-cp310-manylinux_2_28_x86_64.whl

Total release size: 5.3 MB

Release files / straw_queue-0.1.4.tar.gz

Download URL straw_queue-0.1.4.tar.gz
Size 1.8 MB
Tags Source
SHA-256 checksum
How to use checksums
a071c616348e4131eaafee8bbceddc955381f68010ad453db49e37d95ae6858f
BLAKE2b-256 checksum
How to use checksums
dfc8bdddb4aa693f56c16117eb5a3a538848422c2f111a2e45b166c8a3fb666c
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
Yes
Uploaded via twine/7.0.0 CPython/3.13.16

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Oct 26, 2026.

Transparency log

Release files / straw_queue-0.1.4-cp313-cp313-manylinux_2_28_x86_64.whl

Download URL straw_queue-0.1.4-cp313-cp313-manylinux_2_28_x86_64.whl
Size 879.5 kB
Tags CPython 3.13 Linux glibc 2.28+ x86-64
SHA-256 checksum
How to use checksums
24ae90307f44fe0d6adb495bd91e930b66fa4d36c8ecd834e2d0eaccc46d1dad
BLAKE2b-256 checksum
How to use checksums
4386386ed6bb30e372f2069d4e7e07850f5f0d75e0a066cd32fb6e0ebd32c90f
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
Yes
Uploaded via twine/7.0.0 CPython/3.13.16

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Oct 26, 2026.

Transparency log

Release files / straw_queue-0.1.4-cp312-cp312-manylinux_2_28_x86_64.whl

Download URL straw_queue-0.1.4-cp312-cp312-manylinux_2_28_x86_64.whl
Size 880.1 kB
Tags CPython 3.12 Linux glibc 2.28+ x86-64
SHA-256 checksum
How to use checksums
34ad394aba305a38a12b304693637a8284235e1bc6ece599179ab8e4060aaf20
BLAKE2b-256 checksum
How to use checksums
ce76e3794bc2b9fd1314067213e0139afab5d45d3f759928f63f50d2582a312a
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
Yes
Uploaded via twine/7.0.0 CPython/3.13.16

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Oct 26, 2026.

Transparency log

Release files / straw_queue-0.1.4-cp311-cp311-manylinux_2_28_x86_64.whl

Download URL straw_queue-0.1.4-cp311-cp311-manylinux_2_28_x86_64.whl
Size 880.6 kB
Tags CPython 3.11 Linux glibc 2.28+ x86-64
SHA-256 checksum
How to use checksums
b6986c2c8d6629c86d310d719ad7a0a1a9b696d7d4330695764b67033a23ad32
BLAKE2b-256 checksum
How to use checksums
af5b315767704f527d8a7274582eee2816dc45ba9c1e00f3509d612e285933c9
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
Yes
Uploaded via twine/7.0.0 CPython/3.13.16

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Oct 26, 2026.

Transparency log

Release files / straw_queue-0.1.4-cp310-cp310-manylinux_2_28_x86_64.whl

Download URL straw_queue-0.1.4-cp310-cp310-manylinux_2_28_x86_64.whl
Size 877.5 kB
Tags CPython 3.10 Linux glibc 2.28+ x86-64
SHA-256 checksum
How to use checksums
a5477457431605edfdb9c592aa6b9f099acad8b9ef9604150bbab3c489c0c03a
BLAKE2b-256 checksum
How to use checksums
8485ce956760f557eb57e621c83b5d0b9e039568e63febb5eb5770ab3491280c
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
Yes
Uploaded via twine/7.0.0 CPython/3.13.16

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Oct 26, 2026.

Transparency log

Security for straw-queue 0.1.4

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

Full provenance

All 5 files in this release have been uploaded with provenance metadata.

Select from 5 files

Attestation for straw_queue-0.1.4.tar.gz

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Oct 26, 2026.

Transparency log

A note on package security

PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.

To learn more about protecting yourself from malicious actors, visit our package security documentation.

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page