batchcorder
A Rust-backed Python library for caching Arrow record-batch streams so they can be replayed multiple times from a source that can only be read once.
The problem
Arrow RecordBatchReader is a single-use stream — once consumed, it is gone.
Training loops and multi-pass data pipelines need to iterate the same stream
repeatedly without re-reading from disk or the network each time.
What batchcorder does
StreamCache wraps any Arrow stream source (anything that implements
__arrow_c_stream__) and stores each RecordBatch in a cache. Two storage
modes are supported:
- Memory-only (default): batches are stored as reference-counted pointers in RAM; reads are zero-copy and no IPC serialisation happens.
- Hybrid memory+disk: batches are serialised to an append-only IPC file on disk; a configurable hot layer keeps recently ingested batches in RAM to reduce disk I/O.
Multiple independent readers can replay the stream concurrently, each maintaining their own position in the batch sequence.
flowchart LR
U["upstream source<br/>(read once)"] --> D["StreamCache<br/>[mem cache / mem+disk cache]"]
D --> R0["StreamCacheReader 0<br/>(from batch 0)"]
D --> R1["StreamCacheReader 1<br/>(from batch 0)"]
D --> R2["StreamCacheReader 2<br/>(from batch 3)"]
Installation
pip install batchcorder
Usage
import tempfile
import pyarrow as pa
from batchcorder import StreamCache
table = pa.table({"x": [1, 2, 3], "y": [4, 5, 6]})
# Memory-only (default capacity = total physical RAM)
ds = StreamCache(table.to_reader(max_chunksize=1))
# Memory-only with explicit capacity
ds = StreamCache(
table.to_reader(max_chunksize=1),
memory_capacity=64 * 1024 * 1024, # 64 MB
)
# Hybrid memory+disk
tmp = tempfile.mkdtemp()
ds = StreamCache(
table.to_reader(max_chunksize=1),
memory_capacity=64 * 1024 * 1024, # 64 MB in RAM
disk_path=tmp,
disk_capacity=512 * 1024 * 1024, # 512 MB on disk
)
# Replay as many times as needed
for batch in ds:
print(batch)
# Or get an independent reader handle
reader = ds.reader()
result = pa.RecordBatchReader.from_stream(reader).read_all()
# Bounded-memory: evict batches once all readers have passed them
ds = StreamCache(
table.to_reader(max_chunksize=1),
max_readers=2, # at most 2 reads; batches evicted when both advance
)
# Pre-ingest everything upfront
ds.ingest_all()
Compatibility
StreamCache and StreamCacheReader implement both __arrow_c_stream__
and __arrow_c_schema__, so they work with any Arrow-compatible library:
import pyarrow as pa
import duckdb
pa.table(ds) # PyArrow
pa.table(ds.reader()) # via StreamCacheReader
duckdb.table("ds") # DuckDB
Key properties
- Single-read source: the upstream stream is consumed exactly once; all subsequent reads come from the cache.
- Concurrent readers: multiple
StreamCacheReaderinstances from the same stream cache are fully independent and thread-safe. - Lazy ingestion: batches are fetched from the upstream source on demand as readers advance, not upfront.
- Replay from any position:
ds.reader(from_start=True)(default) replays from batch 0;ds.reader(from_start=False)starts from the current ingestion frontier (next batch not yet ingested). - Bounded-memory streaming: set
max_readers=Nto evict batches once allNreaders exist and have advanced past them — eviction does not begin until allNreaders have been created.max_readersis a hard cap on total readers ever created (dropping a reader does not free a slot). Once eviction has started,reader(from_start=True)raisesValueError. When unset, all batches are retained indefinitely.
Development
# Install dependencies and build the extension
uv sync --no-install-project --dev
uv run maturin develop --uv
# Run tests
uv run pytest
# Run all pre-commit checks
uv run pre-commit run --all-files
Metadata
Release files for batchcorder 0.1.4
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | Provenance | Actions |
|---|---|---|---|---|
| batchcorder-0.1.4.tar.gz | 66.1 kB | With provenance | Details for batchcorder-0.1.4.tar.gz |
Built distributions (wheels)
Total release size: 7.8 MB
Release files / batchcorder-0.1.4.tar.gz
| Download URL | batchcorder-0.1.4.tar.gz |
|---|---|
| Size | 66.1 kB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
8b2e06d1128735f05b7d6ec733c92e370d3a314092467fcdf0f167d1ca10162d
|
|
BLAKE2b-256 checksum How to use checksums |
63b34f3633350a0c9b3f1c5ee5ca8c8ea428fe12660e06405695f7b5948a23cd
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
Yes |
| Uploaded via |
twine/7.0.0 CPython/3.13.16
|
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on Nov 13, 2026.
Transparency logRelease files / batchcorder-0.1.4-cp310-abi3-win_amd64.whl
| Download URL | batchcorder-0.1.4-cp310-abi3-win_amd64.whl |
|---|---|
| Size | 1.2 MB |
| Tags | CPython 3.10 Windows x86-64 abi3 |
|
SHA-256 checksum How to use checksums |
47cc96a5e45ba8dc9a55ccda0edcdcb57441b7df8e263b2a1c26f955adad4007
|
|
BLAKE2b-256 checksum How to use checksums |
bf2d5f8db128bcf1f7a583339cb085608ad3df34c31d75c2be79a7a0c42df274
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
Yes |
| Uploaded via |
twine/7.0.0 CPython/3.13.16
|
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on Nov 13, 2026.
Transparency logRelease files / batchcorder-0.1.4-cp310-abi3-win32.whl
| Download URL | batchcorder-0.1.4-cp310-abi3-win32.whl |
|---|---|
| Size | 1.0 MB |
| Tags | CPython 3.10 Windows x86-32 abi3 |
|
SHA-256 checksum How to use checksums |
cd6ede2aab73404c40c35dc4d86ec0bba7b927d2c006ad4c6f97a4f739bc399c
|
|
BLAKE2b-256 checksum How to use checksums |
1e84822d17f509d8c58ebce963c113b5e338a711624db685f0d8f81ed26aedc1
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
Yes |
| Uploaded via |
twine/7.0.0 CPython/3.13.16
|
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on Nov 13, 2026.
Transparency logRelease files / batchcorder-0.1.4-cp310-abi3-manylinux_2_28_aarch64.whl
| Download URL | batchcorder-0.1.4-cp310-abi3-manylinux_2_28_aarch64.whl |
|---|---|
| Size | 1.5 MB |
| Tags | CPython 3.10 Linux glibc 2.28+ ARM64 abi3 |
|
SHA-256 checksum How to use checksums |
f75a388d4052414796bce06eef0f871ae4f1bc92071f8def5d132edd1b2be26d
|
|
BLAKE2b-256 checksum How to use checksums |
adcb0ea80e91cd6c066890706d9b9e698008c22fa74cfa6e8800fd6a80a49162
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
Yes |
| Uploaded via |
twine/7.0.0 CPython/3.13.16
|
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on Nov 13, 2026.
Transparency logRelease files / batchcorder-0.1.4-cp310-abi3-manylinux2014_x86_64.manylinux_2_17_x86_64.whl
| Download URL | batchcorder-0.1.4-cp310-abi3-manylinux2014_x86_64.manylinux_2_17_x86_64.whl |
|---|---|
| Size | 1.5 MB |
| Tags | CPython 3.10 Linux glibc 2.17+ x86-64 abi3 |
|
SHA-256 checksum How to use checksums |
858363068705bb42da54cc52a05795e65262d9aa30a0602583830fe05d19acf6
|
|
BLAKE2b-256 checksum How to use checksums |
3fce529c6f3396af8d667fbb57358553722b020f2e24b8d343a467ed74f1b0d2
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
Yes |
| Uploaded via |
twine/7.0.0 CPython/3.13.16
|
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on Nov 13, 2026.
Transparency logRelease files / batchcorder-0.1.4-cp310-abi3-macosx_11_0_arm64.whl
| Download URL | batchcorder-0.1.4-cp310-abi3-macosx_11_0_arm64.whl |
|---|---|
| Size | 1.2 MB |
| Tags | CPython 3.10 abi3 macOS 11.0+ ARM64 |
|
SHA-256 checksum How to use checksums |
b0366f2e71d358f3ab0ffdb7b8dd67788b39a51b750411393433c4f39c6d1897
|
|
BLAKE2b-256 checksum How to use checksums |
8667b4052d843420229dbf846dfcc3d46b20065ceb5192c201918f632bf03f28
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
Yes |
| Uploaded via |
twine/7.0.0 CPython/3.13.16
|
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on Nov 13, 2026.
Transparency logRelease files / batchcorder-0.1.4-cp310-abi3-macosx_10_12_x86_64.whl
| Download URL | batchcorder-0.1.4-cp310-abi3-macosx_10_12_x86_64.whl |
|---|---|
| Size | 1.3 MB |
| Tags | CPython 3.10 abi3 macOS 10.12+ x86-64 |
|
SHA-256 checksum How to use checksums |
f78a28c6b50486b9f52872629d81fa41df24635135cc048ed56a1b6019a35e2a
|
|
BLAKE2b-256 checksum How to use checksums |
8ebc3c6adef37b10c6678c2b5a84d2f288d449b95ac65545d9a4fcd64a5a3e9d
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
Yes |
| Uploaded via |
twine/7.0.0 CPython/3.13.16
|
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on Nov 13, 2026.
Transparency logSecurity for batchcorder 0.1.4
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
Full provenance
All 7 files in this release have been uploaded with provenance metadata.
Attestation for batchcorder-0.1.4.tar.gz
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on Nov 13, 2026.
Transparency logA note on package security
PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.
To learn more about protecting yourself from malicious actors, visit our package security documentation.