Skip to main content

batchcorder

A Rust-backed Python library for caching Arrow record-batch streams so they can be replayed multiple times from a source that can only be read once.

The problem

Arrow RecordBatchReader is a single-use stream — once consumed, it is gone. Training loops and multi-pass data pipelines need to iterate the same stream repeatedly without re-reading from disk or the network each time.

What batchcorder does

StreamCache wraps any Arrow stream source (anything that implements __arrow_c_stream__) and stores each RecordBatch in a cache. Two storage modes are supported:

  • Memory-only (default): batches are stored as reference-counted pointers in RAM; reads are zero-copy and no IPC serialisation happens.
  • Hybrid memory+disk: batches are serialised to an append-only IPC file on disk; a configurable hot layer keeps recently ingested batches in RAM to reduce disk I/O.

Multiple independent readers can replay the stream concurrently, each maintaining their own position in the batch sequence.

flowchart LR
    U["upstream source<br/>(read once)"] --> D["StreamCache<br/>[mem cache / mem+disk cache]"]
    D --> R0["StreamCacheReader 0<br/>(from batch 0)"]
    D --> R1["StreamCacheReader 1<br/>(from batch 0)"]
    D --> R2["StreamCacheReader 2<br/>(from batch 3)"]

Installation

pip install batchcorder

Usage

import tempfile

import pyarrow as pa

from batchcorder import StreamCache

table = pa.table({"x": [1, 2, 3], "y": [4, 5, 6]})

# Memory-only (default capacity = total physical RAM)
ds = StreamCache(table.to_reader(max_chunksize=1))

# Memory-only with explicit capacity
ds = StreamCache(
    table.to_reader(max_chunksize=1),
    memory_capacity=64 * 1024 * 1024,  # 64 MB
)

# Hybrid memory+disk
tmp = tempfile.mkdtemp()
ds = StreamCache(
    table.to_reader(max_chunksize=1),
    memory_capacity=64 * 1024 * 1024,  # 64 MB in RAM
    disk_path=tmp,
    disk_capacity=512 * 1024 * 1024,  # 512 MB on disk
)

# Replay as many times as needed
for batch in ds:
    print(batch)

# Or get an independent reader handle
reader = ds.reader()
result = pa.RecordBatchReader.from_stream(reader).read_all()

# Bounded-memory: evict batches once all readers have passed them
ds = StreamCache(
    table.to_reader(max_chunksize=1),
    max_readers=2,  # at most 2 reads; batches evicted when both advance
)

# Pre-ingest everything upfront
ds.ingest_all()

Compatibility

StreamCache and StreamCacheReader implement both __arrow_c_stream__ and __arrow_c_schema__, so they work with any Arrow-compatible library:

import pyarrow as pa
import duckdb

pa.table(ds)  # PyArrow
pa.table(ds.reader())  # via StreamCacheReader
duckdb.table("ds")  # DuckDB

Key properties

  • Single-read source: the upstream stream is consumed exactly once; all subsequent reads come from the cache.
  • Concurrent readers: multiple StreamCacheReader instances from the same stream cache are fully independent and thread-safe.
  • Lazy ingestion: batches are fetched from the upstream source on demand as readers advance, not upfront.
  • Replay from any position: ds.reader(from_start=True) (default) replays from batch 0; ds.reader(from_start=False) starts from the current ingestion frontier (next batch not yet ingested).
  • Bounded-memory streaming: set max_readers=N to evict batches once all N readers exist and have advanced past them — eviction does not begin until all N readers have been created. max_readers is a hard cap on total readers ever created (dropping a reader does not free a slot). Once eviction has started, reader(from_start=True) raises ValueError. When unset, all batches are retained indefinitely.

Development

# Install dependencies and build the extension
uv sync --no-install-project --dev
uv run maturin develop --uv

# Run tests
uv run pytest

# Run all pre-commit checks
uv run pre-commit run --all-files

Metadata

Release files for batchcorder 0.1.4

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for batchcorder 0.1.4
File Size Uploaded Provenance Actions
batchcorder-0.1.4.tar.gz 66.1 kB With provenance Details for batchcorder-0.1.4.tar.gz

Built distributions (wheels)

Table of built distributions (wheels) for batchcorder 0.1.4
File Actions
batchcorder-0.1.4-cp310-abi3-win_amd64.whl CPython 3.10 abi3 Windows x86-64 Details for batchcorder-0.1.4-cp310-abi3-win_amd64.whl
batchcorder-0.1.4-cp310-abi3-win32.whl CPython 3.10 abi3 Windows x86-32 Details for batchcorder-0.1.4-cp310-abi3-win32.whl
batchcorder-0.1.4-cp310-abi3-manylinux_2_28_aarch64.whl CPython 3.10 abi3 Linux glibc 2.28+ ARM64 Details for batchcorder-0.1.4-cp310-abi3-manylinux_2_28_aarch64.whl
batchcorder-0.1.4-cp310-abi3-manylinux2014_x86_64.manylinux_2_17_x86_64.whl CPython 3.10 abi3 Linux glibc 2.17+ x86-64 Details for batchcorder-0.1.4-cp310-abi3-manylinux2014_x86_64.manylinux_2_17_x86_64.whl
batchcorder-0.1.4-cp310-abi3-macosx_11_0_arm64.whl CPython 3.10 abi3 macOS 11.0+ ARM64 Details for batchcorder-0.1.4-cp310-abi3-macosx_11_0_arm64.whl
batchcorder-0.1.4-cp310-abi3-macosx_10_12_x86_64.whl CPython 3.10 abi3 macOS 10.12+ x86-64 Details for batchcorder-0.1.4-cp310-abi3-macosx_10_12_x86_64.whl

Total release size: 7.8 MB

Release files / batchcorder-0.1.4.tar.gz

Download URL batchcorder-0.1.4.tar.gz
Size 66.1 kB
Tags Source
SHA-256 checksum
How to use checksums
8b2e06d1128735f05b7d6ec733c92e370d3a314092467fcdf0f167d1ca10162d
BLAKE2b-256 checksum
How to use checksums
63b34f3633350a0c9b3f1c5ee5ca8c8ea428fe12660e06405695f7b5948a23cd
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
Yes
Uploaded via twine/7.0.0 CPython/3.13.16

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Nov 13, 2026.

Transparency log

Release files / batchcorder-0.1.4-cp310-abi3-win_amd64.whl

Download URL batchcorder-0.1.4-cp310-abi3-win_amd64.whl
Size 1.2 MB
Tags CPython 3.10 Windows x86-64 abi3
SHA-256 checksum
How to use checksums
47cc96a5e45ba8dc9a55ccda0edcdcb57441b7df8e263b2a1c26f955adad4007
BLAKE2b-256 checksum
How to use checksums
bf2d5f8db128bcf1f7a583339cb085608ad3df34c31d75c2be79a7a0c42df274
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
Yes
Uploaded via twine/7.0.0 CPython/3.13.16

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Nov 13, 2026.

Transparency log

Release files / batchcorder-0.1.4-cp310-abi3-win32.whl

Download URL batchcorder-0.1.4-cp310-abi3-win32.whl
Size 1.0 MB
Tags CPython 3.10 Windows x86-32 abi3
SHA-256 checksum
How to use checksums
cd6ede2aab73404c40c35dc4d86ec0bba7b927d2c006ad4c6f97a4f739bc399c
BLAKE2b-256 checksum
How to use checksums
1e84822d17f509d8c58ebce963c113b5e338a711624db685f0d8f81ed26aedc1
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
Yes
Uploaded via twine/7.0.0 CPython/3.13.16

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Nov 13, 2026.

Transparency log

Release files / batchcorder-0.1.4-cp310-abi3-manylinux_2_28_aarch64.whl

Download URL batchcorder-0.1.4-cp310-abi3-manylinux_2_28_aarch64.whl
Size 1.5 MB
Tags CPython 3.10 Linux glibc 2.28+ ARM64 abi3
SHA-256 checksum
How to use checksums
f75a388d4052414796bce06eef0f871ae4f1bc92071f8def5d132edd1b2be26d
BLAKE2b-256 checksum
How to use checksums
adcb0ea80e91cd6c066890706d9b9e698008c22fa74cfa6e8800fd6a80a49162
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
Yes
Uploaded via twine/7.0.0 CPython/3.13.16

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Nov 13, 2026.

Transparency log

Release files / batchcorder-0.1.4-cp310-abi3-manylinux2014_x86_64.manylinux_2_17_x86_64.whl

Download URL batchcorder-0.1.4-cp310-abi3-manylinux2014_x86_64.manylinux_2_17_x86_64.whl
Size 1.5 MB
Tags CPython 3.10 Linux glibc 2.17+ x86-64 abi3
SHA-256 checksum
How to use checksums
858363068705bb42da54cc52a05795e65262d9aa30a0602583830fe05d19acf6
BLAKE2b-256 checksum
How to use checksums
3fce529c6f3396af8d667fbb57358553722b020f2e24b8d343a467ed74f1b0d2
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
Yes
Uploaded via twine/7.0.0 CPython/3.13.16

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Nov 13, 2026.

Transparency log

Release files / batchcorder-0.1.4-cp310-abi3-macosx_11_0_arm64.whl

Download URL batchcorder-0.1.4-cp310-abi3-macosx_11_0_arm64.whl
Size 1.2 MB
Tags CPython 3.10 abi3 macOS 11.0+ ARM64
SHA-256 checksum
How to use checksums
b0366f2e71d358f3ab0ffdb7b8dd67788b39a51b750411393433c4f39c6d1897
BLAKE2b-256 checksum
How to use checksums
8667b4052d843420229dbf846dfcc3d46b20065ceb5192c201918f632bf03f28
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
Yes
Uploaded via twine/7.0.0 CPython/3.13.16

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Nov 13, 2026.

Transparency log

Release files / batchcorder-0.1.4-cp310-abi3-macosx_10_12_x86_64.whl

Download URL batchcorder-0.1.4-cp310-abi3-macosx_10_12_x86_64.whl
Size 1.3 MB
Tags CPython 3.10 abi3 macOS 10.12+ x86-64
SHA-256 checksum
How to use checksums
f78a28c6b50486b9f52872629d81fa41df24635135cc048ed56a1b6019a35e2a
BLAKE2b-256 checksum
How to use checksums
8ebc3c6adef37b10c6678c2b5a84d2f288d449b95ac65545d9a4fcd64a5a3e9d
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
Yes
Uploaded via twine/7.0.0 CPython/3.13.16

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Nov 13, 2026.

Transparency log

Security for batchcorder 0.1.4

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

Full provenance

All 7 files in this release have been uploaded with provenance metadata.

Select from 7 files

Attestation for batchcorder-0.1.4.tar.gz

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Nov 13, 2026.

Transparency log

A note on package security

PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.

To learn more about protecting yourself from malicious actors, visit our package security documentation.

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page