Skip to main content

batchcorder-datafusion

A DataFusion table provider that tees one read-once Arrow stream to every scan in a query.

Register a TeeTableProvider with a DataFusion SessionContext, and every scan of the table in a query (self-joins, CTEs referenced twice, subqueries) replays the same upstream stream, which is read only once. With the default bounding="sealed", each planned scan reserves a reader at planning time, and each batch is evicted as soon as every scan has read past it.

Install

The provider plugs into a DataFusion host through datafusion-ffi. It does not install a host: install the one you use, in a version from the compatibility table below.

pip install batchcorder-datafusion "datafusion>=55,<56"        # datafusion-python
pip install batchcorder-datafusion "xorq-datafusion>=0.3,<0.4" # xorq-datafusion

Usage

import pyarrow as pa
from datafusion import SessionContext

from batchcorder_datafusion import TeeTableProvider


schema = pa.schema([("k", pa.int64())])
reader = pa.RecordBatchReader.from_batches(
    schema, [pa.record_batch([[1, 2, 3]], schema=schema)]
)

ctx = SessionContext()
ctx.register_table("t", TeeTableProvider(reader))
ctx.sql("select * from t a join t b using (k)").collect()

The upstream can be any object that implements the Arrow PyCapsule stream interface (__arrow_c_stream__).

A sealed provider serves one query. Register a new provider per query, or pass bounding="unbounded" for a table that is queried repeatedly.

Do not pass a batchcorder.StreamCache

The provider keeps its own cache. Passing a batchcorder reader caches the data twice. Pass the original upstream reader, unless you also need to replay the stream outside DataFusion.

Compatibility

datafusion-ffi structs are only compatible within one DataFusion major, so the provider and the host must be built on the same one. The major version of batchcorder-datafusion is that DataFusion major.

batchcorder-datafusion DataFusion datafusion (datafusion-python) xorq-datafusion
55.x 55 >=55,<56 >=0.3,<0.4

Registering a TeeTableProvider with a host outside these ranges raises RuntimeError, naming the host version found and the range needed, before any data crosses the FFI boundary.

batchcorder-datafusion does not depend on the batchcorder package. The cache is compiled into its extension, and data enters through the Arrow C stream interface.

Metadata

Release files for batchcorder-datafusion 55.0.0

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for batchcorder-datafusion 55.0.0
File Size Uploaded Provenance Actions
batchcorder_datafusion-55.0.0.tar.gz 71.1 kB With provenance Details for batchcorder_datafusion-55.0.0.tar.gz

Built distributions (wheels)

Table of built distributions (wheels) for batchcorder-datafusion 55.0.0
File Actions
batchcorder_datafusion-55.0.0-cp310-abi3-win_amd64.whl CPython 3.10 abi3 Windows x86-64 Details for batchcorder_datafusion-55.0.0-cp310-abi3-win_amd64.whl
batchcorder_datafusion-55.0.0-cp310-abi3-manylinux_2_28_aarch64.whl CPython 3.10 abi3 Linux glibc 2.28+ ARM64 Details for batchcorder_datafusion-55.0.0-cp310-abi3-manylinux_2_28_aarch64.whl
batchcorder_datafusion-55.0.0-cp310-abi3-manylinux2014_x86_64.manylinux_2_17_x86_64.whl CPython 3.10 abi3 Linux glibc 2.17+ x86-64 Details for batchcorder_datafusion-55.0.0-cp310-abi3-manylinux2014_x86_64.manylinux_2_17_x86_64.whl
batchcorder_datafusion-55.0.0-cp310-abi3-macosx_11_0_arm64.whl CPython 3.10 abi3 macOS 11.0+ ARM64 Details for batchcorder_datafusion-55.0.0-cp310-abi3-macosx_11_0_arm64.whl
batchcorder_datafusion-55.0.0-cp310-abi3-macosx_10_12_x86_64.whl CPython 3.10 abi3 macOS 10.12+ x86-64 Details for batchcorder_datafusion-55.0.0-cp310-abi3-macosx_10_12_x86_64.whl

Total release size: 102.2 MB

Release files / batchcorder_datafusion-55.0.0.tar.gz

Download URL batchcorder_datafusion-55.0.0.tar.gz
Size 71.1 kB
Tags Source
SHA-256 checksum
How to use checksums
1cc0da5aa2cdf4385dc0f75d6c6a2eb12d4fa9979674c80063369c671484d91d
BLAKE2b-256 checksum
How to use checksums
5d48edac9a1e1bf817ebb905297a23c3ee90505bc1dab481ea2b565a19a645ec
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
Yes
Uploaded via twine/7.0.0 CPython/3.13.16

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Nov 13, 2026.

Transparency log

Release files / batchcorder_datafusion-55.0.0-cp310-abi3-win_amd64.whl

Download URL batchcorder_datafusion-55.0.0-cp310-abi3-win_amd64.whl
Size 18.9 MB
Tags CPython 3.10 Windows x86-64 abi3
SHA-256 checksum
How to use checksums
e6a5d56e9c305733f58d55355b3b4613e364765ff87adf61f1cd7489853a51e9
BLAKE2b-256 checksum
How to use checksums
b1c59405fc91a3bdebb122e7088bd1c3ce36118c85df4332d6ae50025343c7ee
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
Yes
Uploaded via twine/7.0.0 CPython/3.13.16

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Nov 13, 2026.

Transparency log

Release files / batchcorder_datafusion-55.0.0-cp310-abi3-manylinux_2_28_aarch64.whl

Download URL batchcorder_datafusion-55.0.0-cp310-abi3-manylinux_2_28_aarch64.whl
Size 22.9 MB
Tags CPython 3.10 Linux glibc 2.28+ ARM64 abi3
SHA-256 checksum
How to use checksums
c1d11865c377b82d5d287892cc236306f749da21b3861982e083665e490f7c6e
BLAKE2b-256 checksum
How to use checksums
9b1901955eb58e0cff4061bdbc916f7f81986ce5ddc4d2623f4a572410be738d
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
Yes
Uploaded via twine/7.0.0 CPython/3.13.16

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Nov 13, 2026.

Transparency log

Release files / batchcorder_datafusion-55.0.0-cp310-abi3-manylinux2014_x86_64.manylinux_2_17_x86_64.whl

Download URL batchcorder_datafusion-55.0.0-cp310-abi3-manylinux2014_x86_64.manylinux_2_17_x86_64.whl
Size 21.4 MB
Tags CPython 3.10 Linux glibc 2.17+ x86-64 abi3
SHA-256 checksum
How to use checksums
11cc0b1016f0c7d447f51799ee40f437628a9f9900863795dc57515b37f75cc7
BLAKE2b-256 checksum
How to use checksums
2a1558c9c4897d2fbf7185c2ea188a914652301559d1ac8a69f3208b86bb01be
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
Yes
Uploaded via twine/7.0.0 CPython/3.13.16

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Nov 13, 2026.

Transparency log

Release files / batchcorder_datafusion-55.0.0-cp310-abi3-macosx_11_0_arm64.whl

Download URL batchcorder_datafusion-55.0.0-cp310-abi3-macosx_11_0_arm64.whl
Size 19.0 MB
Tags CPython 3.10 abi3 macOS 11.0+ ARM64
SHA-256 checksum
How to use checksums
e07f1fd193412628edc55045a8ac5896f6695342aa7b265750f85951543f6dad
BLAKE2b-256 checksum
How to use checksums
ec69625d27a2d414f0df18bdba65b9b07ba168bdb0708c6f40723e34e7a22e42
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
Yes
Uploaded via twine/7.0.0 CPython/3.13.16

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Nov 13, 2026.

Transparency log

Release files / batchcorder_datafusion-55.0.0-cp310-abi3-macosx_10_12_x86_64.whl

Download URL batchcorder_datafusion-55.0.0-cp310-abi3-macosx_10_12_x86_64.whl
Size 19.8 MB
Tags CPython 3.10 abi3 macOS 10.12+ x86-64
SHA-256 checksum
How to use checksums
b6dd30d566f11ee7d3109b08b46b175ab93a964a8851f9a8d26201b37667a600
BLAKE2b-256 checksum
How to use checksums
7df3cad30d26c31819881ab5e9bda9de86ab7ca591347b22c5d2dcc73433e080
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
Yes
Uploaded via twine/7.0.0 CPython/3.13.16

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Nov 13, 2026.

Transparency log

Security for batchcorder-datafusion 55.0.0

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

Full provenance

All 6 files in this release have been uploaded with provenance metadata.

Select from 6 files

Attestation for batchcorder_datafusion-55.0.0.tar.gz

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Nov 13, 2026.

Transparency log

A note on package security

PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.

To learn more about protecting yourself from malicious actors, visit our package security documentation.

Release history Release notifications | RSS feed

This release

55.0.0 This release

6 release files Full provenance

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page