batchcorder-datafusion
A DataFusion table provider that tees one read-once Arrow stream to every scan in a query.
Register a TeeTableProvider with a DataFusion SessionContext, and every
scan of the table in a query (self-joins, CTEs referenced twice, subqueries)
replays the same upstream stream, which is read only once. With the
default bounding="sealed", each planned scan reserves a reader at planning
time, and each batch is evicted as soon as every scan has read past it.
Install
The provider plugs into a DataFusion host through datafusion-ffi. It does
not install a host: install the one you use, in a version from the
compatibility table below.
pip install batchcorder-datafusion "datafusion>=55,<56" # datafusion-python
pip install batchcorder-datafusion "xorq-datafusion>=0.3,<0.4" # xorq-datafusion
Usage
import pyarrow as pa
from datafusion import SessionContext
from batchcorder_datafusion import TeeTableProvider
schema = pa.schema([("k", pa.int64())])
reader = pa.RecordBatchReader.from_batches(
schema, [pa.record_batch([[1, 2, 3]], schema=schema)]
)
ctx = SessionContext()
ctx.register_table("t", TeeTableProvider(reader))
ctx.sql("select * from t a join t b using (k)").collect()
The upstream can be any object that implements the Arrow PyCapsule stream
interface (__arrow_c_stream__).
A sealed provider serves one query. Register a new provider per query, or
pass bounding="unbounded" for a table that is queried repeatedly.
Do not pass a batchcorder.StreamCache
The provider keeps its own cache. Passing a batchcorder reader caches the
data twice. Pass the original upstream reader, unless you also need to replay
the stream outside DataFusion.
Compatibility
datafusion-ffi structs are only compatible within one DataFusion major, so
the provider and the host must be built on the same one. The major version of
batchcorder-datafusion is that DataFusion major.
| batchcorder-datafusion | DataFusion | datafusion (datafusion-python) | xorq-datafusion |
|---|---|---|---|
| 55.x | 55 | >=55,<56 |
>=0.3,<0.4 |
Registering a TeeTableProvider with a host outside these ranges raises
RuntimeError, naming the host version found and the range needed, before any
data crosses the FFI boundary.
batchcorder-datafusion does not depend on the batchcorder package. The
cache is compiled into its extension, and data enters through the Arrow C
stream interface.
Metadata
Release files for batchcorder-datafusion 55.0.0
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | Provenance | Actions |
|---|---|---|---|---|
| batchcorder_datafusion-55.0.0.tar.gz | 71.1 kB | With provenance | Details for batchcorder_datafusion-55.0.0.tar.gz |
Built distributions (wheels)
Total release size: 102.2 MB
Release files / batchcorder_datafusion-55.0.0.tar.gz
| Download URL | batchcorder_datafusion-55.0.0.tar.gz |
|---|---|
| Size | 71.1 kB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
1cc0da5aa2cdf4385dc0f75d6c6a2eb12d4fa9979674c80063369c671484d91d
|
|
BLAKE2b-256 checksum How to use checksums |
5d48edac9a1e1bf817ebb905297a23c3ee90505bc1dab481ea2b565a19a645ec
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
Yes |
| Uploaded via |
twine/7.0.0 CPython/3.13.16
|
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on Nov 13, 2026.
Transparency logRelease files / batchcorder_datafusion-55.0.0-cp310-abi3-win_amd64.whl
| Download URL | batchcorder_datafusion-55.0.0-cp310-abi3-win_amd64.whl |
|---|---|
| Size | 18.9 MB |
| Tags | CPython 3.10 Windows x86-64 abi3 |
|
SHA-256 checksum How to use checksums |
e6a5d56e9c305733f58d55355b3b4613e364765ff87adf61f1cd7489853a51e9
|
|
BLAKE2b-256 checksum How to use checksums |
b1c59405fc91a3bdebb122e7088bd1c3ce36118c85df4332d6ae50025343c7ee
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
Yes |
| Uploaded via |
twine/7.0.0 CPython/3.13.16
|
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on Nov 13, 2026.
Transparency logRelease files / batchcorder_datafusion-55.0.0-cp310-abi3-manylinux_2_28_aarch64.whl
| Download URL | batchcorder_datafusion-55.0.0-cp310-abi3-manylinux_2_28_aarch64.whl |
|---|---|
| Size | 22.9 MB |
| Tags | CPython 3.10 Linux glibc 2.28+ ARM64 abi3 |
|
SHA-256 checksum How to use checksums |
c1d11865c377b82d5d287892cc236306f749da21b3861982e083665e490f7c6e
|
|
BLAKE2b-256 checksum How to use checksums |
9b1901955eb58e0cff4061bdbc916f7f81986ce5ddc4d2623f4a572410be738d
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
Yes |
| Uploaded via |
twine/7.0.0 CPython/3.13.16
|
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on Nov 13, 2026.
Transparency logRelease files / batchcorder_datafusion-55.0.0-cp310-abi3-manylinux2014_x86_64.manylinux_2_17_x86_64.whl
| Download URL | batchcorder_datafusion-55.0.0-cp310-abi3-manylinux2014_x86_64.manylinux_2_17_x86_64.whl |
|---|---|
| Size | 21.4 MB |
| Tags | CPython 3.10 Linux glibc 2.17+ x86-64 abi3 |
|
SHA-256 checksum How to use checksums |
11cc0b1016f0c7d447f51799ee40f437628a9f9900863795dc57515b37f75cc7
|
|
BLAKE2b-256 checksum How to use checksums |
2a1558c9c4897d2fbf7185c2ea188a914652301559d1ac8a69f3208b86bb01be
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
Yes |
| Uploaded via |
twine/7.0.0 CPython/3.13.16
|
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on Nov 13, 2026.
Transparency logRelease files / batchcorder_datafusion-55.0.0-cp310-abi3-macosx_11_0_arm64.whl
| Download URL | batchcorder_datafusion-55.0.0-cp310-abi3-macosx_11_0_arm64.whl |
|---|---|
| Size | 19.0 MB |
| Tags | CPython 3.10 abi3 macOS 11.0+ ARM64 |
|
SHA-256 checksum How to use checksums |
e07f1fd193412628edc55045a8ac5896f6695342aa7b265750f85951543f6dad
|
|
BLAKE2b-256 checksum How to use checksums |
ec69625d27a2d414f0df18bdba65b9b07ba168bdb0708c6f40723e34e7a22e42
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
Yes |
| Uploaded via |
twine/7.0.0 CPython/3.13.16
|
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on Nov 13, 2026.
Transparency logRelease files / batchcorder_datafusion-55.0.0-cp310-abi3-macosx_10_12_x86_64.whl
| Download URL | batchcorder_datafusion-55.0.0-cp310-abi3-macosx_10_12_x86_64.whl |
|---|---|
| Size | 19.8 MB |
| Tags | CPython 3.10 abi3 macOS 10.12+ x86-64 |
|
SHA-256 checksum How to use checksums |
b6dd30d566f11ee7d3109b08b46b175ab93a964a8851f9a8d26201b37667a600
|
|
BLAKE2b-256 checksum How to use checksums |
7df3cad30d26c31819881ab5e9bda9de86ab7ca591347b22c5d2dcc73433e080
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
Yes |
| Uploaded via |
twine/7.0.0 CPython/3.13.16
|
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on Nov 13, 2026.
Transparency logSecurity for batchcorder-datafusion 55.0.0
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
Full provenance
All 6 files in this release have been uploaded with provenance metadata.
Attestation for batchcorder_datafusion-55.0.0.tar.gz
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on Nov 13, 2026.
Transparency logA note on package security
PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.
To learn more about protecting yourself from malicious actors, visit our package security documentation.