minarrow

Arrow-compatible data that moves cleanly between Rust and Python, stays ready for SIMD, and plugs into the rest of the Python data ecosystem.

Apache 2.0

Documentation: minarrow.com

minarrow is a compact Python interface over Minarrow Rust.

It gives Rust and Python the same columnar data model, with Arrow-compatible memory layouts and 64-byte-aligned buffers. Data produced in Rust can be exposed directly to Python, operated on by native SIMD kernels, and passed to PyArrow, Polars, DuckDB and other Arrow-aware libraries through the Arrow PyCapsule interface.

import minarrow as ma

table = ma.Table(
    {
        "id": [1, 2, 3],
        "price": [9.5, 10.0, 11.2],
    },
    name="prices",
)

series = table["price"].to_polars()
relation = table.to_duckdb()

Minarrow is useful when:

  • Data is produced or processed in Rust and consumed in Python
  • Downstream libraries such as Polars benefit from SIMD-compatible, 64-byte-aligned memory
  • Arrow interoperability is required without taking an internal dependency on PyArrow
  • A small set of typed containers is preferable to a large object hierarchy
  • Higher-level computation is delegated to Polars, DuckDB, PyArrow or another execution engine

The columnar API centres on four containers:

  • Array for a single typed column
  • Table for a fixed set of equal-length columns
  • ChunkedArray for one logical column split across multiple chunks
  • ChunkedTable for a sequence of table batches

Together, they cover the common Arrow tabular data model. Nested Struct and List types are not currently supported.

The tensor API adds three focused containers:

  • NdArray for contiguous or zero-copy windowed f32/f64 data
  • ChunkedNdArray for compatible NdArray pieces backed by Rust SuperNdArray
  • XArray for named dimensions and coordinate-based selection over an NdArray

They provide storage, indexing, selection, and DLPack interchange. NdArray and XArray expose one tensor directly; each ChunkedNdArray chunk is a separate NdArray DLPack producer with its own data pointer. Numerical algorithms, statistics, and general tensor computation remain the job of NumPy, PyTorch, JAX, or another execution library.

Why not just use PyArrow?

PyArrow is an excellent Arrow implementation, but it solves a different problem. It brings the full Arrow C++ runtime into Python, together with a large type system, compute engine, file formats and dataset APIs.

Minarrow is for applications where the important path is Rust -> Python -> native analytics.

It uses the same Minarrow data model on both sides of the Rust–Python boundary, allocates supported buffers at 64-byte alignment (PyArrow only guarantees 8-byte), and exposes them through the Arrow PyCapsule interface. Rust can produce the data, Python can inspect and compose it, and Polars, DuckDB, PyArrow or a custom native kernel can consume it without an intermediate serialisation step.

For example, a Rust service can use Lightstream to receive a network feed directly into 64-byte-aligned Arrow buffers, expose those buffers to Python through Minarrow, and continue processing them with Python, native SIMD kernels, or machine-learning libraries without serialising or rebuilding the dataset.

Minarrow PyArrow
Rust–Python data model Native Minarrow types across both runtimes Rust integration through Arrow interchange interfaces
Host runtime Rust, embedded in the application service Arrow C++ runtime loaded into Python
Buffer alignment 64-byte aligned for Minarrow-backed buffers 8-byte Arrow alignment guarantee
Python API Focused columnar and tensor containers Full Arrow type and container hierarchy
Native SIMD use Buffers are ready for aligned SIMD kernels Consumers must inspect alignment or realign the data
Runtime role Application data model, interchange and composition Arrow compute, storage and dataset platform
Ecosystem integration Arrow PyCapsule and direct runtime bridges PyArrow APIs and Arrow interoperability
Dependency footprint Focused Rust extension Full Arrow C++ and Python distribution

Minarrow avoids making PyArrow a required part of the data path. Applications can depend only on Minarrow, hand its objects directly to capsule-aware libraries, and introduce PyArrow only where its broader functionality is useful.

import minarrow as ma

table = ma.Table(
    {
        "id": [1, 2, 3],
        "price": [9.5, 10.0, 11.2],
    }
)

polars_frame = table.to_polars()
duckdb_relation = table.to_duckdb()

PyArrow remains fully interoperable:

import pyarrow as pa

arrow_table = pa.table(table)
minarrow_table = ma.Table.from_arrow(arrow_table)

Rust and Python share the same data model

Minarrow arrays and tables exist on both sides of the Rust–Python boundary.

Rust code can construct the data, perform native processing, and expose it to Python without first serialising it into another format. Python can then pass the same Arrow-compatible buffers into Polars, DuckDB, PyArrow or another capsule-aware library zero-copy.

This suits:

  • Rust-based parsers and decoders
  • Python extensions with columnar outputs
  • Low-latency ingestion systems
  • Scientific and simulation code
  • Feature-generation pipelines
  • Native analytics kernels

This pattern suits systems where Rust provides the production application, transport and data services, while Python delivers the intelligence layer.

SIMD-ready buffers

Minarrow allocates supported buffers on 64-byte boundaries, matching the alignment required by wide SIMD paths such as AVX2 and AVX-512.

For compatible workloads, this adds a low-latency layer of parallelism alongside standard multi-core execution:

  • Native kernels can operate directly on the buffers without copying or realigning the data.
  • The buffers remain in host memory and can be processed immediately by the CPU.
  • GPUs offer greater peak throughput for sufficiently large, highly parallel workloads, but first require the data to be transferred into device memory.
  • For smaller batches and latency-sensitive workloads, the transfer can take longer than the computation itself, making CPU SIMD the faster end-to-end path for many tabular workloads that require a rapid decision.

The alignment is preserved when Minarrow-owned data moves from Rust into Python. Python extensions and Arrow PyCapsule consumers can access the same underlying buffers and pass them directly to compatible native kernels.

By contrast, PyArrow guarantees 8-byte alignment rather than 64-byte alignment. A native kernel that requires 64-byte-aligned input may therefore need to copy the data into a new allocation before processing it. For small and latency-sensitive workloads, that copy can take longer than the SIMD calculation itself.

Small Python surface

Minarrow exposes one container per structural role rather than one Python class for every Arrow type.

array.dtype
array.dtype.group
array.bit_width
array.arrow_type

.dtype provides a compact application-level type, while .arrow_type retains the complete Arrow logical type.

Pluggable execution

Minarrow stores and exchanges columnar data. It does not attempt to replace a dataframe or query engine.

Use Minarrow for the Rust–Python boundary, then choose the execution engine appropriate to the workload:

polars_frame = table.to_polars()
duckdb_relation = table.to_duckdb()
arrow_table = table.to_arrow()

This keeps the data layer small without restricting the rest of the application to a Minarrow-specific API.

Project links

Key dates

PyPI data
Data sourced directly from PyPI's database.
  • Released:
Latest release

1 maintainer

PyPI data
Data sourced directly from PyPI's database.
Avatar for spacecell from gravatar.com spacecell

Credits

Author: Peter Bower

License expression

Apache-2.0
View SPDX License List

Requires

Python >=3.9

Provides Extra

dev

Tags

arrow columnar dataframe pyarrow analytics simd

Classifiers

Report project as malware

Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

minarrow-0.16.0.tar.gz (2.8 MB view details)

Uploaded Source

Built Distributions

If you're not sure about the file name format, learn more about wheel file names.

minarrow-0.16.0-cp39-abi3-win_amd64.whl (806.3 kB view details)

Uploaded CPython 3.9+Windows x86-64

minarrow-0.16.0-cp39-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (826.9 kB view details)

Uploaded CPython 3.9+manylinux: glibc 2.17+ x86-64

minarrow-0.16.0-cp39-abi3-manylinux_2_17_aarch64.manylinux2014_aarch64.whl (765.5 kB view details)

Uploaded CPython 3.9+manylinux: glibc 2.17+ ARM64

minarrow-0.16.0-cp39-abi3-macosx_11_0_arm64.whl (756.2 kB view details)

Uploaded CPython 3.9+macOS 11.0+ ARM64

minarrow-0.16.0-cp39-abi3-macosx_10_12_x86_64.whl (792.3 kB view details)

Uploaded CPython 3.9+macOS 10.12+ x86-64

File details

Details for the file minarrow-0.16.0.tar.gz.

File metadata

  • Download URL: minarrow-0.16.0.tar.gz
  • Upload date:
  • Size: 2.8 MB
  • Tags: Source
  • Uploaded using Trusted Publishing? Yes
  • Uploaded via: twine/6.1.0 CPython/3.13.12

File hashes

Hashes for minarrow-0.16.0.tar.gz
Algorithm Hash digest
SHA256 3a1f90fecb2f2e5a39af58f0ca9d8c9f3e0ae52817f5be07f83c2e8917868b49
MD5 ef6a1f05130d36542347e4de12c43117
BLAKE2b-256 c4b7a41e3cb990003aba4dd730532d3d5e1fcc61333a4631c94ce295e77d7a53

See more details on using hashes here.

Provenance

The following attestation bundles were made for minarrow-0.16.0.tar.gz:

Publisher: release.yml on SpaceCell/minarrow

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

File details

Details for the file minarrow-0.16.0-cp39-abi3-win_amd64.whl.

File metadata

  • Download URL: minarrow-0.16.0-cp39-abi3-win_amd64.whl
  • Upload date:
  • Size: 806.3 kB
  • Tags: CPython 3.9+, Windows x86-64
  • Uploaded using Trusted Publishing? Yes
  • Uploaded via: twine/6.1.0 CPython/3.13.12

File hashes

Hashes for minarrow-0.16.0-cp39-abi3-win_amd64.whl
Algorithm Hash digest
SHA256 86fc8d92f8e9331653e6140340add64b4a9998eee21fc58f091bdf0e318d1777
MD5 31d87ac92ba5813f0de0d73a01fd1fd6
BLAKE2b-256 8cf50eb7cf1ec71429208dcde9320e81ee29d4abc596e80edd55265bb4f55812

See more details on using hashes here.

Provenance

The following attestation bundles were made for minarrow-0.16.0-cp39-abi3-win_amd64.whl:

Publisher: release.yml on SpaceCell/minarrow

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

File details

Details for the file minarrow-0.16.0-cp39-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.

File metadata

File hashes

Hashes for minarrow-0.16.0-cp39-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm Hash digest
SHA256 31eff5c05749102e0eb4530a00864a95fd674352b80d3bf5b17917fdf0d2cd1d
MD5 10d92bed59f909c8c73a406211065e8c
BLAKE2b-256 5519c9ff3dd4593914cecb465c8cd2b557d46d85ea219048721cadd09cd649aa

See more details on using hashes here.

Provenance

The following attestation bundles were made for minarrow-0.16.0-cp39-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl:

Publisher: release.yml on SpaceCell/minarrow

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

File details

Details for the file minarrow-0.16.0-cp39-abi3-manylinux_2_17_aarch64.manylinux2014_aarch64.whl.

File metadata

File hashes

Hashes for minarrow-0.16.0-cp39-abi3-manylinux_2_17_aarch64.manylinux2014_aarch64.whl
Algorithm Hash digest
SHA256 ff5b667710bdfc58dc680cbbec40e5a1d8c4adae69e60236d0055af932705561
MD5 14938c4d8f9fdc0c57b7f4ac24049076
BLAKE2b-256 84f876784749d12714ad485bf57195cd7f3bcf920ee4312ca9f217253e16af1b

See more details on using hashes here.

Provenance

The following attestation bundles were made for minarrow-0.16.0-cp39-abi3-manylinux_2_17_aarch64.manylinux2014_aarch64.whl:

Publisher: release.yml on SpaceCell/minarrow

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

File details

Details for the file minarrow-0.16.0-cp39-abi3-macosx_11_0_arm64.whl.

File metadata

File hashes

Hashes for minarrow-0.16.0-cp39-abi3-macosx_11_0_arm64.whl
Algorithm Hash digest
SHA256 6ad987127b3fba8d57c727d45065b01e38adbb43a9abebb1182659183940a3e8
MD5 04974647c8955fdc37370bba822c82d4
BLAKE2b-256 0a8c82913a54ab6335db02d3e8637b4a8885fd4283e632aa245398f71ef4154e

See more details on using hashes here.

Provenance

The following attestation bundles were made for minarrow-0.16.0-cp39-abi3-macosx_11_0_arm64.whl:

Publisher: release.yml on SpaceCell/minarrow

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

File details

Details for the file minarrow-0.16.0-cp39-abi3-macosx_10_12_x86_64.whl.

File metadata

File hashes

Hashes for minarrow-0.16.0-cp39-abi3-macosx_10_12_x86_64.whl
Algorithm Hash digest
SHA256 d776f5c95ea0d38237a53c99705baee18b6c9e7418705fbc95e6bc1045423f7f
MD5 e2480805672677f953398d8bdf9d88db
BLAKE2b-256 3d49b793673e1eb19ab83426756c4f82a398bd4c5a3449b5302489ea0e426103

See more details on using hashes here.

Provenance

The following attestation bundles were made for minarrow-0.16.0-cp39-abi3-macosx_10_12_x86_64.whl:

Publisher: release.yml on SpaceCell/minarrow

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

Supported by

AWS Cloud computing and Security Sponsor Datadog Monitoring Depot Continuous Integration Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page