minarrow

Arrow-compatible data that moves cleanly between Rust and Python, stays ready for SIMD, and plugs into the rest of the Python data ecosystem.

Apache 2.0

Documentation: minarrow.com

minarrow is a compact Python interface over the Minarrow Rust core.

It gives Rust and Python the same columnar data model, with Arrow-compatible memory layouts and 64-byte-aligned buffers. Data produced in Rust can be exposed directly to Python, operated on by native SIMD kernels, and passed to PyArrow, Polars, DuckDB and other Arrow-aware libraries through the Arrow PyCapsule interface.

import minarrow as ma

table = ma.Table(
    {
        "id": [1, 2, 3],
        "price": [9.5, 10.0, 11.2],
    },
    name="prices",
)

series = table["price"].to_polars()
relation = table.to_duckdb()

Minarrow is useful when:

  • Data is produced or processed in Rust and consumed in Python
  • Downstream libraries such as Polars benefit from SIMD-compatible, 64-byte-aligned memory
  • Arrow interoperability is required without taking an internal dependency on PyArrow
  • A small set of typed containers is preferable to a large object hierarchy
  • Higher-level computation is delegated to Polars, DuckDB, PyArrow or another execution engine

The Python API centres on four containers:

  • Array for a single typed column
  • Table for a fixed set of equal-length columns
  • ChunkedArray for one logical column split across multiple chunks
  • ChunkedTable for a sequence of table batches

Together, they cover the common Arrow tabular data model. Nested Struct and List types are not currently supported.

Why not just use PyArrow?

PyArrow is an excellent Arrow implementation, but it solves a different problem. It brings the full Arrow C++ runtime into Python, together with a large type system, compute engine, file formats and dataset APIs.

Minarrow is for applications where the important path is Rust -> Python -> native analytics.

It uses the same Minarrow data model on both sides of the Rust–Python boundary, allocates supported buffers at 64-byte alignment (PyArrow only guarantees 8-byte), and exposes them through the Arrow PyCapsule interface. Rust can produce the data, Python can inspect and compose it, and Polars, DuckDB, PyArrow or a custom native kernel can consume it without an intermediate serialisation step.

For example, a Rust service can use Lightstream to receive a network feed directly into 64-byte-aligned Arrow buffers, expose those buffers to Python through Minarrow, and continue processing them with Python, native SIMD kernels, or machine-learning libraries without serialising or rebuilding the dataset.

Minarrow PyArrow
Rust–Python data model Native Minarrow types across both runtimes Rust integration through Arrow interchange interfaces
Host runtime Rust, embedded in the application service Arrow C++ runtime loaded into Python
Buffer alignment 64-byte aligned for Minarrow-backed buffers 8-byte Arrow alignment guarantee
Python API Four core containers for flat columnar data Full Arrow type and container hierarchy
Native SIMD use Buffers are ready for aligned SIMD kernels Consumers must inspect alignment or realign the data
Runtime role Application data model, interchange and composition Arrow compute, storage and dataset platform
Ecosystem integration Arrow PyCapsule and direct runtime bridges PyArrow APIs and Arrow interoperability
Dependency footprint Focused Rust extension Full Arrow C++ and Python distribution

Minarrow avoids making PyArrow a required part of the data path. Applications can depend only on Minarrow, hand its objects directly to capsule-aware libraries, and introduce PyArrow only where its broader functionality is useful.

import minarrow as ma

table = ma.Table(
    {
        "id": [1, 2, 3],
        "price": [9.5, 10.0, 11.2],
    }
)

polars_frame = table.to_polars()
duckdb_relation = table.to_duckdb()

PyArrow remains fully interoperable:

import pyarrow as pa

arrow_table = pa.table(table)
minarrow_table = ma.Table.from_arrow(arrow_table)

Rust and Python share the same data model

Minarrow arrays and tables exist on both sides of the Rust–Python boundary.

Rust code can construct the data, perform native processing, and expose it to Python without first serialising it into another format. Python can then pass the same Arrow-compatible buffers into Polars, DuckDB, PyArrow or another capsule-aware library zero-copy.

This suits:

  • Rust-based parsers and decoders
  • Python extensions with columnar outputs
  • Low-latency ingestion systems
  • Scientific and simulation code
  • Feature-generation pipelines
  • Native analytics kernels

This pattern suits systems where Rust provides the production application, transport and data services, while Python delivers the intelligence layer.

SIMD-ready buffers

Minarrow allocates supported buffers on 64-byte boundaries, matching the alignment required by wide SIMD paths such as AVX2 and AVX-512.

For compatible workloads, this adds a low-latency layer of parallelism alongside standard multi-core execution:

  • Native kernels can operate directly on the buffers without copying or realigning the data.
  • The buffers remain in host memory and can be processed immediately by the CPU.
  • GPUs offer greater peak throughput for sufficiently large, highly parallel workloads, but first require the data to be transferred into device memory.
  • For smaller batches and latency-sensitive workloads, the transfer can take longer than the computation itself, making CPU SIMD the faster end-to-end path for many tabular workloads that require a rapid decision.

The alignment is preserved when Minarrow-owned data moves from Rust into Python. Python extensions and Arrow PyCapsule consumers can access the same underlying buffers and pass them directly to compatible native kernels.

By contrast, PyArrow guarantees 8-byte alignment rather than 64-byte alignment. A native kernel that requires 64-byte-aligned input may therefore need to copy the data into a new allocation before processing it. For small and latency-sensitive workloads, that copy can take longer than the SIMD calculation itself.

Small Python surface

Minarrow exposes one container per structural role rather than one Python class for every Arrow type.

array.dtype
array.dtype.group
array.bit_width
array.arrow_type

.dtype provides a compact application-level type, while .arrow_type retains the complete Arrow logical type.

Pluggable execution

Minarrow stores and exchanges columnar data. It does not attempt to replace a dataframe or query engine.

Use Minarrow for the Rust–Python boundary, then choose the execution engine appropriate to the workload:

polars_frame = table.to_polars()
duckdb_relation = table.to_duckdb()
arrow_table = table.to_arrow()

This keeps the data layer small without restricting the rest of the application to a Minarrow-specific API.

Project links

Key dates

PyPI data
Data sourced directly from PyPI's database.
  • Released:
Newer release available (0.16.0)

1 maintainer

PyPI data
Data sourced directly from PyPI's database.
Avatar for spacecell from gravatar.com spacecell

Credits

Author: Peter Bower

License expression

Apache-2.0
View SPDX License List

Requires

Python >=3.9

Provides Extra

dev

Tags

arrow columnar dataframe pyarrow analytics simd

Classifiers

Report project as malware

Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

minarrow-0.1.0.tar.gz (2.4 MB view details)

Uploaded Source

Built Distributions

If you're not sure about the file name format, learn more about wheel file names.

minarrow-0.1.0-cp39-abi3-win_amd64.whl (622.2 kB view details)

Uploaded CPython 3.9+Windows x86-64

minarrow-0.1.0-cp39-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (663.4 kB view details)

Uploaded CPython 3.9+manylinux: glibc 2.17+ x86-64

minarrow-0.1.0-cp39-abi3-manylinux_2_17_aarch64.manylinux2014_aarch64.whl (626.1 kB view details)

Uploaded CPython 3.9+manylinux: glibc 2.17+ ARM64

minarrow-0.1.0-cp39-abi3-macosx_11_0_arm64.whl (612.8 kB view details)

Uploaded CPython 3.9+macOS 11.0+ ARM64

minarrow-0.1.0-cp39-abi3-macosx_10_12_x86_64.whl (658.8 kB view details)

Uploaded CPython 3.9+macOS 10.12+ x86-64

File details

Details for the file minarrow-0.1.0.tar.gz.

File metadata

  • Download URL: minarrow-0.1.0.tar.gz
  • Upload date:
  • Size: 2.4 MB
  • Tags: Source
  • Uploaded using Trusted Publishing? Yes
  • Uploaded via: twine/6.1.0 CPython/3.13.12

File hashes

Hashes for minarrow-0.1.0.tar.gz
Algorithm Hash digest
SHA256 3bb989ada908f61e18e14b0d69181e9a8ec74d7cca676f1b32f6f6c874993378
MD5 700b9ee5ea596b50235c6da0281b4a3f
BLAKE2b-256 9d36c7045891ff38cedf2fdd7b9f09cf1478d8cb1fe08ffb52b785fa6117a54d

See more details on using hashes here.

Provenance

The following attestation bundles were made for minarrow-0.1.0.tar.gz:

Publisher: release.yml on SpaceCell/minarrow-dev

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

File details

Details for the file minarrow-0.1.0-cp39-abi3-win_amd64.whl.

File metadata

  • Download URL: minarrow-0.1.0-cp39-abi3-win_amd64.whl
  • Upload date:
  • Size: 622.2 kB
  • Tags: CPython 3.9+, Windows x86-64
  • Uploaded using Trusted Publishing? Yes
  • Uploaded via: twine/6.1.0 CPython/3.13.12

File hashes

Hashes for minarrow-0.1.0-cp39-abi3-win_amd64.whl
Algorithm Hash digest
SHA256 3d08a1d74aa9ff6cb4ea5889be4ce9836d90805367d484eaefe40e4717d9bd5e
MD5 9fa19d0b05ed9870ebc50630db0c5835
BLAKE2b-256 0913f06b4bc8b9aac8af7c5d67618d5e2fb2fde334d62db1caa85ac3f85aa935

See more details on using hashes here.

Provenance

The following attestation bundles were made for minarrow-0.1.0-cp39-abi3-win_amd64.whl:

Publisher: release.yml on SpaceCell/minarrow-dev

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

File details

Details for the file minarrow-0.1.0-cp39-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.

File metadata

File hashes

Hashes for minarrow-0.1.0-cp39-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm Hash digest
SHA256 4b34627efaa71aa41e9cb6ef6d27c8952f1843be72d8c8c1749681ff27983d31
MD5 a4882c985246ba6ded0735dcfbbc2f96
BLAKE2b-256 608d0b66bfaeeba9e768233670d7eaa0dbfc442f12a2c3f34e59f6a05aee2190

See more details on using hashes here.

Provenance

The following attestation bundles were made for minarrow-0.1.0-cp39-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl:

Publisher: release.yml on SpaceCell/minarrow-dev

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

File details

Details for the file minarrow-0.1.0-cp39-abi3-manylinux_2_17_aarch64.manylinux2014_aarch64.whl.

File metadata

File hashes

Hashes for minarrow-0.1.0-cp39-abi3-manylinux_2_17_aarch64.manylinux2014_aarch64.whl
Algorithm Hash digest
SHA256 0c442aa59650f6669338963f66999b9274d8a6485c7878d245430014721577be
MD5 2d1282f96b6de09771d1208f2f9f9045
BLAKE2b-256 184edb64cfb26d5b8905d540a836d272b540f0c498c817030fcbf6884158cb67

See more details on using hashes here.

Provenance

The following attestation bundles were made for minarrow-0.1.0-cp39-abi3-manylinux_2_17_aarch64.manylinux2014_aarch64.whl:

Publisher: release.yml on SpaceCell/minarrow-dev

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

File details

Details for the file minarrow-0.1.0-cp39-abi3-macosx_11_0_arm64.whl.

File metadata

File hashes

Hashes for minarrow-0.1.0-cp39-abi3-macosx_11_0_arm64.whl
Algorithm Hash digest
SHA256 1432193a5a334c7442751478a1b7d2f847efa5656bfb223b829711980797b143
MD5 4315541a328a1c7c187005433ad488c4
BLAKE2b-256 763f5e9589dd1b3de17f61f5a1f0318c164aaed0e7dffbf42cd78aec647f1e51

See more details on using hashes here.

Provenance

The following attestation bundles were made for minarrow-0.1.0-cp39-abi3-macosx_11_0_arm64.whl:

Publisher: release.yml on SpaceCell/minarrow-dev

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

File details

Details for the file minarrow-0.1.0-cp39-abi3-macosx_10_12_x86_64.whl.

File metadata

File hashes

Hashes for minarrow-0.1.0-cp39-abi3-macosx_10_12_x86_64.whl
Algorithm Hash digest
SHA256 38f7ae95d28313cddfd0ae1416fa2ac631e069b932b5d22d4b1feab1ae8d24b2
MD5 da207006df7717fab09e19f4f300d678
BLAKE2b-256 be2a5967a0b3d05e95043a1650a580c7108127ae6ab4c0223f6b13c67f80000a

See more details on using hashes here.

Provenance

The following attestation bundles were made for minarrow-0.1.0-cp39-abi3-macosx_10_12_x86_64.whl:

Publisher: release.yml on SpaceCell/minarrow-dev

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

Supported by

AWS Cloud computing and Security Sponsor Datadog Monitoring Depot Continuous Integration Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page