polars-avro

build pypi docs

A polars io plugin for reading and writing Apache Avro files, built on arrow-avro. It provides scan support with predicate pushdown, map type reading, and continued avro support as polars deprecates its built-in implementation.

Python Usage

from polars_avro import scan_avro, read_avro, write_avro

lazy = scan_avro(path)
frame = read_avro(path)
write_avro([frame], path)

Both scan_avro and read_avro accept cloud and other URLs (s3://, gs://, az://, http(s)://, ...), read through fsspec — install the relevant backend (e.g. s3fs) and pass storage_options (forwarded to fsspec.open) for credentials:

lazy = scan_avro("s3://bucket/data.avro", storage_options={"anon": True})
frame = read_avro("s3://bucket/data.avro", storage_options={"anon": True})

Rust Usage

There are two main exports: [Reader] for iterating DataFrames from avro sources, and [Writer] for writing DataFrames to an avro file.

use polars_avro::{Reader, Writer, ReadOptions};

// read
let reader = Reader::try_new(
    [File::open("data.avro")],
    ReadOptions::basic(),
).unwrap();
for batch in reader {
    let frame = batch.unwrap();
}

// write
let mut writer = Writer::try_new(file, frame.schema(), None).unwrap();
writer.write(&frame).unwrap();

ℹ️ Avro supports writing with file compression schemes. In rust these need to be enabled via feature flags: deflate, snappy, bzip2, xz, zstd. Decompression is handled automatically.

Idiosyncrasies

Avro and Arrow don't align fully, and polars only supports a subset of arrow. Some types require casting before writing, and some avro types map to different polars types than you might expect when reading.

Writing

The following polars types error when writing and must be cast first:

Polars Type Cast To
large UInt64 Wrap to Int64
Categorical Int32 or String
Enum Int32 or String

Times will get truncated to micro seconds.

Compression is supported via feature flags: deflate, snappy, bzip2, xz, zstd.

Reading

utf8_view behavior — the utf8_view option (default false) changes how certain types are read:

Type utf8_view=false (default) utf8_view=true
UUID binary (16 bytes) formatted string
nullable strings preserves nulls replaces null with "" (lossy)

Since polars tends to work with string views internally, utf8_view=true is likely faster if you don't mind losing null string distinctions.

Type mappings of note:

Avro Type Polars Type
Enum Categorical (not Enum)
Map List of Struct {key, value}
BigDecimal Binary
Duration unsupported (errors)
Date Date (days since epoch)
TimeMillis, TimeMicros Time (nanoseconds)
TimestampMillis/Micros/Nanos Datetime with matching precision and UTC tz
LocalTimestampMillis/Micros/Nanos Datetime with matching precision and no tz

Constraints: the root avro schema must be a Record, and all files in a multi-file read must share the same schema.

Benchmarks

Python reports median (file reads, in-memory writes). Rust reports mean. native = polars built-in avro. Ratio relative to native; bold = fastest. Complex rows use nested/struct types.

Benchmark native polars-avro jetliner
python read 1K × 2 64 µs (1.00x) 99 µs (1.54x) 180 µs (2.79x)
python read 64K × 2 2.7 ms (1.00x) 2.1 ms (0.78x) 2.8 ms (1.04x)
python read 1K × 8 183 µs (1.00x) 242 µs (1.32x) 337 µs (1.84x)
python read 1M × 8 159 ms (1.00x) 114 ms (0.72x) 145 ms (0.91x)
python read 1M × 128 2.6 s (1.00x) 1.8 s (0.69x) 2.8 s (1.09x)
python read complex 1K × 8 — 449 µs 592 µs
python read complex 1M × 8 — 181 ms 260 ms
python read proj 1M × 128 → 8 1.6 s (1.00x) 1.2 s (0.75x) 1.2 s (0.77x)
python read proj 1K × 8 → 2 133 µs (1.00x) 297 µs (2.24x) 264 µs (1.99x)
python write 1K × 2 42 µs (1.00x) 30 µs (0.72x) —
python write 64K × 2 1.5 ms (1.00x) 1.1 ms (0.71x) —
python write 1K × 8 143 µs (1.00x) 114 µs (0.80x) —
python write 1M × 8 87 ms (1.00x) 93 ms (1.07x) —
python write 1M × 128 1.5 s (1.00x) 2.2 s (1.48x) —
rust read 1K × 2 42 µs (1.00x) 34 µs (0.80x) —
rust read 1M × 128 2.8 s (1.00x) 2.0 s (0.69x) —
rust read proj 1M × 128 → 8 1.3 s (1.00x) 1.2 s (0.87x) —
rust read proj 1K × 8 → 2 109 µs (1.00x) 116 µs (1.06x) —
rust write 1K × 2 42 µs (1.00x) 22 µs (0.53x) —
rust write 64K × 2 1.5 ms (1.00x) 1.0 ms (0.67x) —
rust write 1K × 8 135 µs (1.00x) 93 µs (0.69x) —
rust write 1M × 8 97 ms (1.00x) 89 ms (0.92x) —
rust write 1M × 128 1.6 s (1.00x) 1.4 s (0.88x) —

Development

Rust

Standard cargo commands will build and test the rust library.

Python

The python library is built with uv and maturin. The rust components should build once, ance otherwise allow usage and testing.

You may need to recompile the python bindings with uv run maturin develop.

Testing

cargo fmt --check
cargo clippy --all-features --tests
cargo test
uv run ruff format --check
uv run ruff check
uv run pyright
uv run pytest

Benchmarking

Benchmarks must run against an optimized build. maturin develop (used for normal testing) compiles the extension unoptimized, which makes the Python benchmarks meaningless (~20x slower). Build release first:

cargo +nightly bench
uv run maturin develop --release
uv run pytest --benchmark-only

Releasing

rm -rf dist
uv build --sdist
uv run maturin build -r -o dist --target aarch64-apple-darwin
uv run maturin build -r -o dist --target aarch64-unknown-linux-gnu --zig
uv publish --username __token__

To Do

  • reimplement single column reader?
  • reimplement better workarounds for types that don't exist, e.g. serialize polars cat/enum to arrow enum and vice versa

Key dates

PyPI data
Data sourced directly from PyPI's database.
  • Released:
Newer release available (0.11.0)

Owner

PyPI data
Data sourced directly from PyPI's database.

Requires

Python >=3.10

Report project as malware

Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

polars_avro-0.10.1.tar.gz (157.5 kB view details)

Uploaded Source

Built Distributions

If you're not sure about the file name format, learn more about wheel file names.

polars_avro-0.10.1-cp310-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (4.7 MB view details)

Uploaded CPython 3.10+manylinux: glibc 2.17+ x86-64

polars_avro-0.10.1-cp310-abi3-manylinux_2_17_aarch64.manylinux2014_aarch64.whl (4.6 MB view details)

Uploaded CPython 3.10+manylinux: glibc 2.17+ ARM64

polars_avro-0.10.1-cp310-abi3-macosx_11_0_arm64.whl (4.0 MB view details)

Uploaded CPython 3.10+macOS 11.0+ ARM64

File details

Details for the file polars_avro-0.10.1.tar.gz.

File metadata

  • Download URL: polars_avro-0.10.1.tar.gz
  • Upload date:
  • Size: 157.5 kB
  • Tags: Source
  • Uploaded using Trusted Publishing? Yes
  • Uploaded via: twine/6.1.0 CPython/3.13.12

File hashes

Hashes for polars_avro-0.10.1.tar.gz
Algorithm Hash digest
SHA256 92b13a5eb55f32e2ad09767d8eec9ac5a090427a92baaca9110e2e6bcb63055b
MD5 4a93c0e251153e1b1c75e0c957f57f38
BLAKE2b-256 a1f6cbd037b762c61c26d912918b896a96cf02b1b0040b4dd7865044022edbdc

See more details on using hashes here.

Provenance

The following attestation bundles were made for polars_avro-0.10.1.tar.gz:

Publisher: release.yml on hafaio/polars-avro

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

File details

Details for the file polars_avro-0.10.1-cp310-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.

File metadata

File hashes

Hashes for polars_avro-0.10.1-cp310-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm Hash digest
SHA256 1edd257219152ac92452c0b77a55b9a6615172d7aca5c62d10ebbdf86eb946de
MD5 4c4b2c6f41bef595bbcfb723a6edf17f
BLAKE2b-256 e754b86e03f4e9e29d8f607b7fa10fa2a98cbf5cbf3027dfcc558cb03390b69e

See more details on using hashes here.

Provenance

The following attestation bundles were made for polars_avro-0.10.1-cp310-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl:

Publisher: release.yml on hafaio/polars-avro

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

File details

Details for the file polars_avro-0.10.1-cp310-abi3-manylinux_2_17_aarch64.manylinux2014_aarch64.whl.

File metadata

File hashes

Hashes for polars_avro-0.10.1-cp310-abi3-manylinux_2_17_aarch64.manylinux2014_aarch64.whl
Algorithm Hash digest
SHA256 75ebf16507bb8f40de5012c70c67c1a713803f28d84e50c5f32d1ee82134484e
MD5 b76a715db3ae4bf4238ffc570a9172e4
BLAKE2b-256 c94e4445cfcbac368edb8df5aa7b59b7fe4f5849080c647b9b71a3583ea5f492

See more details on using hashes here.

Provenance

The following attestation bundles were made for polars_avro-0.10.1-cp310-abi3-manylinux_2_17_aarch64.manylinux2014_aarch64.whl:

Publisher: release.yml on hafaio/polars-avro

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

File details

Details for the file polars_avro-0.10.1-cp310-abi3-macosx_11_0_arm64.whl.

File metadata

File hashes

Hashes for polars_avro-0.10.1-cp310-abi3-macosx_11_0_arm64.whl
Algorithm Hash digest
SHA256 a0a0aa378b93f3f756fb8ee5e6f4e2b58ce9297c2e0636192a62b12305300e3a
MD5 b2faa1d16703fc7113fc571fc4b6654b
BLAKE2b-256 c085ae7fee977b39d59683519b907df7c2ef3a2900aeefa2703fc7b77654aa3b

See more details on using hashes here.

Provenance

The following attestation bundles were made for polars_avro-0.10.1-cp310-abi3-macosx_11_0_arm64.whl:

Publisher: release.yml on hafaio/polars-avro

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

Supported by

AWS Cloud computing and Security Sponsor Datadog Monitoring Depot Continuous Integration Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page