polars-avro

build pypi docs

A polars io plugin for reading and writing Apache Avro files, built on arrow-avro. It provides scan support with predicate pushdown, map type reading, and continued avro support as polars deprecates its built-in implementation.

Python Usage

from polars_avro import scan_avro, read_avro, write_avro

lazy = scan_avro(path)
frame = read_avro(path)
write_avro([frame], path)

Both scan_avro and read_avro accept cloud and other URLs (s3://, gs://, az://, http(s)://, ...), read through fsspec — install the relevant backend (e.g. s3fs) and pass storage_options (forwarded to fsspec.open) for credentials:

lazy = scan_avro("s3://bucket/data.avro", storage_options={"anon": True})
frame = read_avro("s3://bucket/data.avro", storage_options={"anon": True})

Rust Usage

There are two main exports: [Reader] for iterating arrow RecordBatches from avro sources, and [Writer] for writing RecordBatches to an avro file.

use polars_avro::{FullReadOptions, Reader, Writer};
use std::fs::File;

// `Reader` yields arrow `RecordBatch`es from one or more avro sources
let mut reader = Reader::try_new(
    [File::open("data.avro")],
    FullReadOptions::default(),
).unwrap();

// copy them into a new file; `Writer` needs a schema up front, so take it
// from the first batch
let first = reader.next().unwrap().unwrap();
let mut writer = Writer::try_new(
    File::create("copy.avro").unwrap(),
    first.schema(),
    None,
).unwrap();
writer.write(&first).unwrap();
for batch in reader {
    writer.write(&batch.unwrap()).unwrap();
}
writer.finish().unwrap();

ℹ️ Avro supports writing with file compression schemes. In rust these need to be enabled via feature flags: deflate, snappy, bzip2, xz, zstd. Decompression is handled automatically.

Idiosyncrasies

Avro and Arrow don't align fully, and polars only supports a subset of arrow. Some types require casting before writing, and some avro types map to different polars types than you might expect when reading.

Writing

The following polars types error when writing and must be cast first:

Polars Type Cast To
large UInt64 Wrap to Int64
Categorical Int32 or String
Enum Int32 or String

Times will get truncated to micro seconds.

Compression is supported via feature flags: deflate, snappy, bzip2, xz, zstd.

Reading

utf8_view behavior — the utf8_view option (default false) changes how certain types are read:

Type utf8_view=false (default) utf8_view=true
UUID binary (16 bytes) formatted string
nullable strings preserves nulls replaces null with "" (lossy)

Since polars tends to work with string views internally, utf8_view=true is likely faster if you don't mind losing null string distinctions.

Type mappings of note:

Avro Type Polars Type
Enum Categorical (not Enum)
Map List of Struct {key, value}
BigDecimal Binary
Duration unsupported (errors)
Date Date (days since epoch)
TimeMillis, TimeMicros Time (nanoseconds)
TimestampMillis/Micros/Nanos Datetime with matching precision and UTC tz
LocalTimestampMillis/Micros/Nanos Datetime with matching precision and no tz

Constraints: the root avro schema must be a Record, and all files in a multi-file read must share the same schema.

Benchmarks

Python reports median (file reads, in-memory writes). Rust reports mean. native = polars built-in avro. Ratio relative to native; bold = fastest. Complex rows use nested/struct types.

Benchmark native polars-avro jetliner
python read 1K × 2 64 µs (1.00x) 99 µs (1.54x) 180 µs (2.79x)
python read 64K × 2 2.7 ms (1.00x) 2.1 ms (0.78x) 2.8 ms (1.04x)
python read 1K × 8 183 µs (1.00x) 242 µs (1.32x) 337 µs (1.84x)
python read 1M × 8 159 ms (1.00x) 114 ms (0.72x) 145 ms (0.91x)
python read 1M × 128 2.6 s (1.00x) 1.8 s (0.69x) 2.8 s (1.09x)
python read complex 1K × 8 — 449 µs 592 µs
python read complex 1M × 8 — 181 ms 260 ms
python read proj 1M × 128 → 8 1.6 s (1.00x) 1.2 s (0.75x) 1.2 s (0.77x)
python read proj 1K × 8 → 2 133 µs (1.00x) 297 µs (2.24x) 264 µs (1.99x)
python write 1K × 2 42 µs (1.00x) 30 µs (0.72x) —
python write 64K × 2 1.5 ms (1.00x) 1.1 ms (0.71x) —
python write 1K × 8 143 µs (1.00x) 114 µs (0.80x) —
python write 1M × 8 87 ms (1.00x) 93 ms (1.07x) —
python write 1M × 128 1.5 s (1.00x) 2.2 s (1.48x) —
rust read 1K × 2 42 µs (1.00x) 34 µs (0.80x) —
rust read 1M × 128 2.8 s (1.00x) 2.0 s (0.69x) —
rust read proj 1M × 128 → 8 1.3 s (1.00x) 1.2 s (0.87x) —
rust read proj 1K × 8 → 2 109 µs (1.00x) 116 µs (1.06x) —
rust write 1K × 2 42 µs (1.00x) 22 µs (0.53x) —
rust write 64K × 2 1.5 ms (1.00x) 1.0 ms (0.67x) —
rust write 1K × 8 135 µs (1.00x) 93 µs (0.69x) —
rust write 1M × 8 97 ms (1.00x) 89 ms (0.92x) —
rust write 1M × 128 1.6 s (1.00x) 1.4 s (0.88x) —

Development

Rust

Standard cargo commands will build and test the rust library.

Python

The python library is built with uv and maturin. The rust components should build once, ance otherwise allow usage and testing.

You may need to recompile the python bindings with uv run maturin develop.

Testing

cargo fmt --check
cargo clippy --all-features --tests
cargo test
uv run ruff format --check
uv run ruff check
uv run pyright
uv run pytest

Benchmarking

Benchmarks must run against an optimized build. maturin develop (used for normal testing) compiles the extension unoptimized, which makes the Python benchmarks meaningless (~20x slower). Build release first:

cargo +nightly bench
uv run maturin develop --release
uv run pytest --benchmark-only

Releasing

rm -rf dist
uv build --sdist
uv run maturin build -r -o dist --target aarch64-apple-darwin
uv run maturin build -r -o dist --target aarch64-unknown-linux-gnu --zig
uv publish --username __token__

To Do

  • reimplement single column reader?
  • reimplement better workarounds for types that don't exist, e.g. serialize polars cat/enum to arrow enum and vice versa

Key dates

PyPI data
Data sourced directly from PyPI's database.
  • Released:
Latest release

Owner

PyPI data
Data sourced directly from PyPI's database.

Requires

Python >=3.10

Report project as malware

Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

polars_avro-0.11.0.tar.gz (160.6 kB view details)

Uploaded Source

Built Distributions

If you're not sure about the file name format, learn more about wheel file names.

polars_avro-0.11.0-cp310-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (4.8 MB view details)

Uploaded CPython 3.10+manylinux: glibc 2.17+ x86-64

polars_avro-0.11.0-cp310-abi3-manylinux_2_17_aarch64.manylinux2014_aarch64.whl (4.6 MB view details)

Uploaded CPython 3.10+manylinux: glibc 2.17+ ARM64

polars_avro-0.11.0-cp310-abi3-macosx_11_0_arm64.whl (4.0 MB view details)

Uploaded CPython 3.10+macOS 11.0+ ARM64

File details

Details for the file polars_avro-0.11.0.tar.gz.

File metadata

  • Download URL: polars_avro-0.11.0.tar.gz
  • Upload date:
  • Size: 160.6 kB
  • Tags: Source
  • Uploaded using Trusted Publishing? Yes
  • Uploaded via: twine/6.1.0 CPython/3.13.12

File hashes

Hashes for polars_avro-0.11.0.tar.gz
Algorithm Hash digest
SHA256 d07ad6c56213606fca5094379f3722cc4e57e91a19d87b8d5150b398768f99af
MD5 cf0aaa66857bb5e5969c08ffe27b6885
BLAKE2b-256 169566cea94eba599229d2e26a423f49ec1fd8d6c1d3612f993db6dde72c37b4

See more details on using hashes here.

Provenance

The following attestation bundles were made for polars_avro-0.11.0.tar.gz:

Publisher: release.yml on hafaio/polars-avro

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

File details

Details for the file polars_avro-0.11.0-cp310-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.

File metadata

File hashes

Hashes for polars_avro-0.11.0-cp310-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm Hash digest
SHA256 95bfd016137c53bb231b9fc139489c6f7a6967b5327f9d2f55f709794fc2a14f
MD5 53b18bc4f8a42f1e508a7f072030f502
BLAKE2b-256 d696d429e963eda85fd8a744e82f49acec879697f7e176b93c3075706491eb9b

See more details on using hashes here.

Provenance

The following attestation bundles were made for polars_avro-0.11.0-cp310-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl:

Publisher: release.yml on hafaio/polars-avro

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

File details

Details for the file polars_avro-0.11.0-cp310-abi3-manylinux_2_17_aarch64.manylinux2014_aarch64.whl.

File metadata

File hashes

Hashes for polars_avro-0.11.0-cp310-abi3-manylinux_2_17_aarch64.manylinux2014_aarch64.whl
Algorithm Hash digest
SHA256 5d4bf38ca615c5e20db0c6061aa0ed619c9530a23fbe212bbcb279587c8fd7b4
MD5 e0861e611545381ad153ff057e04a6eb
BLAKE2b-256 653b9288ffcbac3e12bf56aef8d3a0c21b31b8e66a64dd3c4884f09433866b02

See more details on using hashes here.

Provenance

The following attestation bundles were made for polars_avro-0.11.0-cp310-abi3-manylinux_2_17_aarch64.manylinux2014_aarch64.whl:

Publisher: release.yml on hafaio/polars-avro

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

File details

Details for the file polars_avro-0.11.0-cp310-abi3-macosx_11_0_arm64.whl.

File metadata

File hashes

Hashes for polars_avro-0.11.0-cp310-abi3-macosx_11_0_arm64.whl
Algorithm Hash digest
SHA256 f2756fe8ed79c73e6c0607dd18e49b577e90cf7872474b3fd33c226fd2cf0caa
MD5 0c28194657a6fc6f1a449be4a197bf7f
BLAKE2b-256 63fb0aca4982914f2c1ec36ac67491d7b3cc574ac9b4e20a5a0eec842304be65

See more details on using hashes here.

Provenance

The following attestation bundles were made for polars_avro-0.11.0-cp310-abi3-macosx_11_0_arm64.whl:

Publisher: release.yml on hafaio/polars-avro

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

Supported by

AWS Cloud computing and Security Sponsor Datadog Monitoring Depot Continuous Integration Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page