polars-avro

A polars io plugin for reading and writing Apache Avro files, built on arrow-avro. It provides scan support with predicate pushdown, map type reading, and continued avro support as polars deprecates its built-in implementation.

Python Usage

from polars_avro import scan_avro, read_avro, write_avro

lazy = scan_avro(path)
frame = read_avro(path)
write_avro([frame], path)

Both scan_avro and read_avro accept cloud and other URLs (s3://, gs://, az://, http(s)://, ...), read through fsspec — install the relevant backend (e.g. s3fs) and pass storage_options (forwarded to fsspec.open) for credentials:

lazy = scan_avro("s3://bucket/data.avro", storage_options={"anon": True})
frame = read_avro("s3://bucket/data.avro", storage_options={"anon": True})

Rust Usage

There are two main exports: [Reader] for iterating arrow RecordBatches from avro sources, and [Writer] for writing RecordBatches to an avro file.

use polars_avro::{FullReadOptions, Reader, Writer};
use std::fs::File;

// `Reader` yields arrow `RecordBatch`es from one or more avro sources
let mut reader = Reader::try_new(
    [File::open("data.avro")],
    FullReadOptions::default(),
).unwrap();

// copy them into a new file; `Writer` needs a schema up front, so take it
// from the first batch
let first = reader.next().unwrap().unwrap();
let mut writer = Writer::try_new(
    File::create("copy.avro").unwrap(),
    first.schema(),
    None,
).unwrap();
writer.write(&first).unwrap();
for batch in reader {
    writer.write(&batch.unwrap()).unwrap();
}
writer.finish().unwrap();

ℹ️ Avro supports writing with file compression schemes. In rust these need to be enabled via feature flags: deflate, snappy, bzip2, xz, zstd. Decompression is handled automatically.

Idiosyncrasies

Avro and Arrow don't align fully, and polars only supports a subset of arrow. Some types require casting before writing, and some avro types map to different polars types than you might expect when reading.

Writing

The following polars types error when writing and must be cast first:

Times will get truncated to micro seconds.

Compression is supported via feature flags: deflate, snappy, bzip2, xz, zstd.

Reading

utf8_view behavior — the utf8_view option (default false) changes how certain types are read:

Since polars tends to work with string views internally, utf8_view=true is likely faster if you don't mind losing null string distinctions.

Type mappings of note:

Constraints: the root avro schema must be a Record, and all files in a multi-file read must share the same schema.

Benchmarks

Python reports median (file reads, in-memory writes). Rust reports mean. native = polars built-in avro. Ratio relative to native; bold = fastest. Complex rows use nested/struct types.

Development

Rust

Standard cargo commands will build and test the rust library.

Python

The python library is built with uv and maturin. uv sync compiles the rust extension once, after which you can use and test the library from python.

You may need to recompile the python bindings with uv run maturin develop.

Testing

cargo fmt --check
cargo clippy --all-features --tests -- -D warnings
cargo test
uv run ruff format --check
uv run ruff check
uv run pyright
uv run pytest

Benchmarking

Benchmarks must run against an optimized build. maturin develop (used for normal testing) compiles the extension unoptimized, which makes the Python benchmarks meaningless (~20x slower). Build release first:

cargo +nightly bench
uv run maturin develop --release
uv run pytest --benchmark-only

Releasing

Releases are automated. Trigger the release workflow from the Actions tab (or gh workflow run release.yml), choosing the version bump (patch, minor, or major). It gates on the full test suite, builds wheels and an sdist, commits and tags the version bump, and publishes to PyPI via OIDC trusted publishing.