Skip to main content

Avro to Arrow

This package provides tools to convert from Avro record format to Arrow format.

Installation

Simply install the package using pip:

pip install avro-to-arrow

How-to

In its simplest form, you would use:

from avro_to_arrow import avro_to_arrow

avro_records = ...  # list of Avro records
avro_schema = ...  # Avro schema as dict

arrow_table, arrow_schema = avro_to_arrow(avro_schema, avro_records)

We provide some additional methods to decode avro & read schema:

from avro_to_arrow.reader import AvroReader
from avro_to_arrow.schema import decode_avro_schema
from avro_to_arrow import avro_to_arrow

avro_records = AvroReader(path_to_avro).to_list()
avro_schema = decode_avro_schema(path_to_schema)
arrow_table, arrow_schema = avro_to_arrow(avro_schema, avro_records)

Performance

Note that the conversion is done per Avro records, and appended to a pyarrow table. Typical performances for large-scale survey alert data are (avro records to pyarrow table):

Conversion throughput
ZTF (v4.01) 20 ms/avro
LSST (v11) 150 ms/avro

This test has been performed using a simple line_profiler. You can reproduce it using:

cd profiling
./profile.sh

To-do

  • Vectorize transform_records/transform_value
  • Enable iterators for avro_to_arrow to avoid loading all Avro data in memory
  • Perform better cycle/memory profiling (simple line_profiler was used so far).
  • Add CLI (avro to parquet)

Metadata

Release files for avro-to-arrow 0.1.0

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for avro-to-arrow 0.1.0
File Size Uploaded Provenance Actions
avro_to_arrow-0.1.0.tar.gz 10.3 kB Without provenance Details for avro_to_arrow-0.1.0.tar.gz

Built distribution (wheel)

Table of built distributions (wheels) for avro-to-arrow 0.1.0
File Interpreter ABI Platform Actions
avro_to_arrow-0.1.0-py3-none-any.whl Python 3 none any Details for avro_to_arrow-0.1.0-py3-none-any.whl

Total release size: 23.6 kB

Release files / avro_to_arrow-0.1.0.tar.gz

Download URL avro_to_arrow-0.1.0.tar.gz
Size 10.3 kB
Tags Source
SHA-256 checksum
How to use checksums
a7f824f359fe84560d0e975deedfd5e7f62695702ff6507a643939d1bdd57721
BLAKE2b-256 checksum
How to use checksums
d49c5539be9ed7ca7feea62d9d3dd854b1f20bce905fbc7f77e559317d9d30d1
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via twine/7.0.0 CPython/3.11.15

Release files / avro_to_arrow-0.1.0-py3-none-any.whl

Download URL avro_to_arrow-0.1.0-py3-none-any.whl
Size 13.3 kB
Tags Python 3
SHA-256 checksum
How to use checksums
0f3f35bc929a70363dc02fc3a53796818f55c9aa99c30b077b1f1722a5b30b11
BLAKE2b-256 checksum
How to use checksums
766e184b9691c4766ca01897f1e1650604e2dc11f507e89b46d08a1289d97a6e
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via twine/7.0.0 CPython/3.11.15

Security for avro-to-arrow 0.1.0

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

No provenance found

None of the files in this release were uploaded with provenance metadata.

A note on package security

PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.

To learn more about protecting yourself from malicious actors, visit our package security documentation.

Release history Release notifications | RSS feed

This release

0.1.0 This release

2 release files No provenance

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page