Skip to main content

Avro to Arrow

This package provides tools to convert from Avro record format to Arrow format.

Installation

Simply install the package using pip:

pip install avro-to-arrow

How-to

In its simplest form, you would use:

from avro_to_arrow import avro_to_arrow

avro_records = ...  # list of Avro records
avro_schema = ...  # Avro schema as dict

arrow_table, arrow_schema = avro_to_arrow(avro_schema, avro_records)

We provide some additional methods to decode avro & read schema:

from avro_to_arrow.reader import AvroReader
from avro_to_arrow.schema import decode_avro_schema
from avro_to_arrow import avro_to_arrow

avro_records = AvroReader(path_to_avro).to_list()
avro_schema = decode_avro_schema(path_to_schema)
arrow_table, arrow_schema = avro_to_arrow(avro_schema, avro_records)

Performance

Note that the conversion is done per Avro records, and appended to a pyarrow table. Typical performances for large-scale survey alert data are (avro records to pyarrow table):

Conversion throughput
ZTF (v4.01) 20 ms/avro
LSST (v11) 150 ms/avro

This test has been performed using a simple line_profiler. You can reproduce it using:

cd profiling
./profile.sh

To-do

  • Vectorize transform_records/transform_value
  • Enable iterators for avro_to_arrow to avoid loading all Avro data in memory
  • Perform better cycle/memory profiling (simple line_profiler was used so far).
  • Add CLI (avro to parquet)

Metadata

Release files for avro-to-arrow 0.2.0

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for avro-to-arrow 0.2.0
File Size Uploaded Provenance Actions
avro_to_arrow-0.2.0.tar.gz 10.3 kB Without provenance Details for avro_to_arrow-0.2.0.tar.gz

Built distribution (wheel)

Table of built distributions (wheels) for avro-to-arrow 0.2.0
File Interpreter ABI Platform Actions
avro_to_arrow-0.2.0-py3-none-any.whl Python 3 none any Details for avro_to_arrow-0.2.0-py3-none-any.whl

Total release size: 23.5 kB

Release files / avro_to_arrow-0.2.0.tar.gz

Download URL avro_to_arrow-0.2.0.tar.gz
Size 10.3 kB
Tags Source
SHA-256 checksum
How to use checksums
ae10c03cd1a167788556ac042607d114151f7a0753e5683024dcf3df2b83705a
BLAKE2b-256 checksum
How to use checksums
fe8a0c4d00dc91d0dbf5db2efab2250bc652e9a72cd569bebbb6312039d71604
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via twine/7.0.0 CPython/3.11.15

Release files / avro_to_arrow-0.2.0-py3-none-any.whl

Download URL avro_to_arrow-0.2.0-py3-none-any.whl
Size 13.2 kB
Tags Python 3
SHA-256 checksum
How to use checksums
e98d4750fc96589c050080c96dc83aaa31f4c9e0cbe2853ec260f4b10714c006
BLAKE2b-256 checksum
How to use checksums
b86fc0255230c5f6b6836460943ac2a6946d54d6be5126e80bf38f9e3bd06ab4
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via twine/7.0.0 CPython/3.11.15

Security for avro-to-arrow 0.2.0

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

No provenance found

None of the files in this release were uploaded with provenance metadata.

A note on package security

PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.

To learn more about protecting yourself from malicious actors, visit our package security documentation.

Release history Release notifications | RSS feed

This release

0.2.0 This release

2 release files No provenance

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page