Avro to Arrow
This package provides tools to convert from Avro record format to Arrow format.
Installation
Simply install the package using pip:
pip install avro-to-arrow
How-to
In its simplest form, you would use:
from avro_to_arrow import avro_to_arrow
avro_records = ... # list of Avro records
avro_schema = ... # Avro schema as dict
arrow_table, arrow_schema = avro_to_arrow(avro_schema, avro_records)
We provide some additional methods to decode avro & read schema:
from avro_to_arrow.reader import AvroReader
from avro_to_arrow.schema import decode_avro_schema
from avro_to_arrow import avro_to_arrow
avro_records = AvroReader(path_to_avro).to_list()
avro_schema = decode_avro_schema(path_to_schema)
arrow_table, arrow_schema = avro_to_arrow(avro_schema, avro_records)
Performance
Note that the conversion is done per Avro records, and appended to a pyarrow table. Typical performances for large-scale survey alert data are (avro records to pyarrow table):
| Conversion throughput | |
|---|---|
| ZTF (v4.01) | 20 ms/avro |
| LSST (v11) | 150 ms/avro |
This test has been performed using a simple line_profiler. You can reproduce it using:
cd profiling
./profile.sh
To-do
- Vectorize
transform_records/transform_value - Enable iterators for
avro_to_arrowto avoid loading all Avro data in memory - Perform better cycle/memory profiling (simple
line_profilerwas used so far). - Add CLI (avro to parquet)
Metadata
Release files for avro-to-arrow 0.2.0
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | Provenance | Actions |
|---|---|---|---|---|
| avro_to_arrow-0.2.0.tar.gz | 10.3 kB | Without provenance | Details for avro_to_arrow-0.2.0.tar.gz |
Built distribution (wheel)
| File | Interpreter | ABI | Platform | Actions Reset |
|---|---|---|---|---|
| avro_to_arrow-0.2.0-py3-none-any.whl | Python 3 | none | any | Details for avro_to_arrow-0.2.0-py3-none-any.whl |
Total release size: 23.5 kB
Release files / avro_to_arrow-0.2.0.tar.gz
| Download URL | avro_to_arrow-0.2.0.tar.gz |
|---|---|
| Size | 10.3 kB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
ae10c03cd1a167788556ac042607d114151f7a0753e5683024dcf3df2b83705a
|
|
BLAKE2b-256 checksum How to use checksums |
fe8a0c4d00dc91d0dbf5db2efab2250bc652e9a72cd569bebbb6312039d71604
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/7.0.0 CPython/3.11.15
|
Release files / avro_to_arrow-0.2.0-py3-none-any.whl
| Download URL | avro_to_arrow-0.2.0-py3-none-any.whl |
|---|---|
| Size | 13.2 kB |
| Tags | Python 3 |
|
SHA-256 checksum How to use checksums |
e98d4750fc96589c050080c96dc83aaa31f4c9e0cbe2853ec260f4b10714c006
|
|
BLAKE2b-256 checksum How to use checksums |
b86fc0255230c5f6b6836460943ac2a6946d54d6be5126e80bf38f9e3bd06ab4
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/7.0.0 CPython/3.11.15
|
Security for avro-to-arrow 0.2.0
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
No provenance found
None of the files in this release were uploaded with provenance metadata.
A note on package security
PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.
To learn more about protecting yourself from malicious actors, visit our package security documentation.