Skip to main content
Help the Python Software Foundation raise $60,000 USD by December 31st!  Building the PSF Q4 Fundraiser

Avro schema and data converters supporting storing arbitrary nested python data structures.

Project description


Avro support for JSON and other nested data structures.

Rec-avro provides a generic Avro schema and converter functions that allow for storing nested python data structures in avro.

Tested in Python 3 only.


$ pip3 install rec-avro


With fastavro:

from fastavro import writer, reader, schema
from rec_avro import to_rec_avro_destructive, from_rec_avro_destructive, rec_avro_schema

def json_objects():
    return [{'a': 'a'}, {'b':'b'}]

# For efficiency, to_rec_avro_destructive() destroys rec, and reuses it's
# data structures to construct avro_objects 
avro_objects = (to_rec_avro_destructive(rec) for rec in json_objects())

# store records in avro
with open('json_in_avro.avro', 'wb') as f_out:
    writer(f_out, schema.parse_schema(rec_avro_schema()), avro_objects)

#load records from avro
with open('json_in_avro.avro', 'rb') as f_in:
    # For efficiency, from_rec_avro_destructive(rec) destroys rec, and 
    # reuses it's data structures to construct it's output
    loaded_json = [from_rec_avro_destructive(rec) for rec in reader(f_in)]

assert loaded_json == json_objects()


# Running all tests:
$ python pytest

# Running tests manually
$ pip3 install fastavro pytest
$ python develop
$ pytest tests/

Project details

Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Files for rec-avro, version 0.0.4
Filename, size File type Python version Upload date Hashes
Filename, size rec_avro-0.0.4-py3-none-any.whl (4.1 kB) File type Wheel Python version py3 Upload date Hashes View
Filename, size rec-avro-0.0.4.tar.gz (3.0 kB) File type Source Python version None Upload date Hashes View

Supported by

Pingdom Pingdom Monitoring Google Google Object Storage and Download Analytics Sentry Sentry Error logging AWS AWS Cloud computing DataDog DataDog Monitoring Fastly Fastly CDN DigiCert DigiCert EV certificate StatusPage StatusPage Status page