Skip to main content

avro-to-python

avro-to-python is a light tool for compiling avro schema files (.avsc) to python classes making using avro schemata easy.

Installation

Install From Source ()

The sources for avro-to-python can be downloaded source as well.

Clone the public repository:

$ git clone git://github.com/srserves85/avro-to-python

Once you have a copy of the source, you can install it with:

$ python setup.py install

or

$ pip install -e .

Examples

Majority of the use of avro-to-python is assumed to be used as a cli, but you can still import and use the python classes under the hood as well.

CLI (without –pip)

To use the cli, here is the available cli commands:

avro-to-python [source] [target]
    Options:
        --pip TEXT              make package pip installable using this name
        --author TEXT           author name of the pip installable package
        --package_version TEXT  version of the pip intallable package  [default: 0.1.0]
        --help                  Show this message and exit

The above will compile the avsc files and convert the to python classes found in [path_to_target_directory]

An example of doing this is the following:

avro-to-python [path_to_source_avsc_files] [path_to_target_directory]

If you run the above on a valid avro avsc file, you should then be able to import them as you would in the avro idl namespace Here is an example of a single avsc record from the namespace: name.space and name: RecordClass:

from name.space import RecordClass

record = RecordClass({'foo': True, 'bar': 'true', 'baz': 10, 'food': 'CHOCOLATE'})

CLI (with –pip)

You can also choose to make compiled avro packages *pip installable* by adding the “–pip” flags. An example of this is the following: .. code-block:: bash

avro-to-python [path_to_source_avsc_files] [path_to_target_directory] –pip test_avro

By running this, you should be able to pip install the above package you created from the target directory you specified by running:

pip install -e path_to_target_directory

Now that you have the package installed, you can import it by it’s package name and namespace. Here is the same example of the same avsc from above, only with a pip package of test_avro:

from test_avro.name.space import RecordClass

record = RecordClass({'foo': True, 'bar': 'true', 'baz': 10, 'food': 'CHOCOLATE'})

avro-to-python in a Script

You can also use the reader and writer packages in avro-to-python as you would any other python package. Avro to python is split between a reader and writer classes. avro-to-python treates namespaces as acyclic trees and uses depth first search to ensure no duplication or namespace collisions on read and write. An example useage is below:

from avro_to_python.reader import AvscReader
from avro_to_python.writer import AvroWriter

# initialize the reader object
reader = AvscReader(directory='tests/avsc/records/')

# generate the acyclic tree object
reader.read()

# initialize the writer object
writer = AvroWriter(reader.file_tree, pip='test_pip')

# compile python files using 'tests/test_records as the namespace root'
writer.write(root_dir='tests/test_records')

Roadmap

Reader

  • [X] Create Namespace Trees on nested namespaces

  • [X] Read Record and Enum File

  • [X] Primitive types

  • [X] Array Types

  • [X] Union types

  • [X] References to other files

  • [X] Map Types

  • [ ] Logical Types (Currently just converts to primitive types)

Writer

  • [X] Base Schema Writer

  • [X] Base Record Schema

  • [X] Base Enum Schema

  • [X] Primitive Types Schema

  • [X] Array Types Schema

  • [X] Union Types Schema

  • [X] Map Types

  • [ ] Logical Types Schema (Currently just converts to primitive types)

  • [X] Add configs to pip install package

CLI

  • [X] Wrap Writer and Reader into one cli commmit

  • [X] Add pip install option (would include all files to pip install compiled package)

  • [ ] Add better –help documentation

Documentation

  • [ ] Document reader class

  • [ ] Document writer class

  • [ ] Document cli

History

0.3.3 (2022-01-26)

  • Fix array reference handling.

0.3.2 (2021-11-16)

  • Add map type to unions.

0.3.1 (2020-06-10)

  • First release on PyPI.

Metadata

Release files for avro-to-python 0.3.3

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for avro-to-python 0.3.3
File Size Uploaded Provenance Actions
avro_to_python-0.3.3.tar.gz 38.1 kB Without provenance Details for avro_to_python-0.3.3.tar.gz

Built distribution (wheel)

Table of built distributions (wheels) for avro-to-python 0.3.3
File Interpreter ABI Platform Actions
avro_to_python-0.3.3-py2.py3-none-any.whl Python 2, Python 3 none any Details for avro_to_python-0.3.3-py2.py3-none-any.whl

Total release size: 71.1 kB

Release files / avro_to_python-0.3.3.tar.gz

Download URL avro_to_python-0.3.3.tar.gz
Size 38.1 kB
Tags Source
SHA-256 checksum
How to use checksums
bdeb1b697c40a405409f7add4d4ff5c1d8f58b9ce4580adfa9c60dc4a33a86f7
BLAKE2b-256 checksum
How to use checksums
65bf1592b7de3c4573adc1b23a1aa3aa080c5cf6a23ec28143ee6db4694a149b
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via twine/1.14.0 pkginfo/1.7.1 requests/2.26.0 setuptools/49.2.1 requests-toolbelt/0.9.1 tqdm/4.62.3 CPython/3.8.7

Release files / avro_to_python-0.3.3-py2.py3-none-any.whl

Download URL avro_to_python-0.3.3-py2.py3-none-any.whl
Size 33.0 kB
Tags Python 2 Python 3
SHA-256 checksum
How to use checksums
50e0d986a3ade6427dfde4db4350f0b8073243640aa750474da6f2d1ebbf1f97
BLAKE2b-256 checksum
How to use checksums
e2c23f60e2754a5d08e152baf8c93dfa43e48b1a8c3df8de4521ec749c1f4eaa
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via twine/1.14.0 pkginfo/1.7.1 requests/2.26.0 setuptools/49.2.1 requests-toolbelt/0.9.1 tqdm/4.62.3 CPython/3.8.7

Security for avro-to-python 0.3.3

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

No provenance found

None of the files in this release were uploaded with provenance metadata.

A note on package security

PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.

To learn more about protecting yourself from malicious actors, visit our package security documentation.

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page