Generate Avro Schemas from a Python class
Project description
Dataclasses Avro Schema Generator
Generate Avro Schemas from a Python class
Requirements
python 3.7+
Installation
pip install dataclasses-avroschema
Documentation
https://marcosschroh.github.io/dataclasses-avroschema/
Usage
Generating the avro schema
import typing
from dataclasses_avroschema import AvroModel, types
class User(AvroModel):
"An User"
name: str
age: int
pets: typing.List[str]
accounts: typing.Dict[str, int]
favorite_colors: types.Enum = types.Enum(["BLUE", "YELLOW", "GREEN"])
country: str = "Argentina"
address: str = None
class Meta:
namespace = "User.v1"
aliases = ["user-v1", "super user"]
User.avro_schema()
'{
"type": "record",
"name": "User",
"doc": "An User",
"namespace": "User.v1",
"aliases": ["user-v1", "super user"],
"fields": [
{"name": "name", "type": "string"},
{"name": "age", "type": "int"},
{"name": "pets", "type": "array", "items": "string"},
{"name": "accounts", "type": "map", "values": "int"},
{"name": "favorite_colors", "type": "enum", "symbols": ["BLUE", "YELLOW", "GREEN"]},
{"name": "country", "type": ["string", "null"], "default": "Argentina"},
{"name": "address", "type": ["null", "string"], "default": "null"}
]
}'
Serialization to avro or avro-json and json payload
For serialization is neccesary to use python class/dataclasses instance
from dataclasses import dataclass
import typing
from dataclasses_avroschema import AvroModel
@dataclass
class Address(AvroModel):
"An Address"
street: str
street_number: int
@dataclass
class User(AvroModel):
"User with multiple Address"
name: str
age: int
addresses: typing.List[Address]
address_data = {
"street": "test",
"street_number": 10,
}
# create an Address instance
address = Address(**address_data)
data_user = {
"name": "john",
"age": 20,
"addresses": [address],
}
# create an User instance
user = User(**data_user)
user.serialize()
# >>> b"\x08john(\x02\x08test\x14\x00"
user.serialize(serialization_type="avro-json")
# >>> b'{"name": "john", "age": 20, "addresses": [{"street": "test", "street_number": 10}]}'
# Get the json from the instance
user.to_json()
# python dict >>> {'name': 'john', 'age': 20, 'addresses': [{'street': 'test', 'street_number': 10}]}
Deserialization
Deserialization could take place with an instance dataclass or the dataclass itself
import typing
from dataclasses_avroschema import AvroModel
class Address(AvroModel):
"An Address"
street: str
street_number: int
class User(AvroModel):
"User with multiple Address"
name: str
age: int
addresses: typing.List[Address]
avro_binary = b"\x08john(\x02\x08test\x14\x00"
avro_json_binary = b'{"name": "john", "age": 20, "addresses": [{"street": "test", "street_number": 10}]}'
User.deserialize(avro_binary)
# >>> {"name": "john", "age": 20, "addresses": [{"street": "test", "street_number": 10}]}
User.deserialize(avro_json_binary, serialization_type="avro-json")
# >>> {"name": "john", "age": 20, "addresses": [{"street": "test", "street_number": 10}]}
Features
- Primitive types: int, long, float, boolean, string and null support
- Complex types: enum, array, map, fixed, unions and records support
- Logical Types: date, time, datetime, uuid support
- Schema relations (oneToOne, oneToMany)
- Recursive Schemas
- Generate Avro Schemas from
faust.Record
- Instance serialization correspondent to
avro schema
generated - Data deserialization
- Generate json from python class instance
Development
- Create a
virtualenv
:python3.7 -m venv venv && source venv/bin/activate
- Install requirements:
pip install -r requirements.txt
- Code linting:
./scripts/lint
- Run tests:
./scripts/test
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Close
Hashes for dataclasses-avroschema-0.14.4.tar.gz
Algorithm | Hash digest | |
---|---|---|
SHA256 | 22f406a04da50af401962c2327518aed78af935db8aed78056a7abd1e679fccb |
|
MD5 | 20eadd1b3a4ae6020644ff45e9f03659 |
|
BLAKE2b-256 | 360738177e26c26049ecec3725dfa878be8d80364f75962418336723a05b44bf |