A tool to dynamically create protobuf message classes from python data schemas
Project description
PY To Proto
This library holds utilities for converting in-memory data schema representations to Protobuf. The intent is to allow python libraries to leverage the power of protobuf
while maintaining the source-of-truth for their data in pure python and avoiding static build steps.
Why?
The protobuf
langauge is a powerful tool for defining language-agnostic, composable datastructures. Protobuf
also offers cross-language compatibility so that a given set of definitions can be compiled into numerous target programming languages. The downside is that protobuf
requires_a static built step to perform this proto
-> X
conversion step. Alternately, there are multiple ways of representing data schemas in pure python which allow a python library to interact with well-typed data objects. The downside here is that these structures can not easily be used from other programming languages. The pros/cons of these generally fall along the following lines:
Protobuf
:- Advantages
- Compact serialization
- Auto-generated
grpc
client and service libraries - Client libraries can be used from different programming languages
- Disadvantages
- Learning curve to understand the full ecosystem
- Not a familiar tool outside of service engineering
- Static compilation step required to use in code
- Advantages
- Python schemas:
- Advantages
- Can be learned quickly using pure-python documentation
- Can be written inline in pure python
- Disadvantages
- Generally, no standard serialization beyond
json
- No automated service implementations
- No/manual mechanism for usage in other programming languages
- Generally, no standard serialization beyond
- Advantages
This project aims to bring the advantages of both types of schema representation so that a given project can take advantage of the best of both:
- Define your structures in pure python for simplicity
- Dynamically create
google.protobuf.Descriptor
objects to allow forprotobuf
serialization and deserialization - Reverse render a
.proto
file from the generatedDescriptor
so that stubs can be generated in other languages - No static compiliation needed!
Supported Python Schema Types
Currently, objects can be declared using either python dataclasses
or Json TypeDef (JTD). Additional schemas can be added by subclassing ConverterBase
.
Dataclass To Proto
The following example illustrates how dataclasses
and enums
can be converted to proto:
from dataclasses import dataclass
from enum import Enum
from typing import Annotated, Dict, List, Enum
import py_to_proto
# Define the Foo structure as a python dataclass, including a nested enum
@dataclass
class Foo:
class BarEnum(Enum):
EXAM: 0
JOKE_SETTING: 1
foo: bool
bar: List[BarEnum]
# Define the Foo protobuf message class
FooProto = py_to_proto.descriptor_to_message_class(
py_to_proto.dataclass_to_proto(
package="foobar",
dataclass_=Foo,
)
)
# Declare the Bar structure as a python dataclass with a reference to the
# FooProto type
@dataclass
class Bar:
baz: FooProto
# Define the Bar protobuf message class
BarProto = py_to_proto.descriptor_to_message_class(
py_to_proto.dataclass_to_proto(
package="foobar",
dataclass_=Bar,
)
)
# Instantiate a BarProto
print(BarProto(baz=FooProto(foo=True, bar=[Foo.BarEnum.EXAM.value])))
def write_protos(proto_dir: str):
"""Write out the .proto files for FooProto and BarProto to the given
directory
"""
FooProto.write_proto_file(proto_dir)
BarProto.write_proto_file(proto_dir)
JTD To Proto
The following example illustrates how JTD schemas can be converted to proto:
import py_to_proto
# Declare the Foo protobuf message class
Foo = py_to_proto.descriptor_to_message_class(
py_to_proto.jtd_to_proto(
name="Foo",
package="foobar",
jtd_def={
"properties": {
# Bool field
"foo": {
"type": "boolean",
},
# Array of nested enum values
"bar": {
"elements": {
"enum": ["EXAM", "JOKE_SETTING"],
}
}
}
},
)
)
# Declare an object that references Foo as the type for a field
Bar = py_to_proto.descriptor_to_message_class(
py_to_proto.jtd_to_proto(
name="Bar",
package="foobar",
jtd_def={
"properties": {
"baz": {
"type": Foo.DESCRIPTOR,
},
},
},
),
)
def write_protos(proto_dir: str):
"""Write out the .proto files for Foo and Bar to the given directory"""
Foo.write_proto_file(proto_dir)
Bar.write_proto_file(proto_dir)
Similar Projects
There are a number of similar projects in this space that offer slightly different value:
jtd-codegen
: This project focuses on statically generating language-native code (includingpython
) to represent the JTD schema.py-json-to-proto
: This project aims to deduce a schema from an instance of ajson
object.pure-protobuf
: This project has a very similar aim topy-to-proto
, but it skips the intermediatedescriptor
representation and thus is not able to produce nativemessage.Message
classes.
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distributions
Built Distributions
Hashes for py_to_proto-0.1.1-py311-none-any.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 52627bd8965374de1f369fcb4e5efecd559f8d549a98199faca0a472f17d483f |
|
MD5 | 1f839e7faae25978fe9254c8e558e1cd |
|
BLAKE2b-256 | 0f28e2de3e8ace33d6467156503dbc791ce14f577a656d4b0d58561c501d74e1 |
Hashes for py_to_proto-0.1.1-py310-none-any.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 3de689176681e7c6eefcf518bb6817abfed635dc7a2ab14c073569ae00896006 |
|
MD5 | 57b12292f00b92d296db13c957401332 |
|
BLAKE2b-256 | ce5969511bee2295a28e14e50055361a69446638681075259456eaa1b72f1e6f |
Hashes for py_to_proto-0.1.1-py39-none-any.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | ace19fe5ec64b69cea1fae9abe04a1dc0d1bb77440baa78b99e2224ba4d917c9 |
|
MD5 | 09aeb0e8fbc3a4e59a296dfbf5f54d54 |
|
BLAKE2b-256 | c12416db5f60735cb5e1bd04cd03f32369e19d9cd52ef235448193c22fa9aa95 |
Hashes for py_to_proto-0.1.1-py38-none-any.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 1aec63a197cb4971686d857f65cc6c19d7baf8ec8ed738e7edab0fb679492c61 |
|
MD5 | bdaaa6e9f99d02d6698d703aaaa9d268 |
|
BLAKE2b-256 | 9a0f7c9e0b9f464d6be36a5efb789248f0674720c34f050dee1eff60945183b9 |
Hashes for py_to_proto-0.1.1-py37-none-any.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 0eb49d5654bba4f62860131d5de3be5b730a6d3bab2ddf56012bb1dcc5f47aeb |
|
MD5 | 119b419fd8e413857a39b2a2581a15ea |
|
BLAKE2b-256 | 0d42a9a30c7c3f7ce82800d6e68b4e0329d0674e5f87feed1583def494cd838c |