A tool to dynamically create protobuf message classes from python data schemas
Project description
PY To Proto
This library holds utilities for converting in-memory data schema representations to Protobuf. The intent is to allow python libraries to leverage the power of protobuf
while maintaining the source-of-truth for their data in pure python and avoiding static build steps.
Why?
The protobuf
langauge is a powerful tool for defining language-agnostic, composable datastructures. Protobuf
also offers cross-language compatibility so that a given set of definitions can be compiled into numerous target programming languages. The downside is that protobuf
requires_a static built step to perform this proto
-> X
conversion step. Alternately, there are multiple ways of representing data schemas in pure python which allow a python library to interact with well-typed data objects. The downside here is that these structures can not easily be used from other programming languages. The pros/cons of these generally fall along the following lines:
Protobuf
:- Advantages
- Compact serialization
- Auto-generated
grpc
client and service libraries - Client libraries can be used from different programming languages
- Disadvantages
- Learning curve to understand the full ecosystem
- Not a familiar tool outside of service engineering
- Static compilation step required to use in code
- Advantages
- Python schemas:
- Advantages
- Can be learned quickly using pure-python documentation
- Can be written inline in pure python
- Disadvantages
- Generally, no standard serialization beyond
json
- No automated service implementations
- No/manual mechanism for usage in other programming languages
- Generally, no standard serialization beyond
- Advantages
This project aims to bring the advantages of both types of schema representation so that a given project can take advantage of the best of both:
- Define your structures in pure python for simplicity
- Dynamically create
google.protobuf.Descriptor
objects to allow forprotobuf
serialization and deserialization - Reverse render a
.proto
file from the generatedDescriptor
so that stubs can be generated in other languages - No static compiliation needed!
Supported Python Schema Types
Currently, objects can be declared using either python dataclasses
or Json TypeDef (JTD). Additional schemas can be added by subclassing ConverterBase
.
Dataclass To Proto
The following example illustrates how dataclasses
and enums
can be converted to proto:
from dataclasses import dataclass
from enum import Enum
from typing import Annotated, Dict, List, Enum
import py_to_proto
# Define the Foo structure as a python dataclass, including a nested enum
@dataclass
class Foo:
class BarEnum(Enum):
EXAM: 0
JOKE_SETTING: 1
foo: bool
bar: List[BarEnum]
# Define the Foo protobuf message class
FooProto = py_to_proto.descriptor_to_message_class(
py_to_proto.dataclass_to_proto(
package="foobar",
dataclass_=Foo,
)
)
# Declare the Bar structure as a python dataclass with a reference to the
# FooProto type
@dataclass
class Bar:
baz: FooProto
# Define the Bar protobuf message class
BarProto = py_to_proto.descriptor_to_message_class(
py_to_proto.dataclass_to_proto(
package="foobar",
dataclass_=Bar,
)
)
# Instantiate a BarProto
print(BarProto(baz=FooProto(foo=True, bar=[Foo.BarEnum.EXAM.value])))
def write_protos(proto_dir: str):
"""Write out the .proto files for FooProto and BarProto to the given
directory
"""
FooProto.write_proto_file(proto_dir)
BarProto.write_proto_file(proto_dir)
JTD To Proto
The following example illustrates how JTD schemas can be converted to proto:
import py_to_proto
# Declare the Foo protobuf message class
Foo = py_to_proto.descriptor_to_message_class(
py_to_proto.jtd_to_proto(
name="Foo",
package="foobar",
jtd_def={
"properties": {
# Bool field
"foo": {
"type": "boolean",
},
# Array of nested enum values
"bar": {
"elements": {
"enum": ["EXAM", "JOKE_SETTING"],
}
}
}
},
)
)
# Declare an object that references Foo as the type for a field
Bar = py_to_proto.descriptor_to_message_class(
py_to_proto.jtd_to_proto(
name="Bar",
package="foobar",
jtd_def={
"properties": {
"baz": {
"type": Foo.DESCRIPTOR,
},
},
},
),
)
def write_protos(proto_dir: str):
"""Write out the .proto files for Foo and Bar to the given directory"""
Foo.write_proto_file(proto_dir)
Bar.write_proto_file(proto_dir)
Similar Projects
There are a number of similar projects in this space that offer slightly different value:
jtd-codegen
: This project focuses on statically generating language-native code (includingpython
) to represent the JTD schema.py-json-to-proto
: This project aims to deduce a schema from an instance of ajson
object.pure-protobuf
: This project has a very similar aim topy-to-proto
, but it skips the intermediatedescriptor
representation and thus is not able to produce nativemessage.Message
classes.
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distributions
Built Distributions
Hashes for py_to_proto-0.2.0-py311-none-any.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 7acf5cde49f82873834ad83a19abeb6889d4a8074cbce493d98bca3ee0e046d7 |
|
MD5 | 744c805e031e03772c71995dfd65e7f9 |
|
BLAKE2b-256 | 9495f072f90bbcdb0c617ff5356147903aca8944d742b6e928076b977e16161b |
Hashes for py_to_proto-0.2.0-py310-none-any.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | f7eaf34da107fbf1646c077e9eb7c1104877c63b312905fc2b16d52017d531e5 |
|
MD5 | a7a35cc10e10b13cf58cedf510c28991 |
|
BLAKE2b-256 | 8a05d887f1ce044a3253157bc5c20c0d9eda9ef1a42a915127f867f83bb351f7 |
Hashes for py_to_proto-0.2.0-py39-none-any.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 38873bd279b152e8119686e8d618b68534395389c2b6654f8fb10905002d8668 |
|
MD5 | fdd572e7f78ca47d4f8e8691140a368f |
|
BLAKE2b-256 | ab4794f9c823b1b5b0cacffc33c7796aae69901cb40e383266b72f8a45e87a6c |
Hashes for py_to_proto-0.2.0-py38-none-any.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | cfc438fdb269521cca8e988f2ef595bcc190893844b0052dfc63a9d569f9cad8 |
|
MD5 | a7c64eac2008cb15c2434b462551742b |
|
BLAKE2b-256 | fbab44aca155d48e03264903c1a420e92c6ab80cbbd7e97edd409f2c3664491b |
Hashes for py_to_proto-0.2.0-py37-none-any.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 88e2deca7e7647db7e5d0f654ca40cd8cbd10ec82706afa1e82c958036e2e81a |
|
MD5 | 1c6432a3fddb04fc4c5c32ca57e42702 |
|
BLAKE2b-256 | 3aabbfc7b5f875a34610977c0c40462da796a9e6535159513e3e7d50573b9c63 |