A tool to dynamically create protobuf message classes from python data schemas
Project description
PY To Proto
This library holds utilities for converting in-memory data schema representations to Protobuf. The intent is to allow python libraries to leverage the power of protobuf
while maintaining the source-of-truth for their data in pure python and avoiding static build steps.
Why?
The protobuf
langauge is a powerful tool for defining language-agnostic, composable datastructures. Protobuf
also offers cross-language compatibility so that a given set of definitions can be compiled into numerous target programming languages. The downside is that protobuf
requires_a static built step to perform this proto
-> X
conversion step. Alternately, there are multiple ways of representing data schemas in pure python which allow a python library to interact with well-typed data objects. The downside here is that these structures can not easily be used from other programming languages. The pros/cons of these generally fall along the following lines:
Protobuf
:- Advantages
- Compact serialization
- Auto-generated
grpc
client and service libraries - Client libraries can be used from different programming languages
- Disadvantages
- Learning curve to understand the full ecosystem
- Not a familiar tool outside of service engineering
- Static compilation step required to use in code
- Advantages
- Python schemas:
- Advantages
- Can be learned quickly using pure-python documentation
- Can be written inline in pure python
- Disadvantages
- Generally, no standard serialization beyond
json
- No automated service implementations
- No/manual mechanism for usage in other programming languages
- Generally, no standard serialization beyond
- Advantages
This project aims to bring the advantages of both types of schema representation so that a given project can take advantage of the best of both:
- Define your structures in pure python for simplicity
- Dynamically create
google.protobuf.Descriptor
objects to allow forprotobuf
serialization and deserialization - Reverse render a
.proto
file from the generatedDescriptor
so that stubs can be generated in other languages - No static compiliation needed!
Supported Python Schema Types
Currently, objects can be declared using either python dataclasses
or Json TypeDef (JTD). Additional schemas can be added by subclassing ConverterBase
.
Dataclass To Proto
The following example illustrates how dataclasses
and enums
can be converted to proto:
from dataclasses import dataclass
from enum import Enum
from typing import Annotated, Dict, List, Enum
import py_to_proto
# Define the Foo structure as a python dataclass, including a nested enum
@dataclass
class Foo:
class BarEnum(Enum):
EXAM: 0
JOKE_SETTING: 1
foo: bool
bar: List[BarEnum]
# Define the Foo protobuf message class
FooProto = py_to_proto.descriptor_to_message_class(
py_to_proto.dataclass_to_proto(
package="foobar",
dataclass_=Foo,
)
)
# Declare the Bar structure as a python dataclass with a reference to the
# FooProto type
@dataclass
class Bar:
baz: FooProto
# Define the Bar protobuf message class
BarProto = py_to_proto.descriptor_to_message_class(
py_to_proto.dataclass_to_proto(
package="foobar",
dataclass_=Bar,
)
)
# Instantiate a BarProto
print(BarProto(baz=FooProto(foo=True, bar=[Foo.BarEnum.EXAM.value])))
def write_protos(proto_dir: str):
"""Write out the .proto files for FooProto and BarProto to the given
directory
"""
FooProto.write_proto_file(proto_dir)
BarProto.write_proto_file(proto_dir)
JTD To Proto
The following example illustrates how JTD schemas can be converted to proto:
import py_to_proto
# Declare the Foo protobuf message class
Foo = py_to_proto.descriptor_to_message_class(
py_to_proto.jtd_to_proto(
name="Foo",
package="foobar",
jtd_def={
"properties": {
# Bool field
"foo": {
"type": "boolean",
},
# Array of nested enum values
"bar": {
"elements": {
"enum": ["EXAM", "JOKE_SETTING"],
}
}
}
},
)
)
# Declare an object that references Foo as the type for a field
Bar = py_to_proto.descriptor_to_message_class(
py_to_proto.jtd_to_proto(
name="Bar",
package="foobar",
jtd_def={
"properties": {
"baz": {
"type": Foo.DESCRIPTOR,
},
},
},
),
)
def write_protos(proto_dir: str):
"""Write out the .proto files for Foo and Bar to the given directory"""
Foo.write_proto_file(proto_dir)
Bar.write_proto_file(proto_dir)
Similar Projects
There are a number of similar projects in this space that offer slightly different value:
jtd-codegen
: This project focuses on statically generating language-native code (includingpython
) to represent the JTD schema.py-json-to-proto
: This project aims to deduce a schema from an instance of ajson
object.pure-protobuf
: This project has a very similar aim topy-to-proto
, but it skips the intermediatedescriptor
representation and thus is not able to produce nativemessage.Message
classes.
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distributions
Built Distributions
Hashes for py_to_proto-0.5.2-py311-none-any.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 38c08b63837596b285174288255d28757e1ade55c444391485eb55845f295ba9 |
|
MD5 | b8c271d8e42d2ce954acbcf859495936 |
|
BLAKE2b-256 | 7edf1ec98e3a764efe4005bd99da372de179d130c927467ae25cd8b86f9b8aaa |
Hashes for py_to_proto-0.5.2-py310-none-any.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 9b5d633bb47a96ea8fd61b5ae4993deb62a104c0027831e1b5a226aed6b7fce7 |
|
MD5 | 5da8f1217a1861d9eb1251881639db11 |
|
BLAKE2b-256 | eb298b2b5f79f3842406f06b68c4ab74ac43bd9c3ee04c97d93c196544679d7a |
Hashes for py_to_proto-0.5.2-py39-none-any.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | a1a3553b6b0ce87fed08567615c444f1eeafacb3a124920a2ea02b91f62ac0e1 |
|
MD5 | d6f3326bc021391abc0c8c29a1b5abae |
|
BLAKE2b-256 | 09d3c95198744626cad2e209784cf3acf763e9299b5dd378b7393cfe49966f87 |
Hashes for py_to_proto-0.5.2-py38-none-any.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 91f48866c132fbd5901c065e73161195c108b3ce73edf549083c79fed5d44fb8 |
|
MD5 | 645a49a61a6033efedf3f532eab5408e |
|
BLAKE2b-256 | 711f47f90ad339e1f3d0f72b58dd39e57786c1422dd32d37363a3bd5ebec91d1 |
Hashes for py_to_proto-0.5.2-py37-none-any.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 22021b46f216afa899751102d78ac090f23a757ea97635c77aabeff0106d6dad |
|
MD5 | bd93d00f678382ca5a725b4ea133f65b |
|
BLAKE2b-256 | 760555f99fc5c4ba9a6cb048377955dbf1bc5b1ca8081e869ae00d48cb624365 |