A tool to dynamically create protobuf message classes from python data schemas
Project description
PY To Proto
This library holds utilities for converting in-memory data schema representations to Protobuf. The intent is to allow python libraries to leverage the power of protobuf
while maintaining the source-of-truth for their data in pure python and avoiding static build steps.
Why?
The protobuf
langauge is a powerful tool for defining language-agnostic, composable datastructures. Protobuf
also offers cross-language compatibility so that a given set of definitions can be compiled into numerous target programming languages. The downside is that protobuf
requires_a static built step to perform this proto
-> X
conversion step. Alternately, there are multiple ways of representing data schemas in pure python which allow a python library to interact with well-typed data objects. The downside here is that these structures can not easily be used from other programming languages. The pros/cons of these generally fall along the following lines:
Protobuf
:- Advantages
- Compact serialization
- Auto-generated
grpc
client and service libraries - Client libraries can be used from different programming languages
- Disadvantages
- Learning curve to understand the full ecosystem
- Not a familiar tool outside of service engineering
- Static compilation step required to use in code
- Advantages
- Python schemas:
- Advantages
- Can be learned quickly using pure-python documentation
- Can be written inline in pure python
- Disadvantages
- Generally, no standard serialization beyond
json
- No automated service implementations
- No/manual mechanism for usage in other programming languages
- Generally, no standard serialization beyond
- Advantages
This project aims to bring the advantages of both types of schema representation so that a given project can take advantage of the best of both:
- Define your structures in pure python for simplicity
- Dynamically create
google.protobuf.Descriptor
objects to allow forprotobuf
serialization and deserialization - Reverse render a
.proto
file from the generatedDescriptor
so that stubs can be generated in other languages - No static compiliation needed!
Supported Python Schema Types
Currently, objects can be declared using either python dataclasses
or Json TypeDef (JTD). Additional schemas can be added by subclassing ConverterBase
.
Dataclass To Proto
The following example illustrates how dataclasses
and enums
can be converted to proto:
from dataclasses import dataclass
from enum import Enum
from typing import Annotated, Dict, List, Enum
import py_to_proto
# Define the Foo structure as a python dataclass, including a nested enum
@dataclass
class Foo:
class BarEnum(Enum):
EXAM: 0
JOKE_SETTING: 1
foo: bool
bar: List[BarEnum]
# Define the Foo protobuf message class
FooProto = py_to_proto.descriptor_to_message_class(
py_to_proto.dataclass_to_proto(
package="foobar",
dataclass_=Foo,
)
)
# Declare the Bar structure as a python dataclass with a reference to the
# FooProto type
@dataclass
class Bar:
baz: FooProto
# Define the Bar protobuf message class
BarProto = py_to_proto.descriptor_to_message_class(
py_to_proto.dataclass_to_proto(
package="foobar",
dataclass_=Bar,
)
)
# Instantiate a BarProto
print(BarProto(baz=FooProto(foo=True, bar=[Foo.BarEnum.EXAM.value])))
def write_protos(proto_dir: str):
"""Write out the .proto files for FooProto and BarProto to the given
directory
"""
FooProto.write_proto_file(proto_dir)
BarProto.write_proto_file(proto_dir)
JTD To Proto
The following example illustrates how JTD schemas can be converted to proto:
import py_to_proto
# Declare the Foo protobuf message class
Foo = py_to_proto.descriptor_to_message_class(
py_to_proto.jtd_to_proto(
name="Foo",
package="foobar",
jtd_def={
"properties": {
# Bool field
"foo": {
"type": "boolean",
},
# Array of nested enum values
"bar": {
"elements": {
"enum": ["EXAM", "JOKE_SETTING"],
}
}
}
},
)
)
# Declare an object that references Foo as the type for a field
Bar = py_to_proto.descriptor_to_message_class(
py_to_proto.jtd_to_proto(
name="Bar",
package="foobar",
jtd_def={
"properties": {
"baz": {
"type": Foo.DESCRIPTOR,
},
},
},
),
)
def write_protos(proto_dir: str):
"""Write out the .proto files for Foo and Bar to the given directory"""
Foo.write_proto_file(proto_dir)
Bar.write_proto_file(proto_dir)
Similar Projects
There are a number of similar projects in this space that offer slightly different value:
jtd-codegen
: This project focuses on statically generating language-native code (includingpython
) to represent the JTD schema.py-json-to-proto
: This project aims to deduce a schema from an instance of ajson
object.pure-protobuf
: This project has a very similar aim topy-to-proto
, but it skips the intermediatedescriptor
representation and thus is not able to produce nativemessage.Message
classes.
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distributions
Built Distributions
File details
Details for the file py_to_proto-0.5.2-py311-none-any.whl
.
File metadata
- Download URL: py_to_proto-0.5.2-py311-none-any.whl
- Upload date:
- Size: 32.9 kB
- Tags: Python 3.11
- Uploaded using Trusted Publishing? No
- Uploaded via: twine/4.0.2 CPython/3.11.6
File hashes
Algorithm | Hash digest | |
---|---|---|
SHA256 | 38c08b63837596b285174288255d28757e1ade55c444391485eb55845f295ba9 |
|
MD5 | b8c271d8e42d2ce954acbcf859495936 |
|
BLAKE2b-256 | 7edf1ec98e3a764efe4005bd99da372de179d130c927467ae25cd8b86f9b8aaa |
File details
Details for the file py_to_proto-0.5.2-py310-none-any.whl
.
File metadata
- Download URL: py_to_proto-0.5.2-py310-none-any.whl
- Upload date:
- Size: 32.9 kB
- Tags: Python 3.10
- Uploaded using Trusted Publishing? No
- Uploaded via: twine/4.0.2 CPython/3.10.13
File hashes
Algorithm | Hash digest | |
---|---|---|
SHA256 | 9b5d633bb47a96ea8fd61b5ae4993deb62a104c0027831e1b5a226aed6b7fce7 |
|
MD5 | 5da8f1217a1861d9eb1251881639db11 |
|
BLAKE2b-256 | eb298b2b5f79f3842406f06b68c4ab74ac43bd9c3ee04c97d93c196544679d7a |
File details
Details for the file py_to_proto-0.5.2-py39-none-any.whl
.
File metadata
- Download URL: py_to_proto-0.5.2-py39-none-any.whl
- Upload date:
- Size: 32.9 kB
- Tags: Python 3.9
- Uploaded using Trusted Publishing? No
- Uploaded via: twine/4.0.2 CPython/3.9.18
File hashes
Algorithm | Hash digest | |
---|---|---|
SHA256 | a1a3553b6b0ce87fed08567615c444f1eeafacb3a124920a2ea02b91f62ac0e1 |
|
MD5 | d6f3326bc021391abc0c8c29a1b5abae |
|
BLAKE2b-256 | 09d3c95198744626cad2e209784cf3acf763e9299b5dd378b7393cfe49966f87 |
File details
Details for the file py_to_proto-0.5.2-py38-none-any.whl
.
File metadata
- Download URL: py_to_proto-0.5.2-py38-none-any.whl
- Upload date:
- Size: 32.9 kB
- Tags: Python 3.8
- Uploaded using Trusted Publishing? No
- Uploaded via: twine/4.0.2 CPython/3.8.18
File hashes
Algorithm | Hash digest | |
---|---|---|
SHA256 | 91f48866c132fbd5901c065e73161195c108b3ce73edf549083c79fed5d44fb8 |
|
MD5 | 645a49a61a6033efedf3f532eab5408e |
|
BLAKE2b-256 | 711f47f90ad339e1f3d0f72b58dd39e57786c1422dd32d37363a3bd5ebec91d1 |
File details
Details for the file py_to_proto-0.5.2-py37-none-any.whl
.
File metadata
- Download URL: py_to_proto-0.5.2-py37-none-any.whl
- Upload date:
- Size: 32.9 kB
- Tags: Python 3.7
- Uploaded using Trusted Publishing? No
- Uploaded via: twine/4.0.2 CPython/3.7.17
File hashes
Algorithm | Hash digest | |
---|---|---|
SHA256 | 22021b46f216afa899751102d78ac090f23a757ea97635c77aabeff0106d6dad |
|
MD5 | bd93d00f678382ca5a725b4ea133f65b |
|
BLAKE2b-256 | 760555f99fc5c4ba9a6cb048377955dbf1bc5b1ca8081e869ae00d48cb624365 |