A tool to dynamically create protobuf message classes from python data schemas
Project description
PY To Proto
This library holds utilities for converting in-memory data schema representations to Protobuf. The intent is to allow python libraries to leverage the power of protobuf
while maintaining the source-of-truth for their data in pure python and avoiding static build steps.
Why?
The protobuf
langauge is a powerful tool for defining language-agnostic, composable datastructures. Protobuf
also offers cross-language compatibility so that a given set of definitions can be compiled into numerous target programming languages. The downside is that protobuf
requires_a static built step to perform this proto
-> X
conversion step. Alternately, there are multiple ways of representing data schemas in pure python which allow a python library to interact with well-typed data objects. The downside here is that these structures can not easily be used from other programming languages. The pros/cons of these generally fall along the following lines:
Protobuf
:- Advantages
- Compact serialization
- Auto-generated
grpc
client and service libraries - Client libraries can be used from different programming languages
- Disadvantages
- Learning curve to understand the full ecosystem
- Not a familiar tool outside of service engineering
- Static compilation step required to use in code
- Advantages
- Python schemas:
- Advantages
- Can be learned quickly using pure-python documentation
- Can be written inline in pure python
- Disadvantages
- Generally, no standard serialization beyond
json
- No automated service implementations
- No/manual mechanism for usage in other programming languages
- Generally, no standard serialization beyond
- Advantages
This project aims to bring the advantages of both types of schema representation so that a given project can take advantage of the best of both:
- Define your structures in pure python for simplicity
- Dynamically create
google.protobuf.Descriptor
objects to allow forprotobuf
serialization and deserialization - Reverse render a
.proto
file from the generatedDescriptor
so that stubs can be generated in other languages - No static compiliation needed!
Supported Python Schema Types
Currently, objects can be declared using either python dataclasses
or Json TypeDef (JTD). Additional schemas can be added by subclassing ConverterBase
.
Dataclass To Proto
The following example illustrates how dataclasses
and enums
can be converted to proto:
from dataclasses import dataclass
from enum import Enum
from typing import Annotated, Dict, List, Enum
import py_to_proto
# Define the Foo structure as a python dataclass, including a nested enum
@dataclass
class Foo:
class BarEnum(Enum):
EXAM: 0
JOKE_SETTING: 1
foo: bool
bar: List[BarEnum]
# Define the Foo protobuf message class
FooProto = py_to_proto.descriptor_to_message_class(
py_to_proto.dataclass_to_proto(
package="foobar",
dataclass_=Foo,
)
)
# Declare the Bar structure as a python dataclass with a reference to the
# FooProto type
@dataclass
class Bar:
baz: FooProto
# Define the Bar protobuf message class
BarProto = py_to_proto.descriptor_to_message_class(
py_to_proto.dataclass_to_proto(
package="foobar",
dataclass_=Bar,
)
)
# Instantiate a BarProto
print(BarProto(baz=FooProto(foo=True, bar=[Foo.BarEnum.EXAM.value])))
def write_protos(proto_dir: str):
"""Write out the .proto files for FooProto and BarProto to the given
directory
"""
FooProto.write_proto_file(proto_dir)
BarProto.write_proto_file(proto_dir)
JTD To Proto
The following example illustrates how JTD schemas can be converted to proto:
import py_to_proto
# Declare the Foo protobuf message class
Foo = py_to_proto.descriptor_to_message_class(
py_to_proto.jtd_to_proto(
name="Foo",
package="foobar",
jtd_def={
"properties": {
# Bool field
"foo": {
"type": "boolean",
},
# Array of nested enum values
"bar": {
"elements": {
"enum": ["EXAM", "JOKE_SETTING"],
}
}
}
},
)
)
# Declare an object that references Foo as the type for a field
Bar = py_to_proto.descriptor_to_message_class(
py_to_proto.jtd_to_proto(
name="Bar",
package="foobar",
jtd_def={
"properties": {
"baz": {
"type": Foo.DESCRIPTOR,
},
},
},
),
)
def write_protos(proto_dir: str):
"""Write out the .proto files for Foo and Bar to the given directory"""
Foo.write_proto_file(proto_dir)
Bar.write_proto_file(proto_dir)
Similar Projects
There are a number of similar projects in this space that offer slightly different value:
jtd-codegen
: This project focuses on statically generating language-native code (includingpython
) to represent the JTD schema.py-json-to-proto
: This project aims to deduce a schema from an instance of ajson
object.pure-protobuf
: This project has a very similar aim topy-to-proto
, but it skips the intermediatedescriptor
representation and thus is not able to produce nativemessage.Message
classes.
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distributions
Built Distributions
Hashes for py_to_proto-0.4.1-py311-none-any.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | f49564e1ac7fbe4d53ebe33dc4a35ca77efd19338ad3313c1fae41d27fdf9276 |
|
MD5 | 769f6679a33d70122cf48420665c5486 |
|
BLAKE2b-256 | e69182beae21ab02cc7e552469bcef93fb30b1368d68782590f70e62920cc76d |
Hashes for py_to_proto-0.4.1-py310-none-any.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 179a3433421b5e7c2a912d8356e1aa75f30ab4ed7dea25604f36e5558c8a52f9 |
|
MD5 | 7140f2d2fad0a7239e5384298e090e05 |
|
BLAKE2b-256 | 7aadf2ea33523dcde384e8e740a57767f9fb7fef9c87bc143f62dfadababa380 |
Hashes for py_to_proto-0.4.1-py39-none-any.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | b4db6e40161ea84b1c5490959dfb3bc8318a1b7842927606e8c38bbff44daae4 |
|
MD5 | e677902527dc449344c62b9a914d77b4 |
|
BLAKE2b-256 | ee1725a826c8817b63260bc973817cfda25565f10d8db01f9516eb0eed322ff3 |
Hashes for py_to_proto-0.4.1-py38-none-any.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 5dc91da3963db11dafc1f3fd2f3424a909a08e10cd0f0cf08f22d389c33a639c |
|
MD5 | c5250f9fd922bfa9a06108656d46c1de |
|
BLAKE2b-256 | b3ecf5b24d04f00001e06f351c1fd4a9558ed1993f9797a95840fb9119c91c2e |
Hashes for py_to_proto-0.4.1-py37-none-any.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | edad8b95e75376c1f148f438088f75f9097d89466c8079005f1eff473865b04f |
|
MD5 | 2148d4ed5875e2e57ef38b610950006e |
|
BLAKE2b-256 | e159f09763b7a7bf86d7e3f45f4fb52a0274053a4fb4100997506b42b750e5f9 |