DEPRECATED: Please see py-to-proto: https://pypi.org/project/py-to-proto/
Project description
DEPRECATED
This project has been renamed to py-to-proto
to reflect its expansion to include other input schema formats. Please see https://pypi.org/project/py-to-proto/
PY To Proto
This library holds utilities for converting in-memory data schema representations to Protobuf. The intent is to allow python libraries to leverage the power of protobuf
while maintaining the source-of-truth for their data in pure python and avoiding static build steps.
Why?
The protobuf
langauge is a powerful tool for defining language-agnostic, composable datastructures. Protobuf
also offers cross-language compatibility so that a given set of definitions can be compiled into numerous target programming languages. The downside is that protobuf
requires_a static built step to perform this proto
-> X
conversion step. Alternately, there are multiple ways of representing data schemas in pure python which allow a python library to interact with well-typed data objects. The downside here is that these structures can not easily be used from other programming languages. The pros/cons of these generally fall along the following lines:
Protobuf
:- Advantages
- Compact serialization
- Auto-generated
grpc
client and service libraries - Client libraries can be used from different programming languages
- Disadvantages
- Learning curve to understand the full ecosystem
- Not a familiar tool outside of service engineering
- Static compilation step required to use in code
- Advantages
- Python schemas:
- Advantages
- Can be learned quickly using pure-python documentation
- Can be written inline in pure python
- Disadvantages
- Generally, no standard serialization beyond
json
- No automated service implementations
- No/manual mechanism for usage in other programming languages
- Generally, no standard serialization beyond
- Advantages
This project aims to bring the advantages of both types of schema representation so that a given project can take advantage of the best of both:
- Define your structures in pure python for simplicity
- Dynamically create
google.protobuf.Descriptor
objects to allow forprotobuf
serialization and deserialization - Reverse render a
.proto
file from the generatedDescriptor
so that stubs can be generated in other languages - No static compiliation needed!
Supported Python Schema Types
Currently, objects can be declared using either python dataclasses
or Json TypeDef (JTD). Additional schemas can be added by subclassing ConverterBase
.
Dataclass To Proto
The following example illustrates how dataclasses
and enums
can be converted to proto:
from dataclasses import dataclass
from enum import Enum
from typing import Annotated, Dict, List, Enum
import py_to_proto
# Define the Foo structure as a python dataclass, including a nested enum
@dataclass
class Foo:
class BarEnum(Enum):
EXAM: 0
JOKE_SETTING: 1
foo: bool
bar: List[BarEnum]
# Define the Foo protobuf message class
FooProto = py_to_proto.descriptor_to_message_class(
py_to_proto.dataclass_to_proto(
package="foobar",
dataclass_=Foo,
)
)
# Declare the Bar structure as a python dataclass with a reference to the
# FooProto type
@dataclass
class Bar:
baz: FooProto
# Define the Bar protobuf message class
BarProto = py_to_proto.descriptor_to_message_class(
py_to_proto.dataclass_to_proto(
package="foobar",
dataclass_=Bar,
)
)
# Instantiate a BarProto
print(BarProto(baz=FooProto(foo=True, bar=[Foo.BarEnum.EXAM.value])))
def write_protos(proto_dir: str):
"""Write out the .proto files for FooProto and BarProto to the given
directory
"""
FooProto.write_proto_file(proto_dir)
BarProto.write_proto_file(proto_dir)
JTD To Proto
The following example illustrates how JTD schemas can be converted to proto:
import py_to_proto
# Declare the Foo protobuf message class
Foo = py_to_proto.descriptor_to_message_class(
py_to_proto.jtd_to_proto(
name="Foo",
package="foobar",
jtd_def={
"properties": {
# Bool field
"foo": {
"type": "boolean",
},
# Array of nested enum values
"bar": {
"elements": {
"enum": ["EXAM", "JOKE_SETTING"],
}
}
}
},
)
)
# Declare an object that references Foo as the type for a field
Bar = py_to_proto.descriptor_to_message_class(
py_to_proto.jtd_to_proto(
name="Bar",
package="foobar",
jtd_def={
"properties": {
"baz": {
"type": Foo.DESCRIPTOR,
},
},
},
),
)
def write_protos(proto_dir: str):
"""Write out the .proto files for Foo and Bar to the given directory"""
Foo.write_proto_file(proto_dir)
Bar.write_proto_file(proto_dir)
Similar Projects
There are a number of similar projects in this space that offer slightly different value:
jtd-codegen
: This project focuses on statically generating language-native code (includingpython
) to represent the JTD schema.py-json-to-proto
: This project aims to deduce a schema from an instance of ajson
object.pure-protobuf
: This project has a very similar aim topy-to-proto
, but it skips the intermediatedescriptor
representation and thus is not able to produce nativemessage.Message
classes.
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distributions
Built Distribution
Hashes for jtd_to_proto-0.13.0-py39-none-any.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 42461ff004ee869e4261a3fc42a77b8510b73fd456b0eb069eacd36e057ae3d7 |
|
MD5 | 642d0b69f87b8bbeec51eaf9417fc781 |
|
BLAKE2b-256 | ad2cb59eb40d43bf902084c9fcf1434b5951e5d86599438fd57e028ba05ccd92 |