Skip to main content

A tool to dynamically create protobuf message classes from python data schemas

Project description

PY To Proto

This library holds utilities for converting in-memory data schema representations to Protobuf. The intent is to allow python libraries to leverage the power of protobuf while maintaining the source-of-truth for their data in pure python and avoiding static build steps.

Why?

The protobuf langauge is a powerful tool for defining language-agnostic, composable datastructures. Protobuf also offers cross-language compatibility so that a given set of definitions can be compiled into numerous target programming languages. The downside is that protobuf requires_a static built step to perform this proto -> X conversion step. Alternately, there are multiple ways of representing data schemas in pure python which allow a python library to interact with well-typed data objects. The downside here is that these structures can not easily be used from other programming languages. The pros/cons of these generally fall along the following lines:

  • Protobuf:
    • Advantages
      • Compact serialization
      • Auto-generated grpc client and service libraries
      • Client libraries can be used from different programming languages
    • Disadvantages
      • Learning curve to understand the full ecosystem
      • Not a familiar tool outside of service engineering
      • Static compilation step required to use in code
  • Python schemas:
    • Advantages
      • Can be learned quickly using pure-python documentation
      • Can be written inline in pure python
    • Disadvantages
      • Generally, no standard serialization beyond json
      • No automated service implementations
      • No/manual mechanism for usage in other programming languages

This project aims to bring the advantages of both types of schema representation so that a given project can take advantage of the best of both:

  • Define your structures in pure python for simplicity
  • Dynamically create google.protobuf.Descriptor objects to allow for protobuf serialization and deserialization
  • Reverse render a .proto file from the generated Descriptor so that stubs can be generated in other languages
  • No static compiliation needed!

Supported Python Schema Types

Currently, objects can be declared using either python dataclasses or Json TypeDef (JTD). Additional schemas can be added by subclassing ConverterBase.

Dataclass To Proto

The following example illustrates how dataclasses and enums can be converted to proto:

from dataclasses import dataclass
from enum import Enum
from typing import Annotated, Dict, List, Enum
import py_to_proto

# Define the Foo structure as a python dataclass, including a nested enum
@dataclass
class Foo:

    class BarEnum(Enum):
        EXAM: 0
        JOKE_SETTING: 1

    foo: bool
    bar: List[BarEnum]

# Define the Foo protobuf message class
FooProto = py_to_proto.descriptor_to_message_class(
    py_to_proto.dataclass_to_proto(
        package="foobar",
        dataclass_=Foo,
    )
)

# Declare the Bar structure as a python dataclass with a reference to the
# FooProto type
@dataclass
class Bar:
    baz: FooProto

# Define the Bar protobuf message class
BarProto = py_to_proto.descriptor_to_message_class(
    py_to_proto.dataclass_to_proto(
        package="foobar",
        dataclass_=Bar,
    )
)

# Instantiate a BarProto
print(BarProto(baz=FooProto(foo=True, bar=[Foo.BarEnum.EXAM.value])))

def write_protos(proto_dir: str):
    """Write out the .proto files for FooProto and BarProto to the given
    directory
    """
    FooProto.write_proto_file(proto_dir)
    BarProto.write_proto_file(proto_dir)

JTD To Proto

The following example illustrates how JTD schemas can be converted to proto:

import py_to_proto

# Declare the Foo protobuf message class
Foo = py_to_proto.descriptor_to_message_class(
    py_to_proto.jtd_to_proto(
        name="Foo",
        package="foobar",
        jtd_def={
            "properties": {
                # Bool field
                "foo": {
                    "type": "boolean",
                },
                # Array of nested enum values
                "bar": {
                    "elements": {
                        "enum": ["EXAM", "JOKE_SETTING"],
                    }
                }
            }
        },
    )
)

# Declare an object that references Foo as the type for a field
Bar = py_to_proto.descriptor_to_message_class(
    py_to_proto.jtd_to_proto(
        name="Bar",
        package="foobar",
        jtd_def={
            "properties": {
                "baz": {
                    "type": Foo.DESCRIPTOR,
                },
            },
        },
    ),
)

def write_protos(proto_dir: str):
    """Write out the .proto files for Foo and Bar to the given directory"""
    Foo.write_proto_file(proto_dir)
    Bar.write_proto_file(proto_dir)

Similar Projects

There are a number of similar projects in this space that offer slightly different value:

  • jtd-codegen: This project focuses on statically generating language-native code (including python) to represent the JTD schema.
  • py-json-to-proto: This project aims to deduce a schema from an instance of a json object.
  • pure-protobuf: This project has a very similar aim to py-to-proto, but it skips the intermediate descriptor representation and thus is not able to produce native message.Message classes.

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distributions

No source distribution files available for this release.See tutorial on generating distribution archives.

Built Distributions

py_to_proto-0.5.2-py311-none-any.whl (32.9 kB view details)

Uploaded Python 3.11

py_to_proto-0.5.2-py310-none-any.whl (32.9 kB view details)

Uploaded Python 3.10

py_to_proto-0.5.2-py39-none-any.whl (32.9 kB view details)

Uploaded Python 3.9

py_to_proto-0.5.2-py38-none-any.whl (32.9 kB view details)

Uploaded Python 3.8

py_to_proto-0.5.2-py37-none-any.whl (32.9 kB view details)

Uploaded Python 3.7

File details

Details for the file py_to_proto-0.5.2-py311-none-any.whl.

File metadata

File hashes

Hashes for py_to_proto-0.5.2-py311-none-any.whl
Algorithm Hash digest
SHA256 38c08b63837596b285174288255d28757e1ade55c444391485eb55845f295ba9
MD5 b8c271d8e42d2ce954acbcf859495936
BLAKE2b-256 7edf1ec98e3a764efe4005bd99da372de179d130c927467ae25cd8b86f9b8aaa

See more details on using hashes here.

File details

Details for the file py_to_proto-0.5.2-py310-none-any.whl.

File metadata

  • Download URL: py_to_proto-0.5.2-py310-none-any.whl
  • Upload date:
  • Size: 32.9 kB
  • Tags: Python 3.10
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/4.0.2 CPython/3.10.13

File hashes

Hashes for py_to_proto-0.5.2-py310-none-any.whl
Algorithm Hash digest
SHA256 9b5d633bb47a96ea8fd61b5ae4993deb62a104c0027831e1b5a226aed6b7fce7
MD5 5da8f1217a1861d9eb1251881639db11
BLAKE2b-256 eb298b2b5f79f3842406f06b68c4ab74ac43bd9c3ee04c97d93c196544679d7a

See more details on using hashes here.

File details

Details for the file py_to_proto-0.5.2-py39-none-any.whl.

File metadata

  • Download URL: py_to_proto-0.5.2-py39-none-any.whl
  • Upload date:
  • Size: 32.9 kB
  • Tags: Python 3.9
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/4.0.2 CPython/3.9.18

File hashes

Hashes for py_to_proto-0.5.2-py39-none-any.whl
Algorithm Hash digest
SHA256 a1a3553b6b0ce87fed08567615c444f1eeafacb3a124920a2ea02b91f62ac0e1
MD5 d6f3326bc021391abc0c8c29a1b5abae
BLAKE2b-256 09d3c95198744626cad2e209784cf3acf763e9299b5dd378b7393cfe49966f87

See more details on using hashes here.

File details

Details for the file py_to_proto-0.5.2-py38-none-any.whl.

File metadata

  • Download URL: py_to_proto-0.5.2-py38-none-any.whl
  • Upload date:
  • Size: 32.9 kB
  • Tags: Python 3.8
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/4.0.2 CPython/3.8.18

File hashes

Hashes for py_to_proto-0.5.2-py38-none-any.whl
Algorithm Hash digest
SHA256 91f48866c132fbd5901c065e73161195c108b3ce73edf549083c79fed5d44fb8
MD5 645a49a61a6033efedf3f532eab5408e
BLAKE2b-256 711f47f90ad339e1f3d0f72b58dd39e57786c1422dd32d37363a3bd5ebec91d1

See more details on using hashes here.

File details

Details for the file py_to_proto-0.5.2-py37-none-any.whl.

File metadata

  • Download URL: py_to_proto-0.5.2-py37-none-any.whl
  • Upload date:
  • Size: 32.9 kB
  • Tags: Python 3.7
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/4.0.2 CPython/3.7.17

File hashes

Hashes for py_to_proto-0.5.2-py37-none-any.whl
Algorithm Hash digest
SHA256 22021b46f216afa899751102d78ac090f23a757ea97635c77aabeff0106d6dad
MD5 bd93d00f678382ca5a725b4ea133f65b
BLAKE2b-256 760555f99fc5c4ba9a6cb048377955dbf1bc5b1ca8081e869ae00d48cb624365

See more details on using hashes here.

Supported by

AWS AWS Cloud computing and Security Sponsor Datadog Datadog Monitoring Fastly Fastly CDN Google Google Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Sentry Sentry Error logging StatusPage StatusPage Status page