Skip to main content

Python Proto Converter

The Python Proto Converter converts between protos in Python. Proto conversion is often needed when converting between Database Access Object (DAO) and API proto.

Install

pip install python-proto-converter

Run the example

  1. Build the proto (assuming in exmaple/ directory) protoc -I=. --python_out=. ./example_proto.proto

  2. execute python3 ./converter_example.py

Features

  • A base class that auto-converts fields with the same name and type.
  • Custom convert functions can be implemented to handle fields conversion.
  • Fields can be disabled during auto-converting.
  • Unhandled fields assertion during class instantiation.

Example

Basic usage

Let's start with a simple example, suppose you want to convert from one similar proto to another. For this example, these are the MatchaMilkTea to GreenTeaMilkTea protos.

message MatchaMilkTea {
  string name = 1;
  float price = 2;
  string seller = 3;
}
message GreenTeaMilkTea {
  string name = 1;
  int64 price = 2;
  string seller = 3;
}

The name and seller fields can be auto-converted, since the type and the field name are identical. However, we probably don't want to copy the name of MatchaMilkTea to GreenTeaMilkTea. To disable auto-convert on the name field, we mark it ignored and provide our custom function for the name field.

The price field has different types (float vs int64), therefore it can't be auto-converted. Leaving it unhandled will trigger an exception when creating the proto converter. Similar to the name field, we can create a custom method to convert the price field.

from google3.alkali.contrib.certified.python.proto import converter

...

class MatchaToGreenTeaConverter(converter.ProtoConverter):
  def __init__(self):
    super(MatchaToGreenTeaConverter, self).__init__(
        pb_class_from=matcha_milk_tea_pb2.MatchaMilkTea,
        pb_class_to=green_tea_milk_tea_pb2.GreenTeaMilkTea,
        field_names_to_ignore=["name"])

  @converter.convert_field(field_names=["price"])
  def price_convert_function(self, src_proto, dest_proto):
    dest_proto.price = int(src_proto.price)

  @converter.convert_field(field_names=["name"])
  def name_convert_function(self, src_proto, dest_proto):
    dest_proto.name = "GreenTeaMilkTea"

Or you can combine them in the same method since these fields are simple:

@converter.convert_field(field_names=["price", "name"])
def price_name_convert_function(self, src_proto, dest_proto):
  dest_proto.price = int(src_proto.price)
  dest_proto.name = "GreenTeaMilkTea"

Now you can create the converter in code and use it:

...
matcha_to_green_tea_converter = MatchaToGreenTeaConverter()
green_tea_milk_tea_proto = matcha_to_green_tea_converter.convert(matcha_milk_tea_proto)
...

Nested protos

Let's make this example a bit more complicated by adding some fields.

enum Flavor {
  GREEN_TEA = 0;
  MATCHA = 1;
  BERRY = 2;
  SPICY = 3;
}

message MilkTea {
  string name = 1;
  float price = 2;
  Flavor flavor = 3;
}
message MatchaMilkTea {
  MilkTea milk_tea = 1;
  int64 sugar = 2;
  repeated string shops = 3;
  string matcha_provider = 4;
  map<string, int64> ingredients = 5;
  map<string, string> ingredients_calorie_map = 6;
  repeated string cup_sizes = 7;
}
message GreenTeaMilkTea {
  MilkTea milk_tea = 1;
  float sugar = 2;
  repeated string shops = 3;
  string green_tea_provider = 4;
  map<string, int64> ingredients = 5;
  map<string, int32> ingredients_calorie_map = 6;
  repeated int64 cup_sizes = 7;
}

Most of the fields are identical and can be auto-converted, except:

  • float sugar and int64 sugar;
  • string green_tea_provider;
  • string matcha_provider;
  • ingredients_calorie_map;
  • cup_sizes;

You can create a new MatchaToGreenTeaConverter class that inherits ProtoConverter to convert from MatchaMilkTea to GreenTeaMilkTea:

from google3.alkali.contrib.certified.python.proto import converter

...

class MatchaToGreenTeaConverter(converter.ProtoConverter):
  def __init__(self):
    super(MatchaToGreenTeaConverter, self).__init__(
        pb_class_from=matcha_milk_tea_pb2.MatchaMilkTea,
        pb_class_to=green_tea_milk_tea_pb2.GreenTeaMilkTea,
        field_names_to_ignore=["ingredients_calorie_map", "cup_sizes"])

  @converter.convert_field(field_names=["sugar"])
  def sugar_convert_function(self, src_proto, dest_proto):
    dest_proto.sugar = int(src_proto.sugar)

  @converter.convert_field(field_names=["matcha_provider"])
  def provider_convert_function(self, src_proto, dest_proto):
    dest_proto.green_tea_provider = src_proto.matcha_provider
  • pb_class_from and pb_class_to are the constructors of the protos.
  • pb_class_from.Fields in field_names_to_ignore will be ignored during auto-conversion and when validating that all fields have been handled. In the example, ingredients_calorie_map and cup_sizes are ignored during conversion.
  • @converter.convert_field decorates a custom conversion function. In this example, we have two functions to convert the sugar field and the matcha_provider field.
  • All fields that can't be auto-converted from the source proto must either be handled by custom conversion functions or listed in field_names_to_ignore.

Oneof fields

Oneof fields can be tricky and error-prone, therefore it is required to explicitly handle or ignore all the fields in oneofs.

message MochiFlavor {
  string flavor = 1;
}

message Mochi {
  oneof price {
    string price_str = 1;
    float price_float = 2;
  }
  oneof flavor {
    Flavor flavor_enum = 3;
    MochiFlavor flavor_proto = 4;
  }
  int64 calorie = 5;
}
message TaroMochi {
  float price_float = 1;
  MochiFlavor flavor_proto = 2;
  int64 calorie = 3;
}
proto_converter = converter.ProtoConverter(
        pb_class_from=mochi_pb2.Mochi,
        pb_class_to=mochi_pb2.Taromochi,
        field_names_to_ignore=["flavor_enum", "price_str"])
src_proto = mochi_pb2.Mochi(
        price_float=3.14,
        flavor_proto=mochi_pb2.MochiFlavor(flavor="taro"),
        calorie=100)

dest_proto = proto_converter.convert(src_proto=src_proto)

In the above example, even though flavor_enum and price_str fields are not used, ProtoConverter will still raise an exception if these fields are not ignored.

Any fields

Proto to Any and Any to Any are converted automatically as long as the field name matches.

message AnyMochiBox {
  string name = 1;
  google.protobuf.Any mochi = 2;
}

message TaroMochiBox {
  string name = 1;
  TaroMochi mochi = 2;
}

In the example below, ProtoConverter auto-converts a TaroMochi field to a Any field.

taro_mochi = mochi_pb2.TaroMochi(price_float=3.14,
flavor_proto=mochi_pb2.MochiFlavor(flavor="taro"), calorie=100)
proto_converter = converter.ProtoConverter(
pb_class_from=mochi_pb2.TaroMochiBox, pb_class_to=mochi_pb2.AnyMochiBox)

src_proto = mochi_pb2.TaroMochiBox(name="TaroMochiBox", mochi=taro_mochi)

dest_proto = proto_converter.convert(src_proto=src_proto)

Similarily, ProtoConverter auto-converts Proto Any field to Any field.

taro_mochi = mochi_pb2.TaroMochi(
        price_float=3.14,
        flavor_proto=mochi_pb2.MochiFlavor(flavor="taro"),
        calorie=100)
taromochi_any_proto = any_pb2.Any()
taromochi_any_proto.Pack(taro_mochi)
proto_converter = converter.ProtoConverter(
    pb_class_from=mochi_pb2.AnyMochiBox, pb_class_to=mochi_pb2.AnyMochiBox)
src_proto = mochi_pb2.AnyMochiBox(
    name="TaroMochiBox", mochi=taromochi_any_proto)

dest_proto = proto_converter.convert(src_proto=src_proto)

Repeated Any field and Map Any field are also supported.

message AnyMochiBoxes {
  string name = 1;
  repeated google.protobuf.Any mochi = 2;
}

message TaroMochiBoxes {
  string name = 1;
  repeated TaroMochi mochi = 2;
}

message MochiGiftPackage {
  string name = 1;
  map<string, google.protobuf.Any> mochi = 2;
}

message TaroMochiGiftPackage {
  string name = 1;
  map<string, google.protobuf.Any> mochi = 2;
}

The examples below demonstrate the auto-conversion for repeated fields and Map fields with Any proto.

proto_converter = converter.ProtoConverter(
        pb_class_from=mochi_pb2.TaroMochiBoxes,
        pb_class_to=mochi_pb2.AnyMochiBoxes)
src_proto = mochi_pb2.TaroMochiBoxes(name="TaroMochiBoxes",
                                     mochi=[taro_mochi, taro_mochi])
dest_proto = proto_converter.convert(src_proto=src_proto)
proto_converter = converter.ProtoConverter(
        pb_class_from=mochi_pb2.TaroMochiGiftPackage,
        pb_class_to=mochi_pb2.AnyMochiGiftPackage)
src_proto = mochi_pb2.TaroMochiGiftPackage(
    name="TaroMochiBoxes",
    mochi={"taro_mochi": taro_mochi})
dest_proto = proto_converter.convert(src_proto=src_proto)

We decided not to support Any field to Proto field auto conversion to make it less error-pone, since the Any field can contain any type and cause runtime failures. However, it is very easy to add a custom method to handle Any field.

class MochiConverter(converter.ProtoConverter):

  @converter.convert_field(field_names=["mochi"])
  def mochi_field_convert_function(self, src_proto, dest_proto):
    src_proto.mochi.Unpack(dest_proto.mochi)

...

taro_mochi = mochi_pb2.TaroMochi(
        price_float=3.14,
        flavor_proto=mochi_pb2.MochiFlavor(flavor="taro"),
        calorie=100)
taromochi_any_proto = any_pb2.Any()
taromochi_any_proto.Pack(taro_mochi)
proto_converter = MochiConverter(pb_class_from=mochi_pb2.AnyMochiBox,
                                 pb_class_to=mochi_pb2.TaroMochiBox)
src_proto = mochi_pb2.AnyMochiBox(
        name="TaroMochiBox", mochi=_pack_to_any_proto(taro_mochi))

dest_proto = proto_converter.convert(src_proto=src_proto)

Repeated Any field to repeated Proto field

class RepeatedMochiConverter(converter.ProtoConverter):

  @converter.convert_field(field_names=["mochi"])
  def mochi_field_convert_function(self, src_proto, dest_proto):
    for field in src_proto.mochi:
      proto_object = mochi_pb2.TaroMochi()
      field.Unpack(proto_object)
      dest_proto.mochi.append(proto_object)

Map Any field to Map Proto field

class MapMochiConverter(converter.ProtoConverter):

  @converter.convert_field(field_names=["mochi"])
  def mochi_field_convert_function(self, src_proto, dest_proto):
    for key, field in src_proto.mochi.items():
      proto_object = mochi_pb2.TaroMochi()
      field.Unpack(proto_object)
      dest_proto.mochi[key].CopyFrom(proto_object)

Nested conversion

Nested conversion is supported if the source proto and destination proto contains the same proto type (like the above example), while auto-conversion won't work if the nested protos are of different type.

However, it's very easy to support this case with a custom method. We think it's cleaner to create separate converters as you will see in the below example.

message TaroMochi {
  float price_float = 1;
  MochiFlavor flavor_proto = 2;
  int64 calorie = 3;
}

message CocoMochi {
  float price_float = 1;
  MochiFlavor flavor_proto = 2;
  int64 calorie = 3;
}

message TaroMochiBox {
  string name = 1;
  TaroMochi mochi = 2;
}

message CocoMochiBox {
  string name = 1;
  CocoMochi mochi = 2;
}
class NestedMochiBoxConverter(converter.ProtoConverter):
  taro_to_coco_converter: converter.ProtoConverter = None

  def __init__(self):
    super(RecursiveMochiBoxConverter, self).__init__(
        pb_class_from=mochi_pb2.TaroMochiBox,
        pb_class_to=mochi_pb2.CocoMochiBox
    )
    self.taro_to_coco_converter = converter.ProtoConverter(
        pb_class_from=mochi_pb2.TaroMochi, pb_class_to=mochi_pb2.CocoMochi)

  @converter.convert_field(field_names=["mochi"])
  def mochi_field_convert_function(self, src_proto, dest_proto):
    dest_proto.mochi.CopyFrom(
      self.taro_to_coco_converter.convert(src_proto.mochi))

...

proto_converter = NestedMochiBoxConverter()
dest_proto = proto_converter.convert(src_proto)

With the additional ProtoConverter between TaroMochi and CocoMochi, it's very easy to update the conversion once the TaroMochi or CocoMochi proto changes.

For nested array protos, we need to iterate through each element and append the conversion result to the destination proto:

message CocoMochiBoxes {
  string name = 1;
  repeated CocoMochi mochi = 2;
}

message TaroMochiBoxes {
  string name = 1;
  repeated TaroMochi mochi = 2;
}
@converter.convert_field(field_names=["mochi"])
def mochi_field_convert_function(self, src_proto, dest_proto):
  for mochi in src_proto.mochi:
    dest_proto.mochi.append(self.taro_to_coco_converter.convert(mochi))

Contributing

See CONTRIBUTING.md for details.

License

Apache 2.0; see LICENSE for details.

Disclaimer

This project is not an official Google project. It is not supported by Google and Google specifically disclaims all warranties as to its quality, merchantability, or fitness for a particular purpose.

Metadata

Release files for python-proto-converter 1.0.4

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for python-proto-converter 1.0.4
File Size Uploaded Provenance Actions
python_proto_converter-1.0.4.tar.gz 15.8 kB Without provenance Details for python_proto_converter-1.0.4.tar.gz

Built distribution (wheel)

Table of built distributions (wheels) for python-proto-converter 1.0.4
File Interpreter ABI Platform Actions
python_proto_converter-1.0.4-py3-none-any.whl Python 3 none any Details for python_proto_converter-1.0.4-py3-none-any.whl

Total release size: 29.0 kB

Release files / python_proto_converter-1.0.4.tar.gz

Download URL python_proto_converter-1.0.4.tar.gz
Size 15.8 kB
Tags Source
SHA-256 checksum
How to use checksums
e72b3e488ebcbd6970a62d7b59d607bda79f47abac3c86dd64466582734eb2a9
BLAKE2b-256 checksum
How to use checksums
c55f6b5920c45271e51070992640e3e7f45f0bcfe2dd9b4510333e38830f1bfa
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via twine/6.2.0 CPython/3.13.7

Release files / python_proto_converter-1.0.4-py3-none-any.whl

Download URL python_proto_converter-1.0.4-py3-none-any.whl
Size 13.2 kB
Tags Python 3
SHA-256 checksum
How to use checksums
9feee3cecbfde527fea7d42b763c39d673f7974205edff62102d6ac5f55fb3de
BLAKE2b-256 checksum
How to use checksums
dd3bb9ef462ad00002a12d58cd5410cd34574a23d5a76cffe7fb4f65b8c69e29
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via twine/6.2.0 CPython/3.13.7

Security for python-proto-converter 1.0.4

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

No provenance found

None of the files in this release were uploaded with provenance metadata.

A note on package security

PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.

To learn more about protecting yourself from malicious actors, visit our package security documentation.

Release history Release notifications | RSS feed

This release

1.0.4 This release

2 release files No provenance

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page