protobuf-schematics 0.4.1

Convert ProtoBuf proto file to cute Schematics classes file

ProtoBuf Schematics

Convert ProtoBuf proto file to cute Schematics classes file.

Google Protobuf is great when it comes to high performance schema aware APIs, but when Google designed Protobuf, it didn’t tried to make the generated code idiomatic in Python, which brings a problem when exporting messages outside interface modules or having nice IDE auto-completions. Schematics is a cute and Pythonic schema library that goes well with most applications. Why not join both?

Currently this package does not support the Protobuf binary format and will work with a any other textual representation which is easily generated with the original Protobuf API for any language. Ease of use was prioritized while writing this package rather than mere performance.

• Free software: Apache Software License 2.0

• Documentation: https://protobuf-schematics.readthedocs.io.

Usage

1. Convert the proto file to python Schematics classes:

   protobuf_schematics <path-to-file.proto> generated_schematics_proto.py # or any output filename

2. Convert your ProtoBuf message to Json:

In Java:

import com.google.protobuf.util.JsonFormat;

FileWriter file = new FileWriter("protoBufMessage.json")
JsonFormat.Printer printer = JsonFormat.printer().preservingProtoFieldNames();
String message = printer.print(someProtoBufMessage);
file.write(message)

or from Python:

import json
from google.protobuf.json_format import MessageToJson

json = MessageToJson(org, preserving_proto_field_name=True, including_default_value_fields=True)
with open("protoBufMessage.json", 'w') as output:
    json.dump(json, output)

3. In your project, load the message in python as Schematics object:

import json
from generated_schematics_proto import SomeMessage # import the schematics message class

schematics_root_message = SomeMessage(json.load(open('protoBufMessage.json')))

Or use a message loaded in python by Protobuf:

import json
from generated_schematics_proto import SomeMessage # import the schematics message class

# ... get your protobuf message as the pb class representation
schematics_root_message = SomeMessage.import_from_protobuf_message(protobuf_message)

Example

This proto file:

syntax = "proto3";

enum IPAddressFamily {
    INVALID = 0;
    IPv4 = 1;
    IPv6 = 2;
};

message ProtocolAndPorts {
    repeated uint32 ports = 3;
}

message FlowFilter {
    enum SomeEnum {
        VALUE = 0;
    };
    string id = 1 [deprecated = true];
    SomeEnum consumer_filter_id = 2;
    map<string, ProtocolAndPorts> ports = 3;
    repeated ProtocolAndPorts protocol_and_ports = 4;
}

Will be converted to:

class IPAddressFamily(Enum):
    INVALID = 0
    IPv4 = 1
    IPv6 = 2


class ProtocolAndPorts(ProtobufMessageModel):
    ports = ListType(IntType())


class FlowFilter(ProtobufMessageModel):
    class InnerEnum(Enum):
        VALUE = 0

    id = StringType()
    consumer_filter_id = EnumType(SomeEnum)
    ports = DictType(ModelType(ProtocolAndPorts), str)
    protocol_and_ports = ListType(ModelType(ProtocolAndPorts))

Features

• Support both Protobuf syntax 2 and 3.

• Support builtin types such as StringType, IntType.

• Support proto map fields as Schematics DictType.

• Support repeated modifier as convert to ListType.

• Support Enum class generation and custom Schematics EnumType.

• Support custom schematics ByteArrayType base64 encoded byte arrays converted from Java.

Development

First, install the Pipfile and create the proper virtual environment:

pipenv install --dev

To check linting with flake8, run:

make lint

To run the unittests against your working python version:

py.test

To see coverage report:

make coverage

To run tests against all supported python versions:

tox

To make the docs (which will be automatically published to readthedocs on commits to the master branch):

make docs

Credits

The parsing work of .proto files is provided thanks to the awesome guys at PyroBuf.

This package was created with Cookiecutter and the elgertam/cookiecutter-pipenv project template, based on audreyr/cookiecutter-pypackage.

History

TODO

• Add more meaningful unittests.

• Improve Protobuf support and integration.

• add support for parsing the Protobuf generated files as it is more comprehensive than the current Pyrobuf support.

Read for Release

0.4.0 (2019-01-16)

• Added more meaningful unittests.

• Export Types and Models out of the templates.

• Export heavy definition logic (such as field definition) from the jinja template to pythonic filters.

• Improve integration with Protobuf.

0.2.0 (2019-01-14)

• Improve README - Add usage examples.

• Update the setup.py to point to official PyroBuf 0.8.5 PyPi release.

0.1.0 (2019-01-13)

• First release on PyPI.

• CLI interface for compiling a given proto file to schematics definition file.