protox 0.3.0

Protobuf for humans

Protox

Protocol buffers implementation for Python 3 that generates human-readable code with type hinting support

Quick example

from enum import IntEnum
from protox import Message
from protox import UInt32, String, EnumField


class User(Message):
    class Role(IntEnum):
        USER = 1
        ADMIN = 2

    id: int = UInt32(number=1, required=True)
    email: str = String(number=2, required=True)
    role: Role = EnumField(Role, number=3, required=True, default=Role.USER)


user = User(id=1, email="[e-mail]", role=User.Role.USER)
user_data = user.to_bytes()

print(User.from_bytes(user_data))
# Outputs:
# message User
#     id = 1
#     email = '[e-mail]'
#     role = Role.USER

More examples

• EXAMPLES.md
• Grpclib example

Code generation

Code generator implemented as a protoc plugin so you have to install it first:

Install protoc (Ubuntu)

sudo apt install protobuf-compiler

Install protoc (Mac OS)

brew install protobuf

Install protox

python3 -m pip install protox

Generate messages

protoc \
    --proto_path=protobuf_src \
    --protox_out=. \
    ./protobuf_src/user.proto

Generate grpclib service with all required dependencies

The following code generates service ping_pong.proto from the protobuf_src directory into {$PROJECT_ROOT}/app/protobuf.

The protox plugin supports 3 options:

• --base-package=path/to/protobuf/out
• --grpclib # generates grpclib services
• --with-dependencies # generates dependencies for the given protobuf file

The options to the plugin are passed using the --protox_opt="{plugin options here}" command

protoc \
    --proto_path=protobuf_src \
    --protox_out=. \
    --protox_opt="--base-package=app/protobuf --grpclib --with-dependencies" \
    ./protobuf_src/ping_pong.proto

Core concepts

• Human-readable python3.6+ generated code with type hinting
• Support protobuf 2 and 3 at the same time
• None values instead of zero values in fields for both proto2 and proto3
• Message.has_field() in both proto2 and proto3
• Protocols are easy to describe without code generation
• Messages implemented in more pythonic way: to_bytes() instead of SerializeToString()
• Enums are just enums python int enums

Features

• Messages
• Enums
• Nested messages
• Maps
• Well-known types
• Repeated fields
• Repeated messages
• Repeated enums
• Custom Message.to_python() / from_python() functions
• Group fields (Deprecated by protobuf)

Code generator features

• Protobuf
• Grpclib
• Grpc.io
• Custom python package for protobuf out messages
• Compile protobuf file with dependencies
• Names mangling to avoid reserved names collisions
• Recursive messages/enums support
• Field names to_snake_case support

Difference with google's protobuf implementation

Binary protocol works exactly as google's implementation does.

The difference is in the way messages behave:

• Fields that were not explicitly set are None rather than zero-values
• Methods like SerializeToString() were changed to more pythonic alternatives like to_bytes() / from_bytes()
• Enums are just python int enums

Generated code example

Grpclib service example

from enum import IntEnum

import protox


class User(protox.Message):
    class Status(IntEnum):
        USER = 1
        ADMIN = 2

    id: int
    email: str
    status: 'User.Status'

    def __init__(
        self,
        *,
        id: int = None,
        email: str = None,
        status: 'User.Status' = None,
    ):
        super().__init__(
            id=id,
            email=email,
            status=status,
        )


protox.define_fields(
    User,
    id=protox.Int32(
        number=1, required=True
    ),
    email=protox.String(
        number=2, required=True
    ),
    status=protox.EnumField(
        number=3, py_enum=User.Status, default=User.Status.USER, required=True
    ),
)