protoletariat 0.5.0

Python protocol buffers for the rest of us

Protocol Buffers for the Rest of Us

Motivation

protoletariat has one goal: fixing the broken imports for the Python code generated by protoc.

Usage

Here's an example of how to use the tool, called protol:

1. Create a few protobuf files

// thing1.proto
syntax = "proto3";

import "thing2.proto";

package things;

message Thing1 {
  Thing2 thing2 = 1;
}

// thing2.proto
syntax = "proto3";

package things;

message Thing2 {
  string data = 1;
}

2. Run protoc on those files

$ mkdir out
$ protoc \
  --python_out=out \
  --proto_path=directory/containing/protos thing1.proto thing2.proto

3. Run protol on the generated code

$ protol \
  --create-package \
  --in-place \
  --python-out out \
  protoc --proto-path=directory/containing/protos thing1.proto thing2.proto

The out/thing1_pb2.py file should show a diff containing at least these lines:

-import thing2_pb2 as thing2__pb2
-
+from . import thing2_pb2 as thing2__pb2

How it works

At a high level protoletariat converts absolute imports to relative imports.

However, it doesn't convert just any absolute import to a relative import.

The protol tool will only convert imports that were generated from .proto files. It does this by inspecting FileDescriptorProtos from the protobuf files.

The core rewrite mechanism is implemented using a simplified form of pattern matching, that looks at the Python AST, and if an import pattern is matched a corresponding rewrite rule is invoked.

Help

$ protol
Usage: protol [OPTIONS] COMMAND [ARGS]...

  Rewrite protoc or buf-generated imports for use by the protoletariat.

Options:
  -o, --python-out DIRECTORY      Directory containing protoc or buf-generated Python code  [required]
  --in-place / --not-in-place     Overwrite all relevant files under `--python-out` with adjusted imports  [default: not-in-place]
  --create-package / --dont-create-package
                                  Recursively create __init__.py files under `--python-out`  [default: dont-create-package]
  -s, --module-suffixes TEXT      Suffixes of Python modules to process  [default: _pb2.py, _pb2_grpc.py]
  --help                          Show this message and exit.

Commands:
  buf     Use buf to generate the FileDescriptorSet blob
  protoc  Use protoc to generate the FileDescriptorSet blob