protosaurus 0.4.0

Protocol Buffers at runtime

Protosaurus

Parse and create Protobuf messages at runtime in Python — no protoc required. Also includes a CLI to deserialize Protobuf from Kafka using kcat and a schema registry.

Installation

Requires Python >= 3.12.

uv add protosaurus

Or using pip:

pip install protosaurus

Usage

Parse and serialize Protobuf in Python

Protosaurus can parse .proto definitions at runtime without using protoc. This allows Protobuf byte arrays to be converted to JSON and vice versa. The Context object is thread-safe and can be shared across threads.

import json
from protosaurus import Context
from base64 import b64decode

# create a context which stores the proto schemas
ctx = Context()

# add protos by specifying name and content
ctx.add_proto('diet.proto',
    """
    syntax = "proto3";
    enum Diet {
        carnivorous = 0;
        herbivorous = 1;
    }
    """)

# the proto can be imported via the specified name
ctx.add_proto('animal.proto',
    """
    syntax = "proto3";
    import "diet.proto";
    message Animal {
        string name = 1;
        Diet diet = 2;
        double length = 3;
    }
    """)

# convert a message from base64 string...
data = ctx.to_json('Animal', b64decode('CglJZ3Vhbm9kb24QARkAAAAAAAAkQA=='))

# ...or hex string
data = ctx.to_json('Animal', bytes.fromhex('0a09496775616e6f646f6e1001190000000000002440'))

print(data)
# >>> '{"name":"Iguanodon","diet":"herbivorous","length":10}'


# convert json to protobuf
data = ctx.from_json('Animal', json.dumps({"name":"Iguanodon","diet":"herbivorous","length":10}))

print(data)
# >>> b'\n\tIguanodon\x10\x01\x19\x00\x00\x00\x00\x00\x00$@'

Deserialize Protobuf from Kafka using a schema registry

Protosaurus also ships a CLI that can deserialize Protobuf messages from Kafka automatically when a schema registry is available:

kcat -C -e -F <kafka.config> -t <topic> -f "%o\\n%k\\n%R%s" | protosaurus - --schema-registry <url>

To disable SSL certificate verification (e.g. for self-signed certificates), pass --no-verify:

kcat -C -e -F <kafka.config> -t <topic> -f "%o\\n%k\\n%R%s" | protosaurus - --schema-registry <url> --no-verify

Using uvx (no installation required):

kcat -C -e -F <kafka.config> -t <topic> -f "%o\\n%k\\n%R%s" | uvx protosaurus - --schema-registry <url>

License

ISC License — see LICENSE for details.