Protosaurus

Parse and create Protobuf messages at runtime in Python — no protoc required. Also includes a CLI to deserialize Protobuf from Kafka using kcat and a schema registry.

Installation

Requires Python >= 3.12.

uv add protosaurus

Or using pip:

pip install protosaurus

Usage

Parse and serialize Protobuf in Python

Protosaurus can parse .proto definitions at runtime without using protoc. This allows Protobuf byte arrays to be converted to JSON and vice versa. The Context object is thread-safe and can be shared across threads.

import json
from protosaurus import Context
from base64 import b64decode

# create a context which stores the proto schemas
ctx = Context()

# add protos by specifying name and content
ctx.add_proto('diet.proto',
    """
    syntax = "proto3";
    enum Diet {
        carnivorous = 0;
        herbivorous = 1;
    }
    """)

# the proto can be imported via the specified name
ctx.add_proto('animal.proto',
    """
    syntax = "proto3";
    import "diet.proto";
    message Animal {
        string name = 1;
        Diet diet = 2;
        double length = 3;
    }
    """)

# convert a message from base64 string...
data = ctx.to_json('Animal', b64decode('CglJZ3Vhbm9kb24QARkAAAAAAAAkQA=='))

# ...or hex string
data = ctx.to_json('Animal', bytes.fromhex('0a09496775616e6f646f6e1001190000000000002440'))

print(data)
# >>> '{"name":"Iguanodon","diet":"herbivorous","length":10}'


# fields left at their default are omitted...
data = ctx.to_json('Animal', b64decode('CglJZ3Vhbm9kb24='))

print(data)
# >>> '{"name":"Iguanodon"}'

# ...unless they are requested explicitly
data = ctx.to_json('Animal', b64decode('CglJZ3Vhbm9kb24='), include_defaults=True)

print(data)
# >>> '{"name":"Iguanodon","diet":"carnivorous","length":0}'


# convert json to protobuf
data = ctx.from_json('Animal', json.dumps({"name":"Iguanodon","diet":"herbivorous","length":10}))

print(data)
# >>> b'\n\tIguanodon\x10\x01\x19\x00\x00\x00\x00\x00\x00$@'

to_json accepts the following keyword-only options, all False by default:

Deserialize Protobuf from Kafka using a schema registry

Protosaurus also ships a CLI that can deserialize Protobuf messages from Kafka automatically when a schema registry is available:

kcat -C -e -F <kafka.config> -t <topic> -f "%o\\n%k\\n%R%s" | protosaurus - --schema-registry <url>

To disable SSL certificate verification (e.g. for self-signed certificates), pass --no-verify:

kcat -C -e -F <kafka.config> -t <topic> -f "%o\\n%k\\n%R%s" | protosaurus - --schema-registry <url> --no-verify

The JSON output options above are available as flags: --defaults, --pretty, --proto-field-names, --enums-as-ints and --unquote-int64.

kcat -C -e -F <kafka.config> -t <topic> -f "%o\\n%k\\n%R%s" | protosaurus - --schema-registry <url> --defaults --pretty

Using uvx (no installation required):

kcat -C -e -F <kafka.config> -t <topic> -f "%o\\n%k\\n%R%s" | uvx protosaurus - --schema-registry <url>

License

ISC License — see LICENSE for details.