Protosaurus

Parse and create Protobuf messages at runtime in Python — no protoc required. Also includes a CLI to deserialize Protobuf from Kafka using kcat and a schema registry.

Installation

Requires Python >= 3.12.

uv add protosaurus

Or using pip:

pip install protosaurus

Usage

Parse and serialize Protobuf in Python

Protosaurus can parse .proto definitions at runtime without using protoc. This allows Protobuf byte arrays to be converted to JSON and vice versa. The Context object is thread-safe and can be shared across threads.

import json
from protosaurus import Context
from base64 import b64decode

# create a context which stores the proto schemas
ctx = Context()

# add protos by specifying name and content
ctx.add_proto('diet.proto',
    """
    syntax = "proto3";
    enum Diet {
        carnivorous = 0;
        herbivorous = 1;
    }
    """)

# the proto can be imported via the specified name
ctx.add_proto('animal.proto',
    """
    syntax = "proto3";
    import "diet.proto";
    message Animal {
        string name = 1;
        Diet diet = 2;
        double length = 3;
    }
    """)

# convert a message from base64 string...
data = ctx.to_json('Animal', b64decode('CglJZ3Vhbm9kb24QARkAAAAAAAAkQA=='))

# ...or hex string
data = ctx.to_json('Animal', bytes.fromhex('0a09496775616e6f646f6e1001190000000000002440'))

print(data)
# >>> '{"name":"Iguanodon","diet":"herbivorous","length":10}'


# fields left at their default are omitted...
data = ctx.to_json('Animal', b64decode('CglJZ3Vhbm9kb24='))

print(data)
# >>> '{"name":"Iguanodon"}'

# ...unless they are requested explicitly
data = ctx.to_json('Animal', b64decode('CglJZ3Vhbm9kb24='), include_defaults=True)

print(data)
# >>> '{"name":"Iguanodon","diet":"carnivorous","length":0}'


# convert json to protobuf
data = ctx.from_json('Animal', json.dumps({"name":"Iguanodon","diet":"herbivorous","length":10}))

print(data)
# >>> b'\n\tIguanodon\x10\x01\x19\x00\x00\x00\x00\x00\x00$@'

to_json accepts the following keyword-only options, all False by default:

from_json accepts one keyword-only option:

# fails: the schema has no such field
ctx.from_json('Animal', '{"name":"Iguanodon","colour":"green"}')

# succeeds, the unknown field is dropped
ctx.from_json('Animal', '{"name":"Iguanodon","colour":"green"}', ignore_unknown_fields=True)

Read varints from the wire format

read_varint decodes a single base-128 varint out of a bytes object and returns the value together with the position just after it, so consecutive reads need no state of their own:

from protosaurus import read_varint

data = b'\xac\x02\x08'

value, offset = read_varint(data)          # (300, 2)
value, offset = read_varint(data, offset)  # (8, 3)

Pass zigzag=True for the sint32/sint64 encoding, which maps signed values onto unsigned ones. Field tags, lengths, int32, int64, uint64, bool and enums are not zigzag encoded, so the default is off:

read_varint(b'\xac\x02')                # (300, 2)
read_varint(b'\xac\x02', zigzag=True)   # (150, 2)

Malformed input is rejected rather than read on indefinitely: data ending mid-varint raises EOFError, more than ten bytes raises RuntimeError, and an offset past the end raises IndexError.

Deserialize Protobuf from Kafka using a schema registry

Protosaurus also ships a CLI that can deserialize Protobuf messages from Kafka automatically when a schema registry is available:

kcat -C -e -F <kafka.config> -t <topic> -f "%o\\n%k\\n%R%s" | protosaurus - --schema-registry <url>

To disable SSL certificate verification (e.g. for self-signed certificates), pass --no-verify:

kcat -C -e -F <kafka.config> -t <topic> -f "%o\\n%k\\n%R%s" | protosaurus - --schema-registry <url> --no-verify

The JSON output options above are available as flags: --defaults, --pretty, --proto-field-names, --enums-as-ints and --unquote-int64.

kcat -C -e -F <kafka.config> -t <topic> -f "%o\\n%k\\n%R%s" | protosaurus - --schema-registry <url> --defaults --pretty

Using uvx (no installation required):

kcat -C -e -F <kafka.config> -t <topic> -f "%o\\n%k\\n%R%s" | uvx protosaurus - --schema-registry <url>

License

ISC License — see LICENSE for details.