protolib 0.3.0

Pure-Python, from-scratch declarative binary protocol (de)serializer — node-protodef style, for Minecraft, ClassiCube and any other binary protocol.

protolib (Python)

A from-scratch, pure-Python implementation of a node-protodef-style system: you describe the shape of a binary protocol's packets in a declarative file (.yml or .json), and the library takes care of parsing raw bytes into a Python dict and serializing a dict back into bytes — without you having to hand-write a parser for every single packet.

Already tested against two real protocols, each with its own .yml (hand-written/maintained) and its equivalent .json (the native ["type", opts] form, generated by the loader itself — so you can see exactly what the engine understands internally, and it's useful for consuming the protocol from something that doesn't want to parse YAML):

• Minecraft Java 1.8.9 (examples/example_protocol.yml / .json) — advanced patterns: parametrizable switch, relative compareTo, bitfield.
• Minecraft Classic / ClassiCube 0x07 + CPE (examples/classicube_protocol.yml / .json) — a complete fixed-size protocol, no varints, with CPE extensions.

---

1. The idea in 30 seconds

from protolib.core import Protocol

proto = Protocol("my_protocol.yml")

# bytes -> dict
parsed = proto.parse_packet("play", "toClient", raw_data)
print(parsed.name, parsed.params, parsed.bytes_read)

# dict -> bytes
data = proto.serialize_packet("play", "toClient", "spawn_player", {
    "playerId": 5, "x": 320, "y": 2080, "z": 320,
})

Everything else in this document is about how to declare my_protocol.yml so that parse_packet/serialize_packet know which fields to expect.

---

2. Install / use

No dependencies beyond pyyaml (for the .yml format). Used as a local package, no need to install it:

import sys
sys.path.insert(0, "path/to/the/folder/containing/protolib")
from protolib.core import Protocol

Protocol(...) accepts:

• a path to a .json, .yml, or .yaml file
• an in-memory string with JSON or YAML content
• an already-parsed dict (the "native" node-protodef format)

Heads up about relative paths: a path like "my_protocol.yml" is resolved relative to your process's current working directory (cwd), not relative to the script file that calls Protocol(...). If you run your script from a different folder than the one containing the .yml, the file won't be found there.

If the file genuinely can't be found, Protocol(...) now raises a clear LoaderError telling you the path it looked for and the cwd it used — it does not silently try to parse the filename string as protocol content (that used to happen and produced a confusing crash deep inside core.py instead of a real "file not found" error).

To make a script work no matter where it's run from, build the path relative to the script itself instead of relying on cwd:

import os
here = os.path.dirname(os.path.abspath(__file__))
proto = Protocol(os.path.join(here, "my_protocol.yml"))

---

3. Structure of a protocol file

types:
  # types valid across the ENTIRE protocol, regardless of state/direction
  varint: native
  myCustomType:
    container:
    - name: x
      type: varint

<state_name>:        # e.g.: "play", "login", "handshaking"...
  toClient:
    types:
      # types that only exist for this direction/state, take
      # PRIORITY over the ones above if there's a name collision
      packet: ...
  toServer:
    types:
      packet: ...

• global types: — shared types (primitives, common structs).
• <state>.toClient.types / <state>.toServer.types — the real entry point to parse a packet is always a type named packet inside one of these blocks. parse_packet(state, direction, data) looks for that packet, reads it, and returns {name, params} based on how you built it (see the "packet container with mapper+switch" pattern in section 7).

Important: Protocol.parse_packet/serialize_packet expect the packet type to ALWAYS have exactly two fields named name and params — that's what builds the ParsedPacket(name=..., params=...) you get back. The standard pattern (used in both example protocols) is:

packet:
  container:
  - name: name
    type:
      mapper:               # wire integer -> packet name
        type: u8            # (or varint, depending on your protocol)
        mappings:
          '0x00': identification
  - name: params
    type:
      switch:                # packet name -> its payload type
        compareTo: name
        fields:
          identification: packet_identification

If your packet doesn't follow this exact shape, parse_packet will fail with KeyError: 'name' — it's not a weird limitation of your protocol, it's literally how this library knows what to return.

state and direction don't have to be called that — they're simply the two keys you'll later use to call parse_packet/serialize_packet. A protocol without real "states" (like ClassiCube) still declares a single play: just to have somewhere to hang toClient/toServer.

---

4. The native .json format (without the YAML shorthand)

Everything in section 3 can be written directly in JSON, without going through YAML — it's the format already used by minecraft-data and the original node-protodef, and this library understands it with no translation needed. The difference with the .yml shorthand is one of shape, not semantics: each composite type (container, switch, array, bitfield, etc.) is written as a 2-element list instead of a single-key mapping:

["<base_type_name>", <options>]

The same switch from section 6, in YAML shorthand:

type:
  switch:
    compareTo: name
    fields:
      identification: packet_identification

...is this in native JSON:

{
  "type": [
    "switch",
    {
      "compareTo": "name",
      "fields": { "identification": "packet_identification" }
    }
  ]
}

And a container (which carries a list of fields instead of an options-dict) follows the same ["container", [...]] pattern:

"packet_identification": [
  "container",
  [
    { "name": "protocolVersion", "type": "u8" },
    { "name": "name", "type": "string64" }
  ]
]

Why would you still want to write .yml? Because by hand, several levels of nested 2-element lists are error-prone and hard to read (does that ] close the container or the outer switch?). The loader (protolib/loader.py) automatically translates the .yml shorthand into this native form before Protocol uses it — that's why example_protocol.json and example_protocol.yml (or classicube_protocol.json/.yml) are exactly equivalent: they serialize and parse byte-for-byte the same, one is just more convenient to write/maintain by hand than the other.

So when should you use the raw .json directly?

• You already have a protocol from minecraft-data or another node-protodef project in JSON and want to reuse it as-is, without rewriting it.
• You want to generate the protocol programmatically from another language or script that has no business knowing the shorthand syntax.
• You want to see "what the engine actually understands internally" with no ambiguity — useful for debugging a .yml that isn't loading the way you expected: generate its .json (see below) and check if the translation came out as you thought.

Generating .json from an existing .yml, using the loader itself (this guarantees it's faithful to what Protocol will actually read, not a separately hand-made translation):

import json
from protolib.loader import load_protocol_dict

d = load_protocol_dict("my_protocol.yml")
with open("my_protocol.json", "w", encoding="utf-8") as f:
    json.dump(d, f, indent=2, ensure_ascii=False)

This is how examples/classicube_protocol.json was generated, and examples/example_protocol.json already existed in this project.

---

5. native: how primitive types get registered

native in the .yml means "don't define this here, look it up by name in protolib/primitives.py". Already registered:

Name	Size	Notes
u8/i8	1 byte	unsigned/signed
u16/i16	2 bytes BE
u32/i32	4 bytes BE
u64/i64	8 bytes BE
f32/f64	4/8 bytes BE	float/double
lu16…lf64	same, little-endian	l prefix
varint/varlong	variable	signed zigzag LEB128 — no, see zigzag32/zigzag64 for those
bool	1 byte
void	0 bytes	useful as an empty case in a switch
cstring	variable	\0-terminated
UUID	16 bytes
nbt/optionalNbt	variable	Minecraft's NBT format
string64	64 fixed bytes	CP437, space-padded (ClassiCube)
buffer1024	1024 fixed bytes	raw, 0x00-padded (ClassiCube)
fixedCoord/fixedCoordDelta	2/1 bytes	fixed-point *32 (ClassiCube)

If your protocol needs a new type that isn't a composition of the ones above (say, a fixed-length string with a different encoding, or a buffer of another fixed size), add it in Python following the pattern of make_fixed_cp437_string/make_fixed_buffer in primitives.py, and register it in the PRIMITIVES dict at the end of the file. After that it's available as native in any .yml, for you and for anyone else using this library.

Non-native types (pstring, container, switch, array, bitfield, bitflags, buffer, mapper, option, count) are resolved by the Python engine (core.py) according to their own logic — explained below with examples.

---

6. Composite types, one by one

container — groups named fields, in order

packet_login:
  container:
  - name: protocolVersion
    type: varint
  - name: username
    type: string

Reads as {"protocolVersion": ..., "username": ...}.

anon: true field: instead of being stored under its own name, its sub-fields get merged directly into the parent container's dict. Useful for "breaking up" a bitfield or a switch into several loose fields at the same level (see bitfield below).

condition field: the field is only read/written if the condition (an expression over already-read fields) is true. Syntax: same path language as compareTo (section 7).

array — list of N elements of the same type

type:
  array:
    countType: varint    # reads a varint first, that's the length
    # count: otherField  # alternative: uses the VALUE of another already-read field
    type: myItemType

countType prepends an integer that's read/written automatically. count, on the other hand, references a sibling field already present (typically built with the count type, see section 9) — it doesn't write anything new, it just reads that existing value as the array's length.

switch — picks the field's type based on another field's value

type:
  switch:
    compareTo: name        # path to an already-read field (see section 7)
    fields:
      valueA: typeForA
      valueB: typeForB
    default: void           # optional, if nothing matches

This is the central piece for "the params field depends on what the name field says" — the pattern of every network packet with an id/name.

mapper — integer ⟷ symbolic name

type:
  mapper:
    type: varint
    mappings:
      '0x00': handshake
      '0x01': status_request

When reading, a 0x00 in the stream becomes the string "handshake" in the resulting dict (and vice versa when writing). This way the switch above can compare against readable names (compareTo: name) instead of the raw packet id integer.

bitfield — several fields packed into less than 1 byte each

type:
  bitfield:
  - name: type
    size: 3
    signed: false
  - name: index
    size: 5
    signed: false

Declares sub-fields that each take up size bits, MSB first, within the minimum necessary bytes (if they don't fill an exact byte, padding is added). Almost always combined with anon: true on the container that holds it, so that type/index end up as loose fields instead of nested under an extra name.

bitflags — an integer as a named set of booleans

type:
  bitflags:
    type: u8
    flags: [air, water, lava, null, fire]   # position = bit number
    # or, as a dict: {air: 0x01, water: 0x02, ...}

Returns {"air": true, "water": false, ..., "_value": <raw integer>}.

buffer — raw bytes (uninterpreted)

type:
  buffer:
    count: 1024        # fixed size, or a reference to a field
    # countType: varint  # alternative: prefixed length
    # rest: true         # alternative: "whatever's left in the frame"

option — present only if a preceding boolean flag says so

type:
  option: myType

Reads a bool first; if true, reads myType right after; if false, the value is None and nothing else is read.

pstring — string with a prefixed length

type:
  pstring:
    countType: varint     # how many length-bytes come before it
    # encoding: utf-8      # default

The string: shortcut already ships built with this (countType: varint, used in Minecraft Java). For ClassiCube, string64 is used instead (ALWAYS fixed size, no prefix — see section 5), since that protocol doesn't prefix a length.

---

7. compareTo and field paths (../, /)

compareTo (in a switch) and condition (on a container field) use the same tiny path language to refer to "a field that's already been read":

Syntax	Means
fieldName	sibling field, same container
../fieldName	goes up one level: the container that holds this one
../../fieldName	goes up two levels
/fieldName	absolute, from the root of the whole packet

Important gotcha, already documented with a full example in examples/README.md: ../ only goes up one real level when the container is written inline (literally inside array.type or another container). If instead you reference a type by name (type: myNamedType), that named type always adds its own nesting level — so a ../field inside it reaches, at most, the named container itself, not the real parent further up. If your switch needs to see a field higher up in the hierarchy, declare the container inline at the point of use instead of moving it into a separately named type.

---

8. Parametrizable types ($arg)

A named type can declare a $-prefixed placeholder instead of a fixed value, and whoever uses it decides the real value at that point:

types:
  itemByType:
    switch:
      compareTo: $compareTo    # placeholder -- the name after
      fields:                 # the $ is arbitrary, you choose it
        '0': i8
        '1': varint

  myContainer:
    container:
    - name: type
      type: u8
    - name: value
      type:
      - itemByType             # referenced as [name, options]
      - compareTo: type          # here the real $compareTo gets resolved

This defines a "generic" type once and reuses it with a different comparison field depending on context — it's exactly how minecraft-data defines entityMetadataItem.

---

9. count — a length-prefix declared apart from the array/buffer

For the (rare, but real in some protocols) case where an array's length prefix isn't attached to that array but is instead its own field elsewhere in the container:

container:
- name: itemCount
  type:
    count:
      type: varint
      countFor: items      # when WRITING, ignores the given value and
- name: items               # writes len(items) automatically
  type:
    array:
      count: itemCount   # when READING, references that already-read field
      type: myItem

---

10. The 4 advanced Minecraft Java patterns

Covered with working, commented examples in examples/README.md + examples/example_protocol.yml (its .json equivalent, already translated to the native ["type", opts] form, lives in examples/example_protocol.json):

1. Parametrizable named type with $arg (section 8 of this doc).
2. compareTo: ../field and the inline-vs-named container gotcha (section 7).
3. Array of containers with a switch inside that looks at a field of the array's parent container.
4. bitfield packed into 1 byte + switch that uses that field to pick the type of another sibling field (real pattern from entity_metadata in Minecraft 1.8.9-1.16).

Run python examples/demo.py to see them in action.

---

11. Full reference protocol: ClassiCube

examples/classicube_protocol.yml (equivalent in examples/classicube_protocol.json, byte-for-byte identical when serializing/parsing — generated by the loader itself, not hand-written) has the complete 0x07 protocol (Identification, Level Init/Data/ Finalize, Set Block, Spawn/Position/Despawn Player, Message, Disconnect, Update User Type) plus several CPE extensions (ExtInfo/ ExtEntry, ClickDistance, CustomBlockSupportLevel, HeldBlock, ExtPlayerList) — a good example of a protocol with no varints, all fixed size, useful if your use case doesn't look like Minecraft Java's.

examples/servidor.py is a real, minimal server (login + generates a world and sends it) built entirely on top of Protocol.parse_packet / serialize_packet — it's an example of how to integrate this library with real sockets (a recv() loop accumulating bytes until parse_packet stops throwing BufferUnderrun).

cd examples
python servidor.py               # caves, port 25565
python servidor.py --modo llano  # flat world

---

12. Errors and what they mean

All of them live in protolib/errors.py:

• UnknownTypeError — a type name isn't in native (primitives.py) nor defined in types: (global or scoped). Usually a typo, or you forgot to declare it.
• SwitchCaseNotFound — a switch has no fields entry for the value compareTo produced, and there's no default either. Check that the comparison value resolves to what you expect (does the mapper building that field convert it to a string? does compareTo point to the right field using the syntax from section 7?).
• InvalidTypeDefinition — a composite type is missing a required option (array without count/countType, buffer without count/countType/rest, etc.).
• BufferUnderrun (in protolib/io.py, not errors.py) — an attempt was made to read more bytes than are available. In a real server this is NOT necessarily a protocol error: it means "the full packet hasn't arrived on the socket yet," and the correct pattern is to catch it and wait for more bytes (see the loop in servidor.py).

Changelog

0.2.1

• New primitive buffer64: fixed 64-byte raw bytes with automatic \x00 padding (same semantics as buffer1024, built with the same make_fixed_buffer mold). Added while migrating PocketNet to a single protocol.yml — the data field of PluginMessagePacket (0x35) needed exactly this behavior, and the generic buffer with a fixed count is strict (fails if the value isn't exactly N bytes long instead of truncating/padding).
• New example: examples/pocketnet_protocol.yml, the complete Minecraft Classic 0.30 + CPE protocol (55 packets) migrated from the "one class per packet" style into a single declarative file. Tested with byte-exact round-trips against the original encode()/ decode() on the most complex cases (packed bitfields, nested arrays of containers, padded buffers).

0.2.0

• Base version: primitives, composites (container, array, switch, mapper, option, bitfield, bitflags, buffer, pstring, count, entityMetadataLoop, topBitSetTerminatedArray), JSON and YAML support, native NBT and UUID.