dybuf python package

Python bindings for the dybuf dynamic buffer library implemented in C. The package exposes the DyBuf class implemented in Cython and ships binary wheels across the major desktop platforms.

Key features

• Thin, fast wrapper around the original dybuf implementation.
• Supports reading and writing unsigned integers (8–64 bits), booleans, raw byte payloads, and the library’s compact var_u64/var_s64 varints.
• Exposes typdex markers so higher-level protocols can tag fields without hauling in a full protobuf-style schema.
• Provides JSON-equivalent helpers for binary JSON round-tripping with shared key dictionaries.
• Compatible with Windows, Linux, and macOS thanks to compiled extension modules.
• Designed for publishing on PyPI with automated release workflows.

Installation

Once released to PyPI the package can be installed with:

pip install dybuf

Quick start

from dybuf import DyBuf

buf = (
    DyBuf(capacity=64)
    .append_uint16(0x1234)
    .append_uint32(0xDEADBEEF)
    .append_bool(True)
)

buf.flip()  # prepare for reading
print(hex(buf.next_uint16()))  # 0x1234
print(hex(buf.next_uint32()))  # 0xdeadbeef
print(buf.next_bool())         # True

write() and read() let you work directly with arbitrary byte payloads, while position, limit, and capacity expose the cursor-style API provided by the original library.

Variable-length integers

DyBuf supports zig-zag encoded signed integers and unsigned integers using the same compact varint scheme the C library exposes. These helpers keep the payload small for numbers that fit in a few bits.

from dybuf import DyBuf

buf = (
    DyBuf(capacity=32)
    .append_var_s64(-123)
    .append_var_u64(300)
)

buf.flip()

print(buf.next_var_s64())  # -123
print(buf.next_var_u64())  # 300

Use append_var_s64 / next_var_s64 when you need negative values; the encoding automatically performs zig-zag conversion under the hood. For values that are always non-negative, stick to append_var_u64 / next_var_u64 to avoid the extra zig-zag step. The legacy append_var_int / next_var_int and append_var_uint / next_var_uint aliases still work but emit DeprecationWarning so callers can migrate. The same encoding also powers append_var_bytes and append_var_string, which prefix their payload with a varint length field for round-tripping arbitrary byte sequences.

When you need compatibility with the C helpers that treat strings as NUL-terminated c-strings, call append_var_cstring / next_var_cstring. Those helpers append the trailing \0 byte on write (and strip it on read) so buffers line up with dyb_append_cstring_with_var_len / dyb_next_cstring_with_var_len.

Typdex markers

DyBuf also exposes helpers for working with the library's typdex encodings—a compact representation of a logical type paired with an index. These markers are commonly used by higher-level protocols such as dypkt to describe field layouts or function identifiers.

from dybuf import DyBuf, TYPDEX_TYP_INT, TYPDEX_TYP_STRING

buf = DyBuf(capacity=32)
buf.append_typdex(TYPDEX_TYP_INT, 3).append_typdex(TYPDEX_TYP_STRING, 0x42)

buf.flip()

assert buf.peek_typdex() == (TYPDEX_TYP_INT, 3)
assert buf.next_typdex() == (TYPDEX_TYP_INT, 3)
assert buf.next_typdex() == (TYPDEX_TYP_STRING, 0x42)

The constants TYPDEX_TYP_* mirror the underlying C enums. TYPDEX_TYP_OBJ (0x0e) marks protocol-defined objects; its index and payload belong to the application protocol and are not interpreted by dybuf. DYPE_F_* mirrors the reserved dypkt function indices (EOF, schema version, protocol name, protocol version). append_typdex validates that the type fits in 8 bits and the index can be represented by the packed encoding. next_typdex and peek_typdex raise EOFError for truncated markers and ValueError when encountering a malformed header.

Schema convention

When using typdex to design an app-level schema, treat one Typdex(type, index) plus its payload as a record. Keep common record IDs in 0..7 for 1-byte typdex markers, and start dypkt-compatible messages with TYPDEX_TYP_F + DYPE_F_VERSION.

See the repository's dypkt schema convention for the canonical typdex layout, reserved function IDs, and compatibility rules.

JSON helpers

encode_json and decode_json store JSON-compatible Python values using the JSON-equivalent dybuf convention documented in DYPKT_SCHEMA_CONVENTION.md. The first version targets round-trip equivalence rather than canonical bytes.

from dybuf import decode_json, encode_json

payload = {"items": [{"id": 1, "name": "alpha"}, {"id": 2, "name": "beta"}]}
encoded = encode_json(payload)

assert decode_json(encoded) == payload

Integers must fit JavaScript's safe integer range for Python/JavaScript parity. Fractional numbers are encoded as doubles. NaN and infinities are rejected because they are not valid JSON values.

Version notes

• 0.5.0 is a wire-format compatibility release. Fixed-width floating point payloads now follow the same big-endian order as fixed-width integers. Data written by older little-endian C builds through the float/double helpers may need migration before reading with 0.5.0.

Developing locally

Create a virtual environment and install the build requirements:

python -m venv .venv
source .venv/bin/activate
pip install -r requirements-dev.txt  # optional, see below

Build the extension in editable mode for local testing:

pip install -e .
pytest

The pytest suite loads the golden vectors generated from the canonical C library (fixtures/v1/*.json). If the fixtures are missing, run tools/generate_fixtures.sh from the repository root before executing the tests so fixtures/v1 exists adjacent to this package.

The project is configured to build wheels via python -m build, producing both source and binary distributions:

python -m build

Generate the Sphinx documentation locally with:

pip install -r requirements-dev.txt  # ensures sphinx/docutils are present
sphinx-build -b html docs docs/_build/html
open docs/_build/html/index.html  # or use your preferred viewer

Automated releases

A GitHub Actions workflow under .github/workflows/pypi-release.yml drives cibuildwheel to produce Windows, Linux, and macOS artifacts and publish them to PyPI. Provide a PYPI_API_TOKEN secret in your repository and tag releases with a semantic version (e.g. v0.4.1) to trigger the pipeline.

Licensing

The wrapper is distributed under the GNU GPL v2, matching the original dybuf project.