SofaBuffers

Structured Objects For Anyone
... so optimized, feels amazing.

Would you like to know more?

SofaBuffers Python library

GitHub repository

A streaming, dependency-free implementation of the SofaBuffers (Sofab) serialization format — a compact, TLV-like binary format. It is the runtime stream core, meant to be driven by generated code: a schema-driven generator emits one class per message with the streaming serialize / deserialize pair and the one-shot encode / decode wrappers over it, all of which call the Encoder / Decoder primitives here — the same way protobuf's generated code calls its runtime.

The public API is one pair of classes with two interchangeable engines selected at import: the hot path (varint / zigzag / buffer management) ships as an optional compiled native accelerator (Cython → C, sofab._speedups) loaded automatically when present, with a pure-Python fallback used when it is not. The two are byte-for-byte interchangeable, so the library runs anywhere CPython runs, with or without a C compiler.

Requirements

Python 3.9 or newer (CPython or PyPy); CI runs 3.9–3.14. The optional native accelerator additionally needs a C compiler and Cython, both build-time only.

Dependencies

None at runtime — the pure-Python path uses only the standard library (struct, io). The one third-party build dependency is Cython (PEP 517), used to compile the accelerator and never imported at runtime.

Packaging

Distribution sofa-buffers-corelib on PyPI; import package sofab.

pip install sofa-buffers-corelib

import sofab   # Encoder, Decoder, Visitor, wire-format types and limits
print(sofab.__version__)   # release of this runtime

sofab.__version__ is the release version of the runtime and is the only place it is written down: pyproject.toml declares the distribution version dynamic and reads it from there, so an install and an import always report the same string.

Why this design

Usage

The codec has four use cases — serialize a message that fits in one buffer, serialize one too large for the buffer (streamed out in chunks), deserialize a whole message, and deserialize one arriving in chunks — plus the generated-code path that wraps them.

Serialize

Write fields into an Encoder and take the finished bytes:

from sofab import Encoder

enc = Encoder()
enc.write_unsigned(1, 42)
enc.write_signed(2, -7)
enc.write_string(3, "hi")
data = enc.getvalue()

Encoder() writes into a fixed 1 KiB scratch buffer and appends each bufferful to the result it hands back — it never grows a buffer mid-message (see Memory handling). Pass a writer (anything with write(bytes)) and the same bufferfuls go there instead, as they are produced:

with open("msg.sofab", "wb") as fh:
    enc = Encoder(fh)                 # streams out; nothing accumulates
    enc.write_unsigned(1, 42)
    enc.flush()                       # push the tail

Serialize stream

Encoder.over_buffer is the same mechanism over a buffer you supply: it writes in place and calls a flush sink whenever the buffer fills, so an arbitrarily large message streams out through however much memory you chose to give it:

from sofab import Encoder

out = bytearray()                                            # or a socket / file write
enc = Encoder.over_buffer(bytearray(16), offset=0, flush=out.extend)  # tiny buffer
for i in range(1_000_000):
    enc.write_unsigned(i % 128, i)
enc.flush()                                                  # push the tail

With no sink the buffer is all the encoder gets: it holds the message or reports SofaBufferError. That is the shape generated code uses when the schema bounds the message — allocate MAX_SIZE, encode in one pass, no flush possible:

buf = bytearray(Point.MAX_SIZE)
enc = Encoder.over_buffer(buf, offset=0)
point.serialize(enc)
wire = memoryview(buf)[: enc.bytes_used()]   # no copy

Deserialize

Decoder is a pull parser: next() returns one field header at a time; read the value with a typed accessor, or skip() an unknown field:

import io
from sofab import Decoder

dec = Decoder(io.BytesIO(data))
while (field := dec.next()) is not None:   # None == clean EOF
    if   field.id == 1: v = dec.unsigned()
    elif field.id == 2: v = dec.signed()
    elif field.id == 3: s = dec.string()
    else:               dec.skip()         # unknown field

A read whose type contradicts the field on the wire is not an error (MESSAGE_SPEC §7.3): it returns None and consumes nothing, so the field is skipped by the following next() exactly like a field with an unknown id and the decode stays complete — dec.float64() on a string field yields None, not an exception. Test field.type / field.subtype before reading (this is what generated code does) or treat None as "not my field"; because nothing was consumed, re-reading the same field with the type the wire carries still works. A read issued when there is no pending value at all — before the first next(), twice for one field, or on a sequence start/end — is a caller mistake and raises SofaRangeError (§6.3 InvalidArgument, the only code that taxonomy has for one). There is no separate "API misuse" class: §6.3 has no code for one, and a wrong-type read is not an error to catch in the first place.

Deserialize stream

Hand Decoder any object with read(n) (a socket, sys.stdin.buffer, gzip.GzipFile, …) and pull fields with next() as they arrive. It refills on demand, so the same loop decodes correctly even when fed one byte at a time, wherever the bytes come from:

from sofab import Decoder

dec = Decoder(reader)                # any read(n) source: file, socket, pipe
while (field := dec.next()) is not None:
    ...                              # pull each field, or dec.skip()

When the bytes have not all arrived yet. A reader that can return b"" before end-of-message — a non-blocking socket, a queue fed by another task — puts the decoder in the position CORELIB_PLAN §5.2 calls INCOMPLETE: the bytes stop inside a field. That is not an error and not the end of the message; it means "feed me more". Two shapes signal it, and both are resumable — the suspended call consumed nothing, so the answer to either is to obtain more bytes and issue the same call again:

• next() returns None — the bytes stopped exactly between fields (§5.2 COMPLETE: a message may end here, and more fields may also still follow);
• SofaIncompleteError is raised — the bytes stopped inside a field header or payload, or inside a sequence that is still open.

while True:
    try:
        field = dec.next()
    except SofaIncompleteError:
        feed_more(); continue        # partial field retained; re-issue next()
    if field is None:
        if stream_ended: break       # your framing decides; the decoder never does
        feed_more(); continue
    value = read_the_value(dec, field)   # same retry rule for the typed reads

Whether an incomplete message is acceptable is the caller's decision, not the decoder's: only your framing (a length prefix, a datagram boundary, EOF) knows whether more bytes can still come.

Code generator

The most common real use is driving the library through generated code: sofabgen --lang python emits a @dataclass per message with exactly four methods (CORELIB_PLAN §6.1.1 fixes the names) — the streaming pair serialize / deserialize, which talks to the primitives above, and the one-shot pair encode / decode, which are thin wrappers over it. A hand-written stand-in:

import io
from dataclasses import dataclass
from sofab import Decoder, Encoder

# generated by: sofabgen --lang python
@dataclass
class Point:
    x: int = 0
    y: int = 0

    def serialize(self, e: Encoder) -> None:    # streaming out: write into any encoder
        e.write_signed(1, self.x)
        e.write_signed(2, self.y)

    def deserialize(self, d: Decoder) -> None:  # streaming in: pull from any decoder
        while (f := d.next()) is not None:
            if   f.id == 1: self.x = d.signed()
            elif f.id == 2: self.y = d.signed()
            else:           d.skip()            # tolerate unknown fields

    def encode(self) -> bytes:                  # one-shot wrapper over serialize()
        e = Encoder()
        self.serialize(e)
        return e.getvalue()

    @classmethod
    def decode(cls, data: bytes) -> "Point":    # one-shot wrapper over deserialize()
        o = cls()
        o.deserialize(Decoder(io.BytesIO(data)))
        return o

wire = Point(x=3, y=4).encode()
got = Point.decode(wire)             # got.x == 3, got.y == 4

The one-shot pair holds the whole message in memory; serialize / deserialize are the same code without that requirement. Out, serialize writes into an encoder over a buffer you sized, draining to a sink as it fills; in, deserialize pulls from a Decoder over any read(n) source, which is this port's chunk-fed reader — the corelib Decoder is the object §6.1.1 calls decoder(), so there is no second handle to obtain:

# streaming out: a 2-byte buffer, drained to the sink as the message is written
packets = bytearray()                        # or a socket / file write
enc = Encoder.over_buffer(bytearray(2), offset=0, flush=packets.extend)
Point(x=3, y=4).serialize(enc)
enc.flush()                                  # push the tail
streamed = bytes(packets)                    # == wire, out of a buffer half its size

class ChunkReader:                           # stand-in for a socket / pipe
    def __init__(self, data): self.data, self.pos = data, 0
    def read(self, n):                       # hands over one byte per call
        chunk = self.data[self.pos : self.pos + 1]
        self.pos += len(chunk)
        return chunk

# streaming in: the same object, fed one byte at a time
got_streamed = Point()
got_streamed.deserialize(Decoder(ChunkReader(streamed)))

A reader that can run dry before the message ends adds one obligation to the generated loop: retry a SofaIncompleteError after obtaining more bytes, as Deserialize stream describes — the suspended call consumed nothing, so re-issuing it is always correct.

Decode limits

Array counts and string/blob lengths are optional on the wire, so by default the decoder allocates whatever a message declares. Untrusted input can abuse that, so Decoder takes optional receiver-side caps that reject an oversize field on its count/length word alone — before any allocation or payload buffering:

dec = Decoder(reader, max_array_count=65536, max_string_len=1 << 20, max_blob_len=1 << 20)

A field whose declared count/length exceeds its cap raises SofaLimitError. That is a policy rejection, distinct from malformed input: it is a sibling of SofaDecodeError under SofaError, not a subclass, so except SofaDecodeError does not catch it. Each limit defaults to None (no cap); the values are meant to be supplied by generated code, not guessed by the runtime. Independent of any limit, the decoder never pre-allocates from an untrusted array count — a truncated oversize claim fails promptly as SofaIncompleteError rather than attempting a huge allocation.

The verdict is reached on the count/length word alone, inside next(), before a single payload byte is read or buffered — the point CORELIB_PLAN §6.2.1 requires it to be decided. It is raised by the call that would consume the field: a typed read, skip(), or the auto-skip the following next() performs. Nothing is read or allocated in between, and the field cannot be got at any other way, so the protection is the same; what the gap buys is the window in which the caller can say the field is not one of the cap's business.

A schema-bounded field is exempt: schema_bounded()

A cap is capacity the deployment commits where the sender chooses the size. Where the schema already states a count:/maxlen:, that bound governs instead and an over-bound value is malformed input, so §6.2.1 forbids the cap there ("MUST NOT be applied to a field the schema already bounds") and §6.3 forbids SofaLimitError on such a field. Only the schema knows which fields those are, so the caller declares them per field:

f = dec.next()
if f.id == 1:                 # `name: { type: string, maxlen: 4194304 }`
    dec.schema_bounded()      # the cap does not bind this field
    if dec.fixlen_len() > 4194304:
        raise SofaDecodeError("name: string byte length above schema maxlen")
    o.name = dec.string()

The declaration covers the current field only — the next next() starts an undeclared, and therefore capped, field again — and it is a no-op on a field no cap has rejected, so generated code emits it unconditionally on the fields its schema bounds. Declaring is a promise to enforce: with the cap off, nothing else stands between an untrusted length word and the allocation it implies, so the caller must reject an over-bound count/length itself, as SofaDecodeError (MESSAGE_SPEC §7.1). fixlen_len() is the peek for that — it consumes nothing and answers whether or not a cap has spoken on the field, so the schema bound can be decided in either order. A Visitor driven by drive() can call schema_bounded() from on_field, which is reached before the typed read.

A schema bound is the opposite kind of thing from a cap: it is part of the message definition, so breaching it is malformed input, not policy. The integer-array reads take the declared element width for exactly that reason — read_unsigned_array(255) for a u8 array, read_signed_array(-128, 127) for an i8 one (either half may be given alone; the other side stays open). An element outside the declared width raises SofaDecodeError the moment its own bytes are decoded, so the verdict never depends on how much of the array followed it or on which engine read it (MESSAGE_SPEC §7.1). Omit the argument for u64/i64, whose range is the value domain, or for an unbounded consumer.

Memory handling

The key point for Python: the library allocates results for you — the caller never provides a value buffer.

• Decode: a suspended call keeps its bytes, and only its bytes. Everything the reader hands over is retained, so a field split across chunks is never half-consumed; the buffer's consumed prefix is dropped on the next refill, down to the first byte of the call in flight — that byte is the one a resumed call re-reads from. The window held is therefore one field (for a skip() over a sequence, one sequence), not one message.

• Decode. Decoder keeps a single internal buffer, refilled from the read(n) source and never handed out, so there is no zero-copy aliasing: string() returns a fresh str, bytes() independent bytes, scalars a fresh int/float, and arrays a new list — every result stays valid after the decoder advances. fixlen_len() peeks the current string/blob field's exact wire byte length without consuming it, so a caller can bound the field against a schema maxlen before reading — no re-encoding a decoded str just to measure it.

• Decode: a value you don't want costs nothing to get rid of. skip() — and the auto-skip next() performs over an unconsumed value — walks a string, blob or fixlen-array payload by advancing the cursor, so nothing is allocated for bytes that are being discarded (CORELIB_PLAN §5.2: a skip consumes). Skipping a 1 MiB blob already in the buffer is a pointer bump, not a 1 MiB copy. The bytes are still buffered when the payload straddles a refill — a suspended skip has to be replayable from its first byte, and skip() over a sequence replays the whole walk — so what a skip saves is the copy, not the window: the memory held is still one field (one sequence for a sequence skip).

• Encode: one ownership model — the output buffer is fixed, and never grows. CORELIB_PLAN §5.1 forbids a corelib to allocate an output buffer or to grow one, so there is a single mechanism here with three ways to reach it, not two competing models:

  • Encoder.over_buffer(buf, offset, flush) is the primitive and the only caller-supplied form: it writes in place through a memoryview, drains to the sink when full and reuses the buffer — or, without a sink, holds the message or reports SofaBufferError. That is the shape generated code uses for a schema whose MAX_SIZE bounds the message.
  • Encoder(writer) installs a 1 KiB scratch buffer with a sink that forwards each bufferful to writer.write — §5.1's "unbounded schema" shape. A 100 MB message costs 1 KiB of encoder memory, and the bytes leave while the message is written, not at flush(). Nothing is retained, so getvalue() raises SofaRangeError: returning the undrained tail would be partial output dressed up as a whole message.
  • Encoder() is the same scratch buffer with the sink appending into the result — the message getvalue() hands back, joined from the drained chunks (a message that fits in the scratch is never chunked at all, and a string/blob run longer than the buffer becomes one chunk rather than being copied through it). What grows is the message being returned, not a buffer being written into: bytes_used() never exceeds 1 KiB.

  The scratch is one allocation per encoder, made at construction and never resized. §5.1 puts even that in the generated layer, which knows the schema — a caller who wants zero library allocation supplies the buffer with over_buffer.

• MIN_OUTPUT_BUFFER is 1, and it applies to a buffer installed with a sink. sofab.MIN_OUTPUT_BUFFER is the smallest output buffer this port accepts for streaming: one byte, because the encoder splits every atomic unit — a header varint, a fixlen_word, an element count, a scalar, one float — at any byte boundary, so a one-byte scratch buffer already yields exactly the one-shot bytes. Encoder.over_buffer(buf, offset, flush) and every mid-stream buffer_set(buf, offset) that carries a flush sink require len(buf) - offset >= MIN_OUTPUT_BUFFER and raise SofaRangeError right there — where the buffer is handed over, never partway through a message. A buffer installed without a sink is subject to no minimum: no flush can occur, so the buffer simply holds the message or reports SofaBufferError, and sizing it from a generated MAX_SIZE stays exact — a message that encodes to two bytes encodes into a two-byte bytearray. There is no pass-through: a string/blob run is copied into the output buffer like any other output, and every flush hands the sink a bytes snapshot of that buffer's prefix — a sink may retain what it receives without pinning caller memory.

• The start offset belongs to the installation, not to the buffer. A flush sink states what it did by what it does before returning. Returning without installing anything means it copied the bytes it was handed: the same buffer stays active and encoding resumes at offset 0. A sink that takes the buffer — queues it for an async write, hands it to a transport — must install a replacement with buffer_set(buf, offset) before it returns, and that call's offset is where encoding resumes. Re-installing is therefore how a sink gets fresh framing-header room in every flushed packet (one header per packet), including when it passes the same buffer back: a bare return would reserve nothing, since the offset is consumed by the installation that carried it.

• Sequence framing is lazy. write_sequence_begin_lazy(id) pushes the id onto a pending run and writes nothing; the first field write inside commits the whole run, outermost header first. write_sequence_end() then drops a sequence that never got content — header and end marker — which is exactly MESSAGE_SPEC §2's "omit a sequence-typed field equal to its declared default", since generated code already omits every child equal to its default. Close with write_sequence_end_keep() wherever the frame carries information regardless of its contents: a wrapper-array element (element presence is what carries a dynamic array's length, §5.1) or an array field that must encode as explicitly empty against a non-empty declared default. The two mistakes are not symmetric — end_keep where end would do costs one non-canonical empty frame every decoder normalizes away, while the reverse changes a decoded array's length — so end_keep is the safe choice when a call site is ambiguous. The pending run grows on demand, so the hold-back reaches the full MAX_DEPTH and every nesting depth is canonical, and it is allocated on the first hold-back — an encoder that never opens a sequence never pays for it. The pending ids are encoder state, never buffer content, so a flush cannot split a run by construction: a held-back header takes no buffer space, and the buffer only fills through a write, which commits the run before its first byte. A tiny output buffer therefore yields exactly the one-shot bytes.

Native accelerator

Encoder / Decoder / Field are re-exported from the compiled sofab._speedups extension when present, and from the pure-Python encoder.py / decoder.py otherwise. The native core is a small Cython implementation of the same algorithm — one contiguous buffer, an advancing cursor, bulk memcpy, and varint/zigzag compiled to C. Both engines import wire constants, enums and exception classes from the shared types.py, so a SofaRangeError is the same class from either, and the two produce byte-for-byte identical output (enforced by tests/test_native_parity.py).

The active engine is reported by sofab.IMPL ("native" or "python"):

import sofab
print(sofab.IMPL)        # "native" when the compiled extension is loaded

Force the pure-Python path with SOFAB_PUREPYTHON=1.

Released wheels ship the accelerator already compiled, so a plain pip install sofa-buffers-corelib gets the native engine without a toolchain on every platform that has a wheel (CPython 3.9–3.14 on Linux — glibc and musl, x86-64 and ARM64 — macOS Intel and Apple Silicon, and Windows x86-64). Anywhere else pip builds the sdist, which compiles the accelerator if a C compiler is present and installs the pure-Python engine if not.

Feature flags

The package always builds the full format (unsigned / signed varints, fp32 / fp64, strings, blobs, arrays and nested sequences). The one build toggle is SOFAB_DISABLE_NATIVE=1, which builds a native-free (pure-Python) distribution; it changes only speed, never the wire format or the public API.

Build & test

python -m venv .venv && . .venv/bin/activate
pip install -e . pytest ruff mypy   # compiles the native accelerator if a C compiler is present
pytest                       # vectors + roundtrip + streaming + malformed + native↔pure parity
ruff check src/sofab tests   # lint
mypy --strict src/sofab      # type-check

If the compile fails or no compiler is available, the install falls back to pure-Python (the extension is marked optional in setup.py). To exercise both engines:

pytest                       # whichever engine is active (native if built)
SOFAB_PUREPYTHON=1 pytest    # force the pure-Python engine

That optionality has a sharp edge: a failed compile removes every native test (they are gated on importing sofab._speedups) instead of failing one, so a run can stay green with the accelerator — and the native↔pure parity tests — gone. SOFAB_REQUIRE_ENGINE=native|python closes it, by making the run assert that sofab.IMPL is the engine it claims to be exercising:

SOFAB_REQUIRE_ENGINE=native pytest                    # fails if the accelerator is missing
SOFAB_PUREPYTHON=1 SOFAB_REQUIRE_ENGINE=python pytest  # the fallback engine, pinned

CI runs the full suite twice on every supported Python — once per engine, each pinned that way — and adds a leg installed with SOFAB_DISABLE_NATIVE=1 that proves the compiler-less install still passes.

Benchmarks

bench/perfbench.py implements the three tools BENCH_SPEC requires — the same workloads, on the same data, measured the same way and printed in the same grammar as the C/C++/Rust/Go/… ports, so the numbers are directly comparable across languages. bench/compare_protobuf.py is extra, and language-native: it compares the native accelerator, the pure-Python fallback, and (for a yardstick) protobuf's Python runtime (upb C backend), with full materialization on both sides so it is apples-to-apples with the SofaBuffers pull API:

python bench/perfbench.py bench           # throughput on this machine, MB/s (MB = 1e6)
python bench/perfbench.py perf            # per-op cost for the shared 12-field message
bash  bench/run_callgrind.sh              # instructions/op (Callgrind) — clock-independent
pip install protobuf                      # optional; the column is dropped if absent
python bench/compare_protobuf.py          # best-of-5 MB/s table

bench / perf measure this machine and move with its load; run_callgrind.sh counts instructions retired, which is deterministic and comparable across hosts, so it is the one to trust when judging a change to the library itself. (time still works as a synonym for bench.)

The workloads are BENCH_SPEC's, not this port's invention:

The three blob 1MB rows are read against each other, never next to typical message: five of its bytes are metadata and a million are payload, so the absolute figure is this machine's memory bandwidth. The signal is the gap between them — one-shot is one contiguous write into a 1,000,005-byte caller buffer with no sink; streaming is the same bytes through a 4096-byte caller buffer with a flush sink, i.e. ~245 flushes of the divisible-run path (CORELIB_PLAN §5.1); decode is fed in 4096-byte chunks. This port grants no pass-through, so BENCH_SPEC's optional blob 1MB passthrough row is absent rather than stubbed.

BENCH_SPEC says to read that pair as Ir/op, and on x86-64 CPython there is a caveat: the one-shot row is a single 1,000,000-byte memcpy, which glibc serves from its ERMS (rep movsb) path and Valgrind counts at ~1 instruction per byte, while the streaming row's 4096-byte copies take the vectorised path at a fraction of that. Ir/op consequently reports one-shot as the dearer of the two (≈1.07M vs ≈0.41M) although it does strictly less work — measured, not assumed: a bare 1 MB memcpy costs ≈967k Ir under Callgrind on this host. For these two rows read the MB/s above (one-shot ≈2.7× streaming, which is the real cost of the flush machinery); every other row is Ir/op's to tell.

Representative result (throughput MB/s, higher is better; one x86-64 host, CPython 3.14 — the ratios are the point, not the absolute numbers):

Two readings worth pulling out. The blob one-shot row is memcpy in both engines and the two are nearly level — there is no per-field Python work left to remove there; the streaming row costs the native engine 2.7× the one-shot and the pure engine 9.3×, and that gap is the flush machinery. And decode: composite skip-all is ≈4.9× decode: composite on the native engine: what a router or filter saves by walking a message without materializing it.

Against protobuf (a separate measurement — bench/compare_protobuf.py, best of 5, one x86-64 host, CPython 3.12; read this table internally, not across into the one above):

The native accelerator is an order of magnitude faster than the pure-Python fallback on small mixed messages and ~85–135× on array-heavy ones, and beats protobuf everywhere except the smallest decode, where the two are level. (The per-field gap has widened since this table was taken — see the row set above, measured on current main.) That last workload is where the streaming pull API costs the most: it crosses the Python↔C boundary twice per field (next() then a typed read), whereas protobuf parses the whole message in one C call — an inherent pull-vs-parse-tree trade-off that only shows on very small messages, and the price of never having to hold one in memory.