datacodec 1.0.0

Binary codec library based upon muesli compile-time codecs

datacodec

A Python binary codec library based upon muesli compile-time codecs.

Installation

pip install datacodec

For NumPy support:

pip install datacodec[numpy]

Quick Start

The easiest way to use datacodec is with automatic codec generation from dataclasses:

from datacodec import dataclass_codec, BinaryFormat
from dataclasses import dataclass
import io

@dataclass
class Vec3:
    x: float  # Automatically uses double_codec
    y: float
    z: float

# Codec is automatically generated from type hints!
vec3_codec = dataclass_codec(Vec3)
format = BinaryFormat(vec3_codec)

# Serialize
output = io.BytesIO()
format.serialize(Vec3(1.0, 2.0, 3.0), output)

# Deserialize
output.seek(0)
result = format.deserialize(output)
print(result)  # Vec3(x=1.0, y=2.0, z=3.0)

Features

• Automatic codec generation from dataclasses with type inference
• Python-native types: int -> varint, float -> double, str -> string
• NumPy support: Automatic detection of NumPy types (optional)
• Fluent API: Chain codec operations with .transform(), .constrain(), .optional()
• Cross-platform: Generate binary data for ARM32, ARM64, x86-32, x86-64 targets
• Type-safe: Full Python typing support
• Based on muesli: Binary compatible with muesli C++ library

Python Type Mappings

When using dataclass_codec(), Python types automatically map to appropriate codecs:

Python Type	Codec	Reason
int	signed_varint_codec	Python ints are arbitrary precision
float	double_codec	Python floats are double precision
bool	bool_codec	Boolean type
str	string_codec	UTF-8 strings

NumPy Types (with pip install datacodec[numpy])

NumPy Type	Codec
np.int8/16/32/64	int8/16/32/64_codec
np.uint8/16/32/64	uint8/16/32/64_codec
np.float32	float_codec
np.float64	double_codec

Binary Format Details

String Encoding

Strings use null-termination (muesli compatible):

• UTF-8 encoded string data
• Null byte (0x00) terminator

# "hello" serialises as:
# [68 65 6c 6c 6f 00]
#  "hello" UTF-8  null

Integer Encoding

• Fixed-size integers: Use native byte order and alignment
• Varint: Variable-length encoding (1-10 bytes depending on value)
  • Unsigned: Direct varint encoding
  • Signed: Zigzag encoding (maps negatives to positives efficiently)

Alignment

Platform-specific alignment is maintained:

• ARM32: 8-byte alignment for int64/double
• ARM64: 8-byte alignment for int64/double
• x86-64: Natural alignment per type

Examples

Automatic Codec Generation

from datacodec import dataclass_codec
from dataclasses import dataclass

@dataclass
class Person:
    name: str      # -> string_codec
    age: int       # -> signed_varint_codec
    height: float  # -> double_codec

# Codec automatically generated!
person_codec = dataclass_codec(Person)

Explicit Fixed-Size Types

Use the @codec decorator when you need specific fixed-size types:

from datacodec import dataclass_codec, codec, uint8_codec, uint32_codec

@dataclass
class Message:
    msg_type: int = codec(uint8_codec)      # Force uint8
    timestamp: int = codec(uint32_codec)    # Force uint32
    payload_size: int                       # Uses varint (default)

protocol_codec = dataclass_codec(Message)

Manual Codec Construction

For full control, build codecs manually:

from datacodec import tuple_codec, float_codec, BinaryFormat

@dataclass
class Vec3:
    x: float
    y: float
    z: float

# Manually specify each field's codec
vec3_codec = tuple_codec(float_codec, float_codec, float_codec).apply(Vec3)

Dict Codec

Python dicts work like std::map in muesli - no special codec needed, just composition:

from datacodec import dict_codec, string_codec, int32_codec

# Dict[str, int]
my_dict_codec = dict_codec(string_codec, int32_codec)
data = {"a": 1, "b": 2, "c": 3}

Cross-Platform

Generate binary data for different architectures:

from datacodec import BinaryFormat, uint32_codec

# Generate data for ARM32 target
format_arm32 = BinaryFormat(uint32_codec, platform='arm32')

# Binary is compatible with muesli C++ on ARM32

Fluent API

Chain transformations and constraints:

from datacodec import int32_codec

validated = int32_codec \
    .constrain(lambda x: x > 0) \
    .transform(lambda x: x * 2, lambda x: x // 2) \
    .optional()

Advanced Usage

NumPy Arrays

import numpy as np
from datacodec import numpy_array_codec

# Fixed-size 3D vector
vec3 = numpy_array_codec(np.float32, shape=(3,))

# Dynamic-size array
float_array = numpy_array_codec(np.float32)

NumPy in Dataclasses

import numpy as np
from datacodec import dataclass_codec

@dataclass
class Point:
    x: np.float32  # Automatically uses float_codec
    y: np.float32
    z: np.float32

point_codec = dataclass_codec(Point)

License

BSD 3-Clause License