Skip to main content

Comserde

Comserde (compact serialization and deserialization) is a Python library for serializing data structures into compact binary data.

Installation

$ pip install comserde

Usage

Serializing dataclasses

Decorating a dataclass with @serializable will mark it as serializable and deserializable. Type annotations are used to infer the format of each field.

from comserde import serializable

# Add a decorator to make the dataclass serializable
# The order of decorators matters, the reverse won't work.
@serializable
@dataclass
class User:
  age: Optional[int]
  name: str

To produce and consume bytes from a serializable object, call dumps() and loads(). A major difference compared to other serialization algorithms (such as json) is that you must provide the type of object you want to decode, User in this example.

from comserde import dumps, loads

user = User(age=34, name="John Doe")

data = dumps(user)
user = loads(data, User)

More complex types can be defined using most of the typical type hints, such as generics and union types.

from typing import NewType

class User:
  id: UserId
  friends: list[Admin | User]
  images: dict[ImageId, tuple[str, Optional[Image]]]
  profile_image: bytes

To prevent a specific field from being serialized, assign it with Comserde's field() instead of dataclasses.field() and set serialize=False. Other field attributes can be set as usual.

from comserde import field

class User:
  age: Optional[int]
  name: str
  _cache: Optional[int] = field(default=None, serialize=False)

Serializing regular classes

Regular classes can also be serialized, although the types of each attribute must be explicitly provided, which is why dataclasses are preferred.

from comserde import serializable

 # Define serialization fields explicitly
@serializable({
  'age': Optional[int],
  'name': str
})
class User:
  def __init__(self, age: Optional[int], name: str):
    self.age = age
    self.name = name

    # Not serialized
    self._cache: Optional[int] = None

Customizing deserialization

If present, the __init_deserialize__ method is called upon deserialization of an object. This is similar to the __setstate__ method called when unpickling.

@serializable({
  'age': Optional[int],
  'name': str
})
class User:
  def __init_deserialize__(self, *, age: Optional[int], name: str):
    self.age = age
    self.name = name

Customizing serialization formats

The full serialization and deserialization process can be customized by adding the __serialize__ and __deserialize__ methods.

from typing import IO

class User:
  def __serialize__(self, file: IO[bytes]):
    file.write(...)

  @classmethod
  def __deserialize__(cls, file: IO[bytes]):
    return cls(file.read(...))

Specifying an explicit format

For every field in a class, a format can be explicitly selected by using the typing.Annotated type introduced by PEP 593. The use of this type doesn't affect the behavior of type checkers while providing additional information to Comserde.

from comserde import SerializationFormat, serializable

@serializable
@dataclass
class Vec2d:
  x: Annotated[float, SerializationFormat('f64')]
  y: Annotated[float, SerializationFormat('f64')]

The same syntax can be used on on regular classes, however the format can also be specified directly.

@serializable({
  'x': 'f64',
  'y': 'f64'
})
class Vec2d:
  ...

Serializing any object

Objects that are not marked with @serializable and that do not implement __serializable__() are pickled when serialized, and a warning is emitted if the the pickle format is not explicit. Keep in mind that pickling a large number of individual objects can significantly increase the output size due to the overhead of each pickled object.

Handling exceptions

Any error caused by corrupt data raises a DeserializationError which can be caught if the data is not trusted.

try:
  user = loads(data, User)
except DeserializationError:
  pass

Reference

Primitive formats

The following serialization formats are supported:

  • Booleans and scalars
    • bool
    • u8, u16, u32, u64
    • i8, i16, i32, i64
    • f32, f64
    • v8, v16, v32, v64 – unsigned variable-length quantity (VLQ) encoding of at least 1, 2, 4 and 8 bytes
    • w8, w16, w32, w64 – signed VLQ encoding
  • Bytes and strings
    • bytes – length-prefixed bytes
    • nt-bytes – null-terminated bytes
    • utf-8, utf-16
  • Others
    • json
    • object – stores the full qualified name of the object's class and uses the class as the serialization format; useful when the class is not known in advance, e.g. because it might be a subclass
    • pickle
    • void

Composite formats

  • deque[T]
  • NewType('X', T)
  • PurePath and all of its subclasses
  • bool
  • bytearray, bytes
  • complex – as two f64
  • dict[K, V]
  • EllipsisType, NoneType, Literal[T], tuple[()] – no data produced
  • float – as f64
  • frozenset
  • int – as w8
  • list[T]
  • Literal[T, S, etc.]
  • set[T]
  • str
  • T | S
  • tuple[T, ...]
  • tuple[T, S, etc.]

Metadata

Release files for comserde 0.2.0

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for comserde 0.2.0
File Size Uploaded Provenance Actions
comserde-0.2.0.tar.gz 11.2 kB Without provenance Details for comserde-0.2.0.tar.gz

Built distribution (wheel)

Table of built distributions (wheels) for comserde 0.2.0
File Interpreter ABI Platform Actions
comserde-0.2.0-py3-none-any.whl Python 3 none any Details for comserde-0.2.0-py3-none-any.whl

Total release size: 22.2 kB

Release files / comserde-0.2.0.tar.gz

Download URL comserde-0.2.0.tar.gz
Size 11.2 kB
Tags Source
SHA-256 checksum
How to use checksums
40b8538e6498a8c24f4f55312f99d3fa3ce93fd68a09aea68b3ae429ddcf657b
BLAKE2b-256 checksum
How to use checksums
630365eb9adb15a4dd5d287b676ddfc4a72536f265d37f03a5c1b13153343a53
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via twine/4.0.1 CPython/3.10.6

Release files / comserde-0.2.0-py3-none-any.whl

Download URL comserde-0.2.0-py3-none-any.whl
Size 11.1 kB
Tags Python 3
SHA-256 checksum
How to use checksums
fd4bfbaf4c0af246cdf1e2fc4eee6964dcf8b91aa8d2eaba3c85e48c9bf090a3
BLAKE2b-256 checksum
How to use checksums
ee3a33274e38dea3f962626a5940ce9c4672c12431639a2d6fee63f6369bedd8
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via twine/4.0.1 CPython/3.10.6

Security for comserde 0.2.0

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

No provenance found

None of the files in this release were uploaded with provenance metadata.

A note on package security

PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.

To learn more about protecting yourself from malicious actors, visit our package security documentation.

Release history Release notifications | RSS feed

This release

0.2.0 This release

2 release files No provenance

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page