e-serde 0.6.0

Universal structured-data loader: native Rust codecs that decode any popular config format into native Python objects — and, when you ask, into frozen msgspec models. Two verbs, loads/dumps, sync and async twins, one wheel. json-module semantics, strict typing, no surprises.

Part of the Eager (e-) stack by damvolkov, built on open-source engines — the speed and robustness of C and Rust, for serialization in Python.

Install

uv add e-serde          # or: pip install e-serde

Usage

import eserde
import msgspec
from eserde import Format
from pathlib import Path

Twelve functions — load/dump whole documents or stream them record by record, in sync and async — with json semantics and keyword-only options. Files are named by Path or plain string path and inferred from the suffix (.json .jsonc .yaml .yml .toml .ini .cfg .conf .csv .tsv); format= accepts a Format member or its name ("json"); pure JSON content sniffs itself, and everything else must declare its format.

loads — decode

cfg = eserde.loads(b'{"host": "0.0.0.0", "port": 8080}', format=Format.JSON)
# {'host': '0.0.0.0', 'port': 8080}

class Server(msgspec.Struct, frozen=True):
    host: str
    port: int


srv = eserde.loads(src, format=Format.JSON, type=Server)  # Server(host='0.0.0.0', port=8080)
srv = eserde.loads(ini, format=Format.INI, type=Server, strict=False)  # "8080" → 8080
net = eserde.loads(src, type=Net, dec_hook=lambda t, v: t(v))  # custom fields inside type=

dumps — encode

raw = eserde.dumps({"name": "demian", "n": 42}, format=Format.YAML)
# b'name: demian\n"n": 42\n'

from fractions import Fraction

eserde.dumps({"f": Fraction(1, 2)}, format=Format.JSON, encoders={Fraction: str})  # b'{"f":"1/2"}'
eserde.dumps({"f": Fraction(1, 2)}, format=Format.JSON, default=float)  # b'{"f":0.5}'

Every input is normalized through the Jsonable encoder first (datetime → ISO, Enum → value, bytes → base64), so each format sees the same tree.

load / dump — files

cfg = eserde.load(Path("config.toml"))  # format from the .toml suffix

with open("app.json", "rb") as fh:
    data = eserde.load(fh, format=Format.JSON)  # an open handle needs the format

eserde.dump(cfg, Path("out.jsonc"))  # writes straight to disk → None

Same kwargs as loads / dumps.

async

aloads · adumps · aload · adump — same signatures; I/O and the GIL-free native parse run off the event loop. The Rust codecs release the GIL, so concurrent aloads parallelizes decode across cores.

async def main():
    srv = await eserde.aloads(Path("config.yaml"), type=Server)
    await eserde.adump(srv, Path("copy.json"))

compat — json drop-in

Frameworks duck-type the stdlib module (json_serialize=, renderers, formatters). eserde.compat speaks json.dumps/json.loads exactly, str output; behaviours e-serde cannot share faithfully delegate to the stdlib — slower, never a surprise.

from eserde import compat

compat.dumps({"a": 1}, ensure_ascii=False)  # '{"a":1}'   native compact utf-8
compat.dumps({"a": 1})  # byte-faithful to stdlib json
compat.loads('{"a": 1.5}', parse_float=Decimal)  # delegated to stdlib, never guessed

Errors are the LoaderError family: FormatError (no/unknown format), LoadError (decode or validation), DumpError / EncoderError (encode), CodecError (backend missing). The full guide lives at damvolkov.github.io/e-serde.

Formats and backends

The contract lives in code — eserde.STANDARDS — and test_standards.py executes every claim against the live codecs and the lockfiles. Bumping an engine or changing a format capability is a deliberate act, never silent drift. Full per-format pages (capabilities, limits, examples): docs → Formats.

Everything Rust is one extension module (eserde._native), compiled by maturin from crates/native. The only runtime dependency is msgspec.

Benchmarks

Median decode of a 100 KB config on CPython 3.14 (release build). Regenerate with make bench.

Because the Rust codecs release the GIL, aloads parallelizes decode: on 10 MB YAML/TOML the async fan-out is ~2× faster than serial sync (JSON stays flat — msgspec's C decoder holds the GIL). More charts in assets/benchmarks/: dumps · typed · async · memory.

Architecture

crates/native/          single Rust cdylib, one submodule per format
src/eserde/
  __init__.py           the one façade: import eserde; eserde.loads(...)
  infra/                contracts with zero internal deps: errors, formats, io, protocols
  backends/             one folder per engine: native/ (Rust), msgspec/ (C) + registry
  logic/                the facade functions: loads/dumps/load/dump/async + Jsonable encoder
tests/
  unit/eserde/          exact mirror of src
  benchmark/            rival matrix + report renderer
  resources/            canonical sample.* fixtures
docs/                   mkdocs-material site

Only the package root has an __init__.py; every subpackage is a namespace folder. Import boundaries are enforced by tach: eserde → logic → backends → infra/_native.

Design rules: decoding is always Rust/C, validation is msgspec's or pydantic's — type= only routes the decoded tree. Round-trip losses are explicit: JSONC comments are dropped on dumps; TOML has no null; YAML non-scalar keys and multi-document streams are rejected.

Development

uv sync                  # installs the dev group, builds the extension in place
make test                # pytest
make check               # ruff + format + ty + tach + pytest (what CI runs)
make bench               # rival benchmark matrix → assets/benchmarks/*.png

Roadmap

Interop landed: eserde is a custom encoder for any framework that ducks-types json (eserde.compat), validates pydantic/attrs models through type=, accepts per-call default=/encoders=/dec_hook= hooks, inlines Markdown bodies and structured sub-documents (embed=), and streams records one at a time (iloads/idumps and async twins). Shipped alongside a comparative concurrency stress harness (make stress).

• Pending consideration: YAML multi-document streaming (--- boundaries, reusing the event-stream scanner) — viable, but only pays for stream-shaped YAML corpora.
• Rejected by design: TOML and INI streaming — their semantics (scattered tables, duplicate-section merge) require the global view; the contract says so loudly.
• Next: broaden interop — msgspec/pydantic request-body and FastAPI response pipelines — and CI-gated regression against rival decoders under sustained load.

License

MIT — see LICENSE.