Skip to main content

CI PyPI Python License

e-serde

Universal structured-data loader: native Rust codecs that decode any popular config format into native Python objects — and, when you ask, into frozen msgspec models. Two verbs, loads/dumps, sync and async twins, one wheel. json-module semantics, strict typing, no surprises.

Part of the EDamLabs eager series: EDamLabs' own e- products (EDamLabs families).

Install

uv add e-serde          # or: pip install e-serde

Usage

import eserde
import msgspec
from eserde import Format
from pathlib import Path

Twelve functions — load/dump whole documents or stream them record by record, in sync and async — with json semantics and keyword-only options. Files are named by Path or plain string path and inferred from the suffix (.json .jsonc .yaml .yml .toml .ini .cfg .conf .csv .tsv); format= accepts a Format member or its name ("json"); pure JSON content sniffs itself, and everything else must declare its format.

Function Input Output
loads bytes | str | Path plain tree — or the model in type=
dumps any object bytes
load str | Path | BinaryIO like loads
dump object → file None
aloads · adumps · aload · adump same awaitables of the same
iloads · idumps records, streamed Iterator of objects / bytes chunks
ailoads · aidumps same AsyncIterator of the same

loads — decode

cfg = eserde.loads(b'{"host": "0.0.0.0", "port": 8080}', format=Format.JSON)
# {'host': '0.0.0.0', 'port': 8080}
class Server(msgspec.Struct, frozen=True):
    host: str
    port: int


srv = eserde.loads(src, format=Format.JSON, type=Server)  # Server(host='0.0.0.0', port=8080)
srv = eserde.loads(ini, format=Format.INI, type=Server, strict=False)  # "8080" → 8080
net = eserde.loads(src, type=Net, dec_hook=lambda t, v: t(v))  # custom fields inside type=
kwarg effect
format= a Format member or its name; inferred from a path, sniffed from pure JSON, required otherwise
type= validate into a Struct, dataclass, TypedDict, attrs or pydantic model; violations raise LoadError
strict=False msgspec coercion — the escape hatch INI needs
object_hook= rewrite every decoded mapping, innermost first (json semantics)
dec_hook= teach type= custom field types (requires type=)
registry= swap the default codec set
embed= inline source:-style references — .md as text, JSON/YAML/TOML/INI/CSV files as decoded trees, recursive, root-confined

dumps — encode

raw = eserde.dumps({"name": "demian", "n": 42}, format=Format.YAML)
# b'name: demian\n"n": 42\n'
from fractions import Fraction

eserde.dumps({"f": Fraction(1, 2)}, format=Format.JSON, encoders={Fraction: str})  # b'{"f":"1/2"}'
eserde.dumps({"f": Fraction(1, 2)}, format=Format.JSON, default=float)  # b'{"f":0.5}'

Every input is normalized through the Jsonable encoder first (datetime → ISO, Enum → value, bytes → base64), so each format sees the same tree.

kwarg effect
format= defaults to Format.JSON
encoders= exact-type hooks, ahead of every built-in; results are re-walked
default= json/orjson-style last resort for unknown types; none → EncoderError
registry= swap the default codec set

load / dump — files

cfg = eserde.load(Path("config.toml"))  # format from the .toml suffix

with open("app.json", "rb") as fh:
    data = eserde.load(fh, format=Format.JSON)  # an open handle needs the format

eserde.dump(cfg, Path("out.jsonc"))  # writes straight to disk → None

Same kwargs as loads / dumps.

async

aloads · adumps · aload · adump — same signatures; I/O and the GIL-free native parse run off the event loop. The Rust codecs release the GIL, so concurrent aloads parallelizes decode across cores.

async def main():
    srv = await eserde.aloads(Path("config.yaml"), type=Server)
    await eserde.adump(srv, Path("copy.json"))

compat — json drop-in

Frameworks duck-type the stdlib module (json_serialize=, renderers, formatters). eserde.compat speaks json.dumps/json.loads exactly, str output; behaviours e-serde cannot share faithfully delegate to the stdlib — slower, never a surprise.

from eserde import compat

compat.dumps({"a": 1}, ensure_ascii=False)  # '{"a":1}'   native compact utf-8
compat.dumps({"a": 1})  # byte-faithful to stdlib json
compat.loads('{"a": 1.5}', parse_float=Decimal)  # delegated to stdlib, never guessed

Errors are the LoaderError family: FormatError (no/unknown format), LoadError (decode or validation), DumpError / EncoderError (encode), CodecError (backend missing). The full guide lives at edamlabs.github.io/e-serde.

Formats and backends

Format Extension Spec Engine (pinned) Notable
JSON .json RFC 8259 msgspec.json 0.21 exact big ints, strict tokens
JSONC .jsonc Deno jsonc jsonc-parser 0.33 comments, trailing commas
YAML .yaml .yml YAML 1.2 core + merge keys saphyr 0.0.12 anchors, <<, bomb-guarded
TOML .toml TOML v1.1 toml-rs 1.1 inf/nan, no null, i64 ints
INI .ini .cfg .conf de-facto rust-ini 0.21 strings; merge on strict=False
CSV .csv RFC 4180 csv 1.4 list[dict], polars-style inference
TSV .tsv RFC 4180 (tab) csv 1.4 same codec, tab-delimited

The contract lives in code — eserde.STANDARDS — and test_standards.py executes every claim against the live codecs and the lockfiles. Bumping an engine or changing a format capability is a deliberate act, never silent drift. Full per-format pages (capabilities, limits, examples): docs → Formats.

Everything Rust is one extension module (eserde._native), compiled by maturin from crates/native. The only runtime dependency is msgspec.

Benchmarks

Median decode of a 100 KB config on CPython 3.14 (release build). Regenerate with make bench.

loads

Format e-serde nearest rival margin
JSON 0.17 ms orjson 0.17 ms ≈ tie — same decoder (msgspec)
YAML 2.20 ms pyyaml C 12–16 ms ≈6× — ruamel 66×
TOML 1.53 ms rtoml 2.3–2.5 ms 1.5× — tomlkit 42×
JSONC 0.79 ms pyjson5 0.39 ms the one format behind (×0.5)
INI 2.00 ms configparser 22 ms 11×
CSV 0.9–1.0 ms polars 1.3–1.7 ms ×1.3–1.8 — and polars never releases the GIL; stdlib csv is comparable and untyped

Because the Rust codecs release the GIL, aloads parallelizes decode: on 10 MB YAML/TOML the async fan-out is ~2× faster than serial sync (JSON stays flat — msgspec's C decoder holds the GIL). More charts in assets/benchmarks/: dumps · typed · async · memory.

Architecture

crates/native/          single Rust cdylib, one submodule per format
src/eserde/
  __init__.py           the one façade: import eserde; eserde.loads(...)
  infra/                contracts with zero internal deps: errors, formats, io, protocols
  backends/             one folder per engine: native/ (Rust), msgspec/ (C) + registry
  logic/                the facade functions: loads/dumps/load/dump/async + Jsonable encoder
tests/
  unit/eserde/          exact mirror of src
  benchmark/            rival matrix + report renderer
  resources/            canonical sample.* fixtures
docs/                   Zensical site

Only the package root has an __init__.py; every subpackage is a namespace folder. Import boundaries are enforced by tach: eserde → logic → backends → infra/_native.

Design rules: decoding is always Rust/C, validation is msgspec's or pydantic's — type= only routes the decoded tree. Round-trip losses are explicit: JSONC comments are dropped on dumps; TOML has no null; YAML non-scalar keys and multi-document streams are rejected.

Development

uv sync                  # installs the dev group, builds the extension in place
make test                # pytest
make check               # ruff + format + ty + tach + pytest (what CI runs)
make bench               # rival benchmark matrix → assets/benchmarks/*.png

Roadmap

Interop landed: eserde is a custom encoder for any framework that ducks-types json (eserde.compat), validates pydantic/attrs models through type=, accepts per-call default=/encoders=/dec_hook= hooks, inlines Markdown bodies and structured sub-documents (embed=), and streams records one at a time (iloads/idumps and async twins). Shipped alongside a comparative concurrency stress harness (make stress).

  • Pending consideration: YAML multi-document streaming (--- boundaries, reusing the event-stream scanner) — viable, but only pays for stream-shaped YAML corpora.
  • Rejected by design: TOML and INI streaming — their semantics (scattered tables, duplicate-section merge) require the global view; the contract says so loudly.
  • Next: broaden interop — msgspec/pydantic request-body and FastAPI response pipelines — and CI-gated regression against rival decoders under sustained load.

License

MIT — see LICENSE.

Metadata

Release files for e-serde 0.6.2

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for e-serde 0.6.2
File Size Uploaded Provenance Actions
e_serde-0.6.2.tar.gz 39.0 kB With provenance Details for e_serde-0.6.2.tar.gz

Built distributions (wheels)

Table of built distributions (wheels) for e-serde 0.6.2
File Actions
e_serde-0.6.2-cp314-cp314-win_amd64.whl CPython 3.14 CPython 3.14 Windows x86-64 Details for e_serde-0.6.2-cp314-cp314-win_amd64.whl
e_serde-0.6.2-cp314-cp314-manylinux_2_28_x86_64.whl CPython 3.14 CPython 3.14 Linux glibc 2.28+ x86-64 Details for e_serde-0.6.2-cp314-cp314-manylinux_2_28_x86_64.whl
e_serde-0.6.2-cp314-cp314-manylinux_2_28_aarch64.whl CPython 3.14 CPython 3.14 Linux glibc 2.28+ ARM64 Details for e_serde-0.6.2-cp314-cp314-manylinux_2_28_aarch64.whl
e_serde-0.6.2-cp314-cp314-macosx_11_0_arm64.whl CPython 3.14 CPython 3.14 macOS 11.0+ ARM64 Details for e_serde-0.6.2-cp314-cp314-macosx_11_0_arm64.whl
e_serde-0.6.2-cp314-cp314-macosx_10_12_x86_64.whl CPython 3.14 CPython 3.14 macOS 10.12+ x86-64 Details for e_serde-0.6.2-cp314-cp314-macosx_10_12_x86_64.whl

Total release size: 2.4 MB

Release files / e_serde-0.6.2.tar.gz

Download URL e_serde-0.6.2.tar.gz
Size 39.0 kB
Tags Source
SHA-256 checksum
How to use checksums
0de985251bf2b1d6e14d29f3d771d6f48ca9907e50349362724ddd67e2838e92
BLAKE2b-256 checksum
How to use checksums
d98b94de8ebe9aab2cd81d3aea1708d2374187ec02f46fd3a6c359ed5aedb4b2
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
Yes
Uploaded via twine/7.0.0 CPython/3.13.14

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Oct 26, 2026.

Transparency log

Release files / e_serde-0.6.2-cp314-cp314-win_amd64.whl

Download URL e_serde-0.6.2-cp314-cp314-win_amd64.whl
Size 424.2 kB
Tags CPython 3.14 Windows x86-64
SHA-256 checksum
How to use checksums
6f633c74c92d87e377f76f9f03a446bb2bcc3373331c5c6e731053b40d0baede
BLAKE2b-256 checksum
How to use checksums
5ebc54f570202ac066a7b653dbf7778adad2af32b981b417cfed3dbc60ba385e
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
Yes
Uploaded via twine/7.0.0 CPython/3.13.14

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Oct 26, 2026.

Transparency log

Release files / e_serde-0.6.2-cp314-cp314-manylinux_2_28_x86_64.whl

Download URL e_serde-0.6.2-cp314-cp314-manylinux_2_28_x86_64.whl
Size 516.5 kB
Tags CPython 3.14 Linux glibc 2.28+ x86-64
SHA-256 checksum
How to use checksums
602adc7affdbd862096c817eb870825d7f0a28ebfa0c0881ab40e845c75b704f
BLAKE2b-256 checksum
How to use checksums
1b104fd699d52661bd28f2759689758d8704f5ff4f789f7ce36abeaf417f6ec6
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
Yes
Uploaded via twine/7.0.0 CPython/3.13.14

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Oct 26, 2026.

Transparency log

Release files / e_serde-0.6.2-cp314-cp314-manylinux_2_28_aarch64.whl

Download URL e_serde-0.6.2-cp314-cp314-manylinux_2_28_aarch64.whl
Size 497.5 kB
Tags CPython 3.14 Linux glibc 2.28+ ARM64
SHA-256 checksum
How to use checksums
6523ba29ef6c54d5d946130e57aedf2ff5bcf831f8cec38a97e67e17106b9965
BLAKE2b-256 checksum
How to use checksums
18e6a70160ec1fe22193a66573971a235499297f3252c2e68951dc0a07552b07
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
Yes
Uploaded via twine/7.0.0 CPython/3.13.14

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Oct 26, 2026.

Transparency log

Release files / e_serde-0.6.2-cp314-cp314-macosx_11_0_arm64.whl

Download URL e_serde-0.6.2-cp314-cp314-macosx_11_0_arm64.whl
Size 467.4 kB
Tags CPython 3.14 macOS 11.0+ ARM64
SHA-256 checksum
How to use checksums
8f2f96f8de640c5088db558dd9bbe98c0fb50d887c9e5895f20509f5f12217c8
BLAKE2b-256 checksum
How to use checksums
821dda80848dfd03fe4cb76feb01639fbf577b5d6ee37f9fe456f3dac45af367
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
Yes
Uploaded via twine/7.0.0 CPython/3.13.14

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Oct 26, 2026.

Transparency log

Release files / e_serde-0.6.2-cp314-cp314-macosx_10_12_x86_64.whl

Download URL e_serde-0.6.2-cp314-cp314-macosx_10_12_x86_64.whl
Size 496.1 kB
Tags CPython 3.14 macOS 10.12+ x86-64
SHA-256 checksum
How to use checksums
f422b2ac9c97f2b5273ef89780d783d80722ee0f572abe31555ec1b43b1c06d4
BLAKE2b-256 checksum
How to use checksums
3f6ab3fb9e73d495146164de0167bdda3f9b717b192c6be27105024d125d8c7e
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
Yes
Uploaded via twine/7.0.0 CPython/3.13.14

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Oct 26, 2026.

Transparency log

Security for e-serde 0.6.2

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

Full provenance

All 6 files in this release have been uploaded with provenance metadata.

Select from 6 files

Attestation for e_serde-0.6.2.tar.gz

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Oct 26, 2026.

Transparency log

A note on package security

PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.

To learn more about protecting yourself from malicious actors, visit our package security documentation.

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page