msgspec-serde

On the one hand:

  • msgspec.Struct objects are extremely efficient and convenient replacements for dataclass; and msgspec JSON encoders and decoders are extremely efficient.
  • FlatBuffers offer both extremely efficient representations of structured data, and a declaration format that allows cross-language compatibility.

On the other hand:

  • FlatBuffers' Python API has extremely inefficient ser/de and an inconvenient builder pattern.
  • msgspec ser/de of np.arrays isn't very efficient and requires encoder/decoder hooks that traverse the python/C boundary.

msgspec-serde solves these two issues. It merges the features of FlatBuffers with msgspec and enables efficient encoding/decoding of NumPy arrays across JSON, MessagePack, and FlatBuffers in one convenient package!

Features

  • Generate typed msgspec.Struct models and lazy, buffer-backed views from FlatBuffers IDL.
    • Use msgspec_flatc tool for generating, then subclass the Struct classes to add standard msgspec validation, methods, etc. ser/de respects subclasses.
    • Tables with (key)-annotated fields are converted to dicts (msgspec) / Maps (views).
    • Additional attributes with special handling include (nested_flatbuffer ...), (dynamic_extension), (dynamic_flatbuffer), and (dynamic_allow). See the tutorial for more details.
  • FAST encoding/decoding of msgspec.Struct containing NumPy arrays into JSON, MessagePack, and FlatBuffers through dedicated Encoder and Decoder APIs.
  • Dynamic class encoding/decoding using registries is supported using constructed hook objects.
  • Supports tables, structs, enums, fixed arrays, unions, keyed vectors, schema includes, and typed or dynamic nested FlatBuffers.

Tutorial

See TUTORIAL.md for installation, schema generation, generated models and views, JSON and MessagePack codecs, unions, nested FlatBuffers, and dynamic payloads.

Backward compatibility

Within one major SemVer version, we try to keep Python code generated by an earlier release working with later msgspec-serde runtimes. This project is young, so backward compatibility is a goal rather than a guarantee.

Forward compatibility is not guaranteed. Code generated by a later release may not work with an earlier runtime.

Every generated module records the generator version and checks the installed runtime during import:

  • If the major versions differ in either direction, import raises GeneratedCodeVersionError.
  • If the major versions match but the runtime minor version is older, import emits GeneratedCodeVersionWarning once for that generator/runtime version pair.
  • Otherwise, import continues without a version warning. Patch-version differences do not produce a warning.

To suppress the older-runtime warning, set the process-wide flag before importing generated modules:

import msgspec_serde

msgspec_serde.warn_on_older_runtime = False

The flag does not suppress GeneratedCodeVersionError for a major-version mismatch.

Benchmarks

  • The generated API encodes about 20–22 times faster and materializes models about 11–13 times faster than the official Python FlatBuffers API in the measured workloads.
  • For the recursive numeric workload, MessagePack encodes fastest with 16-value vectors, while FlatBuffers materializes fastest. With 256-value vectors, FlatBuffers is fastest in both directions.

See benchmarks.md for charts, methodology, environment details, and reproducible commands.

Key dates

PyPI data
Data sourced directly from PyPI's database.
  • Released:
Newer release available (0.1.1)

1 maintainer

PyPI data
Data sourced directly from PyPI's database.
Avatar for ebrevdo from gravatar.com ebrevdo

License expression

Apache-2.0
View SPDX License List

Requires

Python >=3.12

Report project as malware

Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

msgspec_serde-0.1.0.tar.gz (112.6 kB view details)

Uploaded Source

Built Distributions

If you're not sure about the file name format, learn more about wheel file names.

msgspec_serde-0.1.0-cp314-cp314t-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (915.2 kB view details)

Uploaded CPython 3.14tmanylinux: glibc 2.17+ x86-64

msgspec_serde-0.1.0-cp314-cp314t-manylinux_2_17_aarch64.manylinux2014_aarch64.whl (894.5 kB view details)

Uploaded CPython 3.14tmanylinux: glibc 2.17+ ARM64

msgspec_serde-0.1.0-cp314-cp314t-macosx_11_0_arm64.whl (812.7 kB view details)

Uploaded CPython 3.14tmacOS 11.0+ ARM64

msgspec_serde-0.1.0-cp314-cp314t-macosx_10_12_x86_64.whl (834.7 kB view details)

Uploaded CPython 3.14tmacOS 10.12+ x86-64

msgspec_serde-0.1.0-cp312-abi3-win_amd64.whl (707.2 kB view details)

Uploaded CPython 3.12+Windows x86-64

msgspec_serde-0.1.0-cp312-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (914.6 kB view details)

Uploaded CPython 3.12+manylinux: glibc 2.17+ x86-64

msgspec_serde-0.1.0-cp312-abi3-manylinux_2_17_aarch64.manylinux2014_aarch64.whl (894.6 kB view details)

Uploaded CPython 3.12+manylinux: glibc 2.17+ ARM64

msgspec_serde-0.1.0-cp312-abi3-macosx_11_0_arm64.whl (812.9 kB view details)

Uploaded CPython 3.12+macOS 11.0+ ARM64

msgspec_serde-0.1.0-cp312-abi3-macosx_10_12_x86_64.whl (836.7 kB view details)

Uploaded CPython 3.12+macOS 10.12+ x86-64

File details

Details for the file msgspec_serde-0.1.0.tar.gz.

File metadata

  • Download URL: msgspec_serde-0.1.0.tar.gz
  • Upload date:
  • Size: 112.6 kB
  • Tags: Source
  • Uploaded using Trusted Publishing? Yes
  • Uploaded via: twine/7.0.0 CPython/3.13.14

File hashes

Hashes for msgspec_serde-0.1.0.tar.gz
Algorithm Hash digest
SHA256 2090b876be864b8a5ff00f34303daa87520e71eee6c02601775aef6598af4fb8
MD5 705d9b903203a0c17e135c90116d5b0b
BLAKE2b-256 bd306f26a5d3c571f72b84a867cbe1d24729e61e5971e6197b30a3e2513eb4cb

See more details on using hashes here.

Provenance

The following attestation bundles were made for msgspec_serde-0.1.0.tar.gz:

Publisher: release.yml on ebrevdo/msgspec-serde

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

File details

Details for the file msgspec_serde-0.1.0-cp314-cp314t-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.

File metadata

File hashes

Hashes for msgspec_serde-0.1.0-cp314-cp314t-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm Hash digest
SHA256 360346b0d4827f8fcd94301221b5f14992fd7903d9ab5cab1cd92fa80aafe8b8
MD5 5ff85780667d7d7b1a54fa8e7983c892
BLAKE2b-256 9515ba3bd573046d60f9486d94140c761bc37018674f18f3399b845bb838a44e

See more details on using hashes here.

Provenance

The following attestation bundles were made for msgspec_serde-0.1.0-cp314-cp314t-manylinux_2_17_x86_64.manylinux2014_x86_64.whl:

Publisher: release.yml on ebrevdo/msgspec-serde

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

File details

Details for the file msgspec_serde-0.1.0-cp314-cp314t-manylinux_2_17_aarch64.manylinux2014_aarch64.whl.

File metadata

File hashes

Hashes for msgspec_serde-0.1.0-cp314-cp314t-manylinux_2_17_aarch64.manylinux2014_aarch64.whl
Algorithm Hash digest
SHA256 a02b85f63523d2263be5e08ff76030f154b5507603c5c2e7bec623b472769ef2
MD5 e1e3caa0ef662d154c003ae1c82e5b25
BLAKE2b-256 925fc2399daa80748a34fe1200c37ce6eab2a1bef4743a3f83c63681ffdc9030

See more details on using hashes here.

Provenance

The following attestation bundles were made for msgspec_serde-0.1.0-cp314-cp314t-manylinux_2_17_aarch64.manylinux2014_aarch64.whl:

Publisher: release.yml on ebrevdo/msgspec-serde

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

File details

Details for the file msgspec_serde-0.1.0-cp314-cp314t-macosx_11_0_arm64.whl.

File metadata

File hashes

Hashes for msgspec_serde-0.1.0-cp314-cp314t-macosx_11_0_arm64.whl
Algorithm Hash digest
SHA256 d84f5161ea8ca3cb98b92c2c89697a4907670031d8f44d2d8120beb7b016993b
MD5 6167b54bb8781bb03a8c7071bdf7688f
BLAKE2b-256 066ade4e0cee7dd5c027b6ec9dc81ff2f0ba037edc5648247389874b7645ea42

See more details on using hashes here.

Provenance

The following attestation bundles were made for msgspec_serde-0.1.0-cp314-cp314t-macosx_11_0_arm64.whl:

Publisher: release.yml on ebrevdo/msgspec-serde

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

File details

Details for the file msgspec_serde-0.1.0-cp314-cp314t-macosx_10_12_x86_64.whl.

File metadata

File hashes

Hashes for msgspec_serde-0.1.0-cp314-cp314t-macosx_10_12_x86_64.whl
Algorithm Hash digest
SHA256 1f99c9256d36ede6f7160f6ea3eaf3353cd00f20e4adedfb84b985cc65a1b3eb
MD5 6f04f27d3a3082cefe0a41c93a8e8e01
BLAKE2b-256 f9d74c485cc090c9dd06a7c7a70964a698a8b66ccbba9ef9d5f2f754fca01356

See more details on using hashes here.

Provenance

The following attestation bundles were made for msgspec_serde-0.1.0-cp314-cp314t-macosx_10_12_x86_64.whl:

Publisher: release.yml on ebrevdo/msgspec-serde

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

File details

Details for the file msgspec_serde-0.1.0-cp312-abi3-win_amd64.whl.

File metadata

File hashes

Hashes for msgspec_serde-0.1.0-cp312-abi3-win_amd64.whl
Algorithm Hash digest
SHA256 b876f4bfe8efca4b940d222f00dd603c0cf253f501d917d1a11fc32c66878916
MD5 4a2aa0d638cffaf9d4074d23efc1d7b2
BLAKE2b-256 8f41eb99a27f4f4bf125d08284b3296d30d97e6cb4213acdcf8decec208ba52d

See more details on using hashes here.

Provenance

The following attestation bundles were made for msgspec_serde-0.1.0-cp312-abi3-win_amd64.whl:

Publisher: release.yml on ebrevdo/msgspec-serde

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

File details

Details for the file msgspec_serde-0.1.0-cp312-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.

File metadata

File hashes

Hashes for msgspec_serde-0.1.0-cp312-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm Hash digest
SHA256 063e010e925c4b03bbe8f23c52628c13bdd18abe2e627298a40145065d420eae
MD5 5ca757836ff4043fb7fd8c61b1591e7e
BLAKE2b-256 4d2e1dfbc3d61752d43008dd2d8ad73a91873e6e2967af030df897afcca8cd6a

See more details on using hashes here.

Provenance

The following attestation bundles were made for msgspec_serde-0.1.0-cp312-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl:

Publisher: release.yml on ebrevdo/msgspec-serde

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

File details

Details for the file msgspec_serde-0.1.0-cp312-abi3-manylinux_2_17_aarch64.manylinux2014_aarch64.whl.

File metadata

File hashes

Hashes for msgspec_serde-0.1.0-cp312-abi3-manylinux_2_17_aarch64.manylinux2014_aarch64.whl
Algorithm Hash digest
SHA256 9c850e140ff04e9d085a4f360503687ec0cfdc7f79f4345b06b157ecbfcc98f0
MD5 095e683d4736f28c146efa282700ab6f
BLAKE2b-256 c6bd0959a445406cfb97b809f51f9237e70ccd13fb4d30e1f10c2402c6892d30

See more details on using hashes here.

Provenance

The following attestation bundles were made for msgspec_serde-0.1.0-cp312-abi3-manylinux_2_17_aarch64.manylinux2014_aarch64.whl:

Publisher: release.yml on ebrevdo/msgspec-serde

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

File details

Details for the file msgspec_serde-0.1.0-cp312-abi3-macosx_11_0_arm64.whl.

File metadata

File hashes

Hashes for msgspec_serde-0.1.0-cp312-abi3-macosx_11_0_arm64.whl
Algorithm Hash digest
SHA256 33fe3bea418e696be634f40d81cbbf99f73ef9d1bc75344bd6ba656515b727d5
MD5 8f4983a653a7300ada64883021f46ad9
BLAKE2b-256 90eb55c0f39d472274860bf3fb62d00f3ffa232ccf83524c23c23e1f2f5fd4e5

See more details on using hashes here.

Provenance

The following attestation bundles were made for msgspec_serde-0.1.0-cp312-abi3-macosx_11_0_arm64.whl:

Publisher: release.yml on ebrevdo/msgspec-serde

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

File details

Details for the file msgspec_serde-0.1.0-cp312-abi3-macosx_10_12_x86_64.whl.

File metadata

File hashes

Hashes for msgspec_serde-0.1.0-cp312-abi3-macosx_10_12_x86_64.whl
Algorithm Hash digest
SHA256 37b72b4494d11a7d96db84581e29661a45cc2671a62c4b1a9aa5fcbb13ce6688
MD5 80c49c0e8c94698c7991a59434c0ed80
BLAKE2b-256 cc6126b4302d1c455b2be0649fbb0ab4dc45a5c0103c89ea859b36772225d8df

See more details on using hashes here.

Provenance

The following attestation bundles were made for msgspec_serde-0.1.0-cp312-abi3-macosx_10_12_x86_64.whl:

Publisher: release.yml on ebrevdo/msgspec-serde

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

Release history Release notifications | RSS feed

0.1.1

10 files

This release

0.1.0 This release

10 files

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page