msgpack-chunk 0.2.0.dev0

MessagePack serializer for chunked data

Chunked implementation of MessagePack for python.

Description

The library is designed to efficiently pack and unpack large amounts of binary data for data transfer.

This is achieved with an iterative API, which allow to reuse the memory (reducing the memory footprint), and features zero-copy operations (speeding up the transformation).

Benchmark

The main use case is to transfer big binary data together with some metadata. Here is benchmark with few different setups, including the officiel msgpack library as reference.

Here is a log/log diagrams which show GB/s related to the amount of data to process.

Pack

Unpack

Example

Pack

import msgpack_chunk
packer = msgpack_chunk.Packer(
    expose_object_buffer="full",
)

io = ...

packer.pack(python_object)
main_buffer = bytearray(64 * 1024)
while result_buffer := packer.read_into(main_buffer):
    io.write(result_buffer)

Unpack

import msgpack_chunk
unpacker = msgpack_chunk.Unpacker()

io = ...

main_buffer = bytearray(64 * 1024)
while True:
    nb = io.readinto(main_buffer)
    if nb == 0:
        break
    unpacker.feed(main_buffer[:nb])

python_objects = list(unpacker)

Advanced unpack

Allow to write directly inside the final python object, when dealing with big binary or ext type data.

import msgpack_chunk
unpacker = msgpack_chunk.Unpacker(
    bin="bytearray",
    expose_direct_buffer=True,
)

io = ...

main_buffer = bytearray(64 * 1024)
while True:
    direct_buffer = unpacker.get_buffer()
    if direct_buffer is not None and len(direct_buffer) > 16 * 1024:
        nb = io.readinto(direct_buffer)
        if not data:
            break
        unpacker.buffer_updated(nb)
    else:
        nb = io.readinto(main_buffer)
        if nb == 0:
            break
        unpacker.feed(main_buffer[:nb])

python_objects = list(unpacker)

Resources

• msgpack, the official python library library provides an efficiant Cython implementation.
• The msgpack specification