mpacklog 2.0.0

Log dictionaries to file using the MessagePack serialization format.

mpacklog

Log dictionaries to MessagePack files in C++ and Python.

Installation

Python

pip install mpacklog

Bazel

Add a git repository rule to your Bazel WORKSPACE:

load("@bazel_tools//tools/build_defs/repo:git.bzl", "git_repository")

git_repository(
    name = "mpacklog",
    remote = "https://github.com/stephane-caron/mpacklog.git",
    tag = "v1.0.0",
)

You can then use the @mpacklog dependency for C++ targets, or the @mpacklog//:python dependency for Python targets.

Usage

C++

The C++ implementation uses multi-threading. Add messages to the log using the put function, they will be written to file in the background.

#include <mpacklog/Logger.h>
#include <palimpsest/Dictionary.h>

int main() {
    mpacklog::Logger logger("output.mpack");

    for (unsigned bar = 0; bar < 1000u; ++bar) {
        palimpsest::Dictionary dict;
        dict("foo") = bar;
        dict("something") = "else";
        logger.put(dict):
    }
}

Python

The Python implementation uses asynchronous I/O. Add messages to the log using the put function, have them written to file in the separate write coroutine.

import asyncio
import mpacklog

async def main():
    logger = mpacklog.Logger("output.mpack")

    await asyncio.gather(
        main_loop(logger),
        logger.write(),
    )

async def main_loop(logger):
    for bar in range(1000):
        await asyncio.sleep(1e-3)
        await logger.put({"foo": bar, "something": "else"})
    await logger.stop()

if __name__ == "__main__":
    asyncio.run(main())

Command-line

If you pip-installed mpacklog, you can use the mpacklog command to dump logs to JSON:

mpacklog dump my_log.mpack

Alternatively and more generally, two great tools to manipulate logs from the command line are:

• rq: transform from/to MessagePack, JSON, YAML, TOML, ...
• jq: manipulate JSON series to add, remove or extend fields

For instance, mpacklog dump is equivalent to:

rq -mJ < my_log.mpack