mpacklog 4.0.0

Log dictionaries to file using the MessagePack serialization format.

mpacklog.py

Stream dictionaries to files or over the network using MessagePack in Python.

Installation

From conda-forge

conda install -c conda-forge mpacklog

From PyPI

pip install mpacklog

Usage

Asynchronous API

Add messages to the log using the put function, have them written to file in the separate write coroutine.

import asyncio
import mpacklog

async def main():
    logger = mpacklog.AsyncLogger("output.mpack")
    await asyncio.gather(main_loop(logger), logger.write())

async def main_loop(logger):
    for bar in range(1000):
        await asyncio.sleep(1e-3)
        await logger.put({"foo": bar, "something": "else"})
    await logger.stop()

if __name__ == "__main__":
    asyncio.run(main())

Synchronous API

The synchronous API is similar to the asynchronous API, except it doesn't provide a stop method and the put and write methods are blocking.

import mpacklog

logger = mpacklog.SyncLogger("output.mpack")

for bar in range(1000):
    logger.put({"foo": bar, "something": "else"})

# Flush all messages to the file
logger.write()

Command-line

The mpacklog utility provides commands to manipulate .mpack files.

dump

The dump command writes down a log file to newline-delimited JSON:

mpacklog dump my_log.mpack

list

This commands lists all nested dictionary keys encountered in a log file. Nested keys are separated by slashes / in the output. For instance, if some dictionaries in my_log.mpack contain values at dict["foo"]["bar"] and dict["foo"]["cbs"], the command will produce:

$ mpacklog list my_log.mpack
- foo/bar
- foo/cbs

serve

The serve command watches a log file for updates and serves the last dictionary appended to it over the network. Its argument is either a log file or a directory containing log files. In the second case, the most recent log files in the directory is opened:

$ mpacklog serve /var/log/my_mpack_logs/

You can use mpackview to connect a live plot to the server, or develop your own tool (the server API is quite simple).

See also

• foxplot: explore and plot time-series data from MessagePack and line-delimited JSON files.
• jq: manipulate JSON series to add, remove or extend fields.
• mpacklog.cpp: log dictionaries to MessagePack files in C++.
• rq: transform from/to MessagePack, JSON, YAML, TOML, etc. For instance mpacklog dump is equivalent to rq -mJ < my_log.mpack.