mmnpz

mmnpz is a small Python package for handling large .npz files. .npz files are uncompressed zip files containing numpy arrays. mmnpz unlocks their potential as an efficient, standardized option for storing and accessing large datasets.

Installation

mmnpz is available on PyPI. To install the latest release, run:

pip install mmnpz

Usage

mmnpz can write large .npz files incrementally:

>>> import numpy as np
>>> import mmnpz
>>> with mmnpz.NpzWriter("test.npz") as f:
>>>     for i in range(10):
>>>         f.write(f"a{i}", np.full(10000, i))

mmnpz can read large .npz files as memory maps:

>>> import mmnpz
>>> x = mmnpz.load("test.npz")
>>> x["a2"][10:15]
memmap([2, 2, 2, 2, 2])

This allows accessing individual excerpts of large datasets without I/O overhead.

Documentation

The documentation is hosted on readthedocs.

Acknowledgements

The package layout and toolchain is based on Ai2's Python package template, Microsoft's Python package template and pytest's Python package. The package was developed at the Institute of Computational Perception, Johannes Kepler University Linz.