High performance parallel reading of HDF5 files using PyTables, multiprocessing, and shared memory.
multitables is a python library designed for high speed access to HDF5 files. Access to HDF5 is provided by the PyTables library (tables). Multiple processes are launched to read a HDF5 in parallel, allowing concurrent decompression. Data is streamed back to the invoker by use of shared memory space, removing the usual multiprocessing communication overhead.
The data is organised by rows of an array (elements of the outer-most dimension), and groups of these rows form blocks. By default, there is no guarantee on the ordering of the rows and/or blocks returned to the user, due to the concurrent nature of the library. They are returned as they become available. On-disk ordering can be forced using the ordered option, which may result in a performance penalty.
Performance gains of at least 2x can be achieved when reading from an SSD.
This software is distributed under the MIT licence. See the LICENSE.txt file for details.
pip install multitables
multitables depends on tables (the pytables package) and numpy. The package is compatible with the latest versions of python 2 and 3.
import multitables stream = multitables.Streamer(filename='/path/to/h5/file') for row in stream.get_generator(path='/internal/h5/path'): do_something(row)