Skip to main content

High performance parallel reading of HDF5 files using PyTables, multiprocessing, and shared memory.

Project description

multitables is a python library designed for high speed access to HDF5 files. Access to HDF5 is provided by the PyTables library (tables). Multiple processes are launched to read a HDF5 in parallel, allowing concurrent decompression. Data is streamed back to the invoker by use of shared memory space, removing the usual multiprocessing communication overhead.

The data is organised by rows of an array (elements of the outer-most dimension), and groups of these rows form blocks. With the Streamer interface, there is no guarantee on the ordering of the rows and/or blocks returned to the user, due to the concurrent nature of the library. They are returned as they become available. On-disk ordering can be forced using the ordered option, which may result in a performance penalty.

The Reader interface allows random access, which gives fine grained control over the ordering of requests.

Performance gains of at least 2x can be achieved when reading from an SSD.

New with version 2

Random access reads are now possible through asynchronous requests. The results of these requests are placed in shared memory. See the how-to and unit tests for examples of the new interface.

Licence

This software is distributed under the MIT licence. See the LICENSE.txt file for details.

Installation

pip install multitables

multitables depends on tables (the pytables package), numpy, msgpack, and wrapt. The package is compatible with the latest versions of Python 3, as pytables no longer supports Python 2.

Quick start: Streaming

import multitables
stream = multitables.Streamer(filename='/path/to/h5/file')
for row in stream.get_generator(path='/internal/h5/path'):
    do_something(row)

Quick start: Random access

import multitables
reader = multitables.Reader(filename='/path/to/h5/file')

dataset = reader.get_dataset(path='/internal/h5/path')
stage = dataset.create_stage(10) # Size of the shared
                                    # memory stage in rows

req = dataset['col_A'][30:35] # Create a request as you
                                 # would index normally.

future = reader.request(req, stage) # Schedule the request
with future.get_unsafe() as data:
    do_something(data)
data = None # Always set data to None after get_unsafe to
            # prevent a dangling reference

# ... or use a safer proxy method

req = dataset.col('col_A')[30:35,...,:100]

future = reader.request(req, stage)
with future.get_proxy() as data:
    do_something(data)

# ... or get a copy of the data

req = dataset['col_A'][30:35,np.arange(500) > 45]

future = reader.request(req, stage)
do_something(future.get())

Examples

See the how-to for more in-depth documentation, and the unit tests for complete examples.

Documentation

Online documentation is available. A how to gives a basic overview of the library. A benchmark tests the speed of the library using various compression algorithms and hardware configurations.

Offline documentation can be built from the docs folder using sphinx.

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Files for multitables, version 2.0.1
Filename, size File type Python version Upload date Hashes
Filename, size multitables-2.0.1-py2.py3-none-any.whl (34.8 kB) File type Wheel Python version py2.py3 Upload date Hashes View

Supported by

AWS AWS Cloud computing Datadog Datadog Monitoring DigiCert DigiCert EV certificate Facebook / Instagram Facebook / Instagram PSF Sponsor Fastly Fastly CDN Google Google Object Storage and Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Salesforce Salesforce PSF Sponsor Sentry Sentry Error logging StatusPage StatusPage Status page