HDF5 Plugins for windows,MacOS and linux
Project description
This module provides HDF5 compression filters (namely: blosc, bitshuffle and lz4) and registers them to the HDF5 library used by h5py.
Supported platforms are: Linux, Windows, macOS.
Whenever possible, HDF5 compression filter plugins are best installed system-wide or through Anaconda (blosc-hdf5-plugin, hdf5-lz4). Yet, hdf5plugin provides a generic way to enable h5py with the provided HDF5 compression filters.
The HDF5 plugin sources were obtained from:
LZ4 plugin (v0.1.0): https://github.com/nexusformat/HDF5-External-Filter-Plugins
bitshuffle plugin (0.3.5): https://github.com/kiyo-masui/bitshuffle
hdf5-blosc plugin (v1.0.0) and c-blosc (v1.17.0): https://github.com/Blosc/hdf5-blosc and https://github.com/Blosc/c-blosc
Installation
To install, just run:
pip install hdf5plugin
To install locally, run:
pip install hdf5plugin --user
Documentation
To use it, just use import hdf5plugin and supported compression filters are available from h5py.
Sample code:
import numpy
import h5py
import hdf5plugin
# Compression
f = h5py.File('test.h5', 'w')
f.create_dataset('data', data=numpy.arange(100), compression=hdf5plugin.LZ4)
f.close()
# Decompression
f = h5py.File('test.h5', 'r')
data = f['data'][()]
f.close()
hdf5plugin provides:
The HDF5 filter ID of embedded plugins:
BLOSC
BSHUF
LZ4
Compression option helper functions to prepare arguments to provide to h5py.Group.create_dataset:
FILTERS: A dictionary mapping provided filters to their ID
PLUGINS_PATH: The directory where the provided filters library are stored.
bitshuffle_options(nelems=0, lz4=True)
This function takes the following arguments and returns the compression options to feed into h5py.Group.create_dataset for using the bitshuffle filter:
nelems the number of elements per block, needs to be divisible by eight (default is 0, about 8kB per block)
lz4 if True the elements get compressed using lz4 (default is True)
It returns a dict that can be passed as keyword arguments.
Sample code:
f = h5py.File('test.h5', 'w')
f.create_dataset('bitshuffle_with_lz4', data=numpy.arange(100),
**hdf5plugin.bshuf_options(nelems=0, lz4=True))
f.close()
blosc_options(level=9, shuffle=’byte’, compression=’blosclz’)
This function takes the following arguments and returns the compression options to feed into h5py.Group.create_dataset for using the blosc filter:
level the compression level, from 0 to 9 (default is 9)
shuffle the shuffling mode, either ‘none’, ‘bit’ or ‘byte’ (default is ‘byte’)
compression the compressor blosc ID, one of:
‘blosclz’ (default)
‘lz4’
‘lz4hc’
‘zlib’
‘zstd’
It returns a dict that can be passed as keyword arguments.
Sample code:
f = h5py.File('test.h5', 'w')
f.create_dataset('blosc_byte_shuffle_blosclz', data=numpy.arange(100),
**hdf5plugin.blosc_options(level=9, shuffle='byte', compression='blosclz'))
f.close()
lz4_options(nbytes=0)
This function takes the number of bytes per block as argument and returns the compression options to feed into h5py.Group.create_dataset for using the lz4 filter:
nbytes number of bytes per block needs to be in the range of 0 < nbytes < 2113929216 (1,9GB). The default value is 0 (for 1GB).
It returns a dict that can be passed as keyword arguments.
Sample code:
f = h5py.File('test.h5', 'w')
f.create_dataset('lz4', data=numpy.arange(100),
**hdf5plugin.lz4_options(nbytes=0))
f.close()
Dependencies
Testing
To run self-contained tests, from Python:
import hdf5plugin.test
hdf5plugin.test.run_tests()
Or, from the command line:
python -m hdf5plugin.test
To also run tests relying on actual HDF5 files, run from the source directory:
python test/test.py
This tests the installed version of hdf5plugin.
License
The source code of hdf5plugin itself is licensed under the MIT license. Use it at your own risk. See LICENSE
The source code of the embedded HDF5 filter plugin libraries is licensed under different open-source licenses. Please read the different licenses:
bitshuffle: See src/bitshuffle/LICENSE
blosc: See src/hdf5-blosc/LICENSES/ and src/c-blosc/LICENSES/
lz4: See src/LZ4/COPYING
The HDF5 v1.10.5 headers (and Windows .lib file) used to build the filters are stored for convenience in the repository. The license is available here: src/hdf5/COPYING.
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distributions
Hashes for hdf5plugin-2.0.0b0-py3-none-win_amd64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | c0e13338c2f0795d8cfd680006da83b2183f5a9522c067332ea3ebf2417ec21d |
|
MD5 | ba985eb6e390a8899419896be435c909 |
|
BLAKE2b-256 | aa2cd5b8577c9f09ae9f6d0c6d3d97e6beba03a9bbc618307c9f11a97ede07cb |
Hashes for hdf5plugin-2.0.0b0-py2.py3-none-manylinux2010_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 9bfd0365c6e83a8b831d41edb5e871d60520a3c97f9cecbf92ab311a04660c80 |
|
MD5 | a021c1e693adebbc57a4f46a44ee22c7 |
|
BLAKE2b-256 | f22b1423e04b34bdbe7f9f9074c9d50e1418b5f066bb959ba14e5a530d72dd0a |
Hashes for hdf5plugin-2.0.0b0-py2.py3-none-manylinux1_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 8180812c71f5f66cbe96a3106a8fa1350de29d899d11456e8ad3a138ef3513b8 |
|
MD5 | 674f240c60b39b49c0e27e8c045ee28f |
|
BLAKE2b-256 | 303ffaf8255401334d667cfaf782a2d06ba488ee3ef8b93c764bb60a02dfd964 |
Hashes for hdf5plugin-2.0.0b0-py2.py3-none-macosx_10_9_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 4fa8bbdd86d1ff54805ff5770714c9b52b84d9b7d6886a306798ae77894d5f0a |
|
MD5 | 34aa7e32c0ad4b8a9cfc6bc1592f532d |
|
BLAKE2b-256 | 2cfab72bbd0dac5fb9df9d2e7206c8f4eedffdc5d38a22058ef9868750ab8801 |
Hashes for hdf5plugin-2.0.0b0-py2-none-win_amd64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 566837ecaae82c637c1dd57042e07c1c7ad87a5471623f3d5ec5eeefb16c40fa |
|
MD5 | 699c79bd0971039dbebad9672675847e |
|
BLAKE2b-256 | 53ab8536be4c5a2bbdbeb83d733218863ccc03cc46d26199bd9909a14aacad51 |