A (de)serialization library for multidimensional arrays
Project description
seedee
A data format for n-dimensional arrays.
This project is in an early stage of development and should be used with caution
Installation
Python
Pre-built Python wheels can be installed via pip
:
python3 -m pip install seedee
To build a wheel yourself, e.g., if your platform is not supported by the pre-built wheels, download the latest release and extract it. Inside the source directory, run:
$ cmake -S . -B build -DBUILD_PYTHON=ON -DCMAKE_BUILD_TYPE=Release
$ cmake --build build --target python-wheel --verbose
Building requires cmake
version 3.13
or newer.
If everything works, the wheel is created in the build/python/dist/
subdirectory and can be installed via pip
.
C/C++
To build and install the C/C++ libraries, run:
$ cmake -S . -B build -DCMAKE_BUILD_TYPE=Release -DCMAKE_INSTALL_PREFIX=path/to/install/dir
$ cmake --build build --target seedee --verbose
$ cmake --install build
Building requires cmake
version 3.13
or newer.
Usage example
Python
import numpy as np
import seedee
# Create a numpy array of some dimension, shape, and dtype
array = np.arange(48).reshape((6, 8)).astype("uint16")
# Serialize the array using the bitshuffle+lz4 compression method
buffer, meta = seedee.serialize(array, seedee.COMPRESSION_BITSHUFFLE)
# buffer is a 1D numpy array of type uint8
# meta is a string
# Send buffer and meta to a remote process using any preferred method,
# e.g., via a ZMQ multi-message
...
# Deserialize the received buffer and meta into a numpy array
new_array = seedee.deserialize(buffer, meta)
# The resulting array is equal to the original array
assert np.all(array == new_array)
C/C++
See test
directory for some examples. Note that the C/C++ interface will
likely change in the future.
Limitations
Only arrays up to 1GB are supported.
Contribution
Please feel free to open issues or pull requests.
Acknowledgments
seedee
depends on multiple open source libraries that are included in this
repository under the thirdparty
directory for convenience. The licenses of
these dependencies can be found in the corresponding subdirectories and in
LICENSE_THIRDPARTY.txt.
The chunk serializer implementations are based on the corresponding HDF5 filter plugins which are subject to their own licenses:
- LZ4 filter plugin: https://github.com/nexusformat/HDF5-External-Filter-Plugins/blob/master/LZ4/COPYING
- bitshuffle filter plugin: https://github.com/kiyo-masui/bitshuffle/blob/master/LICENSE
- zlib filter plugin: https://github.com/HDFGroup/hdf5/blob/develop/COPYING
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distributions
Hashes for seedee-0.3.0-cp312-cp312-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 8f8cdefad32b2a9ce343e693e77835d616bd2002e4dda41fecffb59267e45ee1 |
|
MD5 | 5ff91f832bdbc7c1e678dcd19e429b34 |
|
BLAKE2b-256 | 4dc48cbf816bfab692eb8f41e527deb183b828afb951f9cf8c125215d543c998 |
Hashes for seedee-0.3.0-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 7df4db3f18d1c94049abe257924704308c971aba251fd1b665d2267daab3b005 |
|
MD5 | 817494b85e0feb98a2a531b02596c627 |
|
BLAKE2b-256 | 99629f210ab35006a2ce2c41f26f18a057a9eb020c51e21ee15e97680e2443ff |
Hashes for seedee-0.3.0-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 46954a612aac0fb94726b37738eb1cfe4e74910cdebd96d9a4c099823a3ea7ec |
|
MD5 | 9fb382be8a50b341798ed8bfb305fdd9 |
|
BLAKE2b-256 | 14998b9e3e3b2bd92450267fb715b82e7abd3e7199b950f330ba74625c69ec0f |
Hashes for seedee-0.3.0-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 73445943489b589b53ca3476462a8ef2e29947bfd31a3168dd50225d20e3de72 |
|
MD5 | 127555d8a8093fce0c305d891acce5e5 |
|
BLAKE2b-256 | dba9ef4f7b8e1c95a6dbaa6b567e4f0591b1f4e43d315c947411fd09bc4fbdd6 |
Hashes for seedee-0.3.0-cp38-cp38-manylinux_2_12_x86_64.manylinux2010_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 3b1ba7338f797b8482038a431ed05753489b94dd709792c7887d4fad08c8e846 |
|
MD5 | b642e92038301c6df1ad6bf1ccc7928b |
|
BLAKE2b-256 | 32c15d98895ce2140d993fece5324194b948fc3caca1b41cc978341d40d49d71 |
Hashes for seedee-0.3.0-cp37-cp37m-manylinux_2_12_x86_64.manylinux2010_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 19a2ba2b0dd76f8278e6e735d2269fb4111e704aa11465e6e6e936fbe35649fa |
|
MD5 | 986b5535ced032616fefc72d7c2fe571 |
|
BLAKE2b-256 | 27c6c05bdcc26f0db76a4697da677d68e956f85e7bc0966d26a01d429258499a |
Hashes for seedee-0.3.0-cp36-cp36m-manylinux_2_12_x86_64.manylinux2010_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 9d65bfbf48d2ab53a6886f4ffaa16f4c90059f10335d2503e49072dea26e68e1 |
|
MD5 | 4a925ebb0d0eb8332b840aed7aff6a59 |
|
BLAKE2b-256 | a57d7291503552ef2fb76693bacccffc9456c9c8eb991f03a083120ac398cece |