HDF5 Research Data Management Toolbox

Tests DOCS pyvers

Note, that the project is still under development!

The "HDF5 Research Data Management Toolbox" (h5RDMtoolbox) is a python package supporting everybody who is working with HDF5 to achieve a sustainable data lifecycle which follows the FAIR (Findable, Accessible, Interoperable, Reusable) principles. It specifically supports the five main steps of

Planning (defining a internal layout for HDF5 a metadata convention for attribute usage)
Collecting data (creating HDF5 files or converting to HDF5 files from other sources)
Analyzing and processing data (Plotting, deriving data, ...)
Sharing data (publishing, archiving, ... e.g. to databases like mongoDB or repositories like Zenodo)
Reusing data (Searching data in databases, local file structures or online repositories like Zenodo).

Quickstart

A quickstart notebook can be tested by clicking on the following badge:

Documentation

Please find a comprehensive documentation with many examples here or by click on the image, which shows the research data lifecycle in the center and the respective toolbox features on the outside:

Installation

Use python 3.8 or higher (tested until 3.10). If you are a regular user, you can install the package via pip:

pip install h5RDMtoolbox

Install from source:

Developers may clone the repository and install the package from source. Clone the repository first:

git clone https://github.com/matthiasprobst/h5RDMtoolbox.git

Then, run

pip install h5RDMtoolbox/

Add --user if you do not have root access.

For development installation run

pip install -e h5RDMtoolbox/

Dependencies

The core functionality depends on the following packages. Some of them are for general management others are very specific to the features of the package:

General dependencies are ...

numpy>=1.20,<1.23.0: Scientific computing, handling of arrays
matplotlib>=3.5.2: Plotting
appdirs>=1.4.4: Managing user and application directories
packaging: Version handling
IPython>=8.4.0: Pretty display of data in notebooks
regex>=2020.7.9: Working with regular expressions

Specific to the package are ...

h5py=3.7.0: HDF5 file interface
xarray>=2022.3.0: Working with scientific arrays in combination with attributes. Allows carrying metadata from HDF5 to user
pint>=0.19.2: Allows working with units
pint_xarray>=0.2.1: Working with units for usage with xarray
python-forge==18.6.0: Used to update function signatures when using the standard attributes
pyyaml: Reading and writing of yaml files, e.g. metadata definitions (conventions)
requests: Used to download files from the internet or validate URLs, e.g. metadata definitions (conventions)

Optional dependencies

To run unit tests or to enable certain features, additional dependencies must be installed.

Install optional dependencies by specifying them in square brackets after the package name, e.g.:

pip install h5RDMtoolbox[mongodb]

[mongodb]

pymongo>=4.2.0: Database solution for HDF5 files

[io]

pco_tools>=1.0.0: Reading of pco image files
opencv-python>=4.5.3.56: Reading of image files (other than pco)
pandas>=1.4.3: Mainly used for reading csv and pretty printing

[snt]

xmltodict: Reading of xml files
tabulate>=0.8.10: Pretty printing of tables
python-gitlab: Access to gitlab repositories
pandoc>=2.3: Conversion of markdown files to html

Planned, future developments

Using JSON schema definitions for layouts and conventions

Contribution

Feel free to contribute. Make sure to write docstrings to your methods and classes and please write tests and use PEP 8 (https://peps.python.org/pep-0008/)

Please write tests for your code and put them into the test/ folder. Visit the README file in the test-folder for more information.

Pleas also add a jupyter notebook in the docs/ folder in order to document your code. Please visit the README file in the docs-folder for more information on how to compile the documentation.

Please use the numpy style for the docstrings: https://sphinxcontrib-napoleon.readthedocs.io/en/latest/example_numpy.html#example-numpy

Metadata

Release files for h5rdmtoolbox 0.12.2

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for h5rdmtoolbox 0.12.2
File	Size	Uploaded	Provenance
h5rdmtoolbox-0.12.2.tar.gz	156.4 kB	Nov 7, 2023	Without provenance	Details

Built distribution (wheel)

Table of built distributions (wheels) for h5rdmtoolbox 0.12.2
File	Interpreter	ABI	Platform	Reset
h5rdmtoolbox-0.12.2-py3-none-any.whl 192.3 kB Nov 7, 2023 Without provenance	Python 3	none	any	Details

Total release size: 348.8 kB

Release files / h5rdmtoolbox-0.12.2.tar.gz

Download URL	h5rdmtoolbox-0.12.2.tar.gz
Size	156.4 kB
Tags	Source
SHA-256 checksum How to use checksums	`13b6eddc7be65fbdb8bafd805d1cc8775cf674932270124d633ac14805ee1209`
BLAKE2b-256 checksum How to use checksums	`622c11b8a9c89c845c48d29ca82cf5cd208795dbc9f89ba38a9974bbd906a4df`
Upload date	Nov 7, 2023
Uploaded using Trusted Publishing? What is trusted publishing?	No
Uploaded via	`twine/4.0.2 CPython/3.8.18`

Release files / h5rdmtoolbox-0.12.2-py3-none-any.whl

Download URL	h5rdmtoolbox-0.12.2-py3-none-any.whl
Size	192.3 kB
Tags	Python 3
SHA-256 checksum How to use checksums	`4d9abb7aa46427c004cf0afa363ff46103e1a94e9a046dfbe97bb754148ad57a`
BLAKE2b-256 checksum How to use checksums	`5959c7ec3094e5c49fbedb100decedaa2e7e14af6ca58c250998bb6c8f3c67bc`
Upload date	Nov 7, 2023
Uploaded using Trusted Publishing? What is trusted publishing?	No
Uploaded via	`twine/4.0.2 CPython/3.8.18`

Security for h5rdmtoolbox 0.12.2

Report project as malware

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

No provenance found

None of the files in this release were uploaded with provenance metadata.

A note on package security

PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.

To learn more about protecting yourself from malicious actors, visit our package security documentation.