Skip to main content

C++ & Python API for Scientific I/O with openPMD

Project description

C++ & Python API for Scientific I/O with openPMD

Supported openPMD Standard Documentation Status Doxygen Gitter chat Supported Platforms License DOI
CodeFactor LGTM: C/C++ LGTM: Python LGTM: Total alerts Coverage Status
Linux/OSX Build Status dev Windows Build Status dev Nightly Packages Status Coverity Scan Build Status

openPMD is an open meta-data schema that provides meaning and self-description to data sets in science and engineering. See the openPMD standard for details of this schema.

This library provides a reference API for openPMD data handling. Since openPMD is a schema (or markup) on top of portable, hierarchical file formats, this library implements various backends such as HDF5, ADIOS1, ADIOS2 and JSON. Writing & reading through those backends and their associated files is supported for serial and MPI-parallel workflows.

Usage

C++

C++11 C++11 API: Alpha

#include <openPMD/openPMD.hpp>
#include <iostream>

// ...

auto s = openPMD::Series("samples/git-sample/data%T.h5", openPMD::AccessType::READ_ONLY);

for( auto const& i : s.iterations ) {
    std::cout << "Iteration: " << i.first << "\n";

    for( auto const& m : i.second.meshes ) {
        std::cout << "  Mesh '" << m.first << "' attributes:\n";
        for( auto const& val : m.second.attributes() )
            std::cout << "    " << val << '\n';
    }

    for( auto const& p : i.second.particles ) {
        std::cout << "  Particle species '" << p.first << "' attributes:\n";
        for( auto const& val : p.second.attributes() )
            std::cout << "    " << val << '\n';
    }
}

Python

Python3 Python3 API: Alpha

import openpmd_api

# ...

series = openpmd_api.Series("samples/git-sample/data%T.h5", openPMD.Access_Type.read_only)

for k_i, i in series.iterations.items():
    print("Iteration: {0}".format(k_i))

    for k_m, m in i.meshes.items():
        print("  Mesh '{0}' attributes:".format(k_m))
        for a in m.attributes:
            print("    {0}".format(a))

    for k_p, p in i.particles.items():
        print("  Particle species '{0}' attributes:".format(k_p))
        for a in p.attributes:
            print("    {0}".format(a))

More!

Curious? Our manual shows full read & write examples, both serial and MPI-parallel!

Dependencies

Required:

  • CMake 3.11.0+
  • C++11 capable compiler, e.g. g++ 4.8+, clang 3.9+, VS 2015+

Shipped internally in share/openPMD/thirdParty/:

I/O backends:

while those can be built either with or without:

  • MPI 2.1+, e.g. OpenMPI 1.6.5+ or MPICH2

Optional language bindings:

  • Python:
    • Python 3.5 - 3.8
    • pybind 2.3.0+
    • numpy 1.15+
    • mpi4py 2.1+

Installation

Spack Package Conda Package PyPI Package From Source

Choose one of the install methods below to get started:

Spack

Spack Version Spack Status

# optional:               +python +adios1 -adios2 -hdf5 -mpi
spack install openpmd-api
spack load -r openpmd-api

Conda

Conda Version Conda Status Conda Downloads

# optional:            OpenMPI support  =*=mpi_openmpi*
# optional:              MPICH support  =*=mpi_mpich*
conda install -c conda-forge openpmd-api

PyPI

PyPI Version PyPI Format PyPI Status PyPI Downloads

Behind the scenes, this install method compiles from source against the found installations of HDF5, ADIOS1, ADIOS2, and/or MPI (in system paths, from other package managers, or loaded via a module system, ...). The current status for this install method is experimental. Please feel free to report how this works for you.

# we need pip 19 or newer
# optional:                   --user
python3 -m pip install -U pip

# optional:                        --user
python3 -m pip install openpmd-api

or with MPI support:

# optional:                                    --user
python3 -m pip install -U pip setuptools wheel
python3 -m pip install -U cmake

# optional:                                                                   --user
openPMD_USE_MPI=ON python3 -m pip install openpmd-api --no-binary openpmd-api

From Source

Source Status

openPMD-api can also be built and installed from source using CMake:

git clone https://github.com/openPMD/openPMD-api.git

mkdir openPMD-api-build
cd openPMD-api-build

# optional: for full tests, with unzip
../openPMD-api/.travis/download_samples.sh

# for own install prefix append:
#   -DCMAKE_INSTALL_PREFIX=$HOME/somepath
# for options append:
#   -DopenPMD_USE_...=...
# e.g. for python support add:
#   -DopenPMD_USE_PYTHON=ON -DPYTHON_EXECUTABLE=$(which python3)
cmake ../openPMD-api

cmake --build .

# optional
ctest

# sudo might be required for system paths
cmake --build . --target install

The following options can be added to the cmake call to control features. CMake controls options with prefixed -D, e.g. -DopenPMD_USE_MPI=OFF:

CMake Option Values Description
openPMD_USE_MPI AUTO/ON/OFF Parallel, Multi-Node I/O for clusters
openPMD_USE_HDF5 AUTO/ON/OFF HDF5 backend (.h5 files)
openPMD_USE_ADIOS1 AUTO/ON/OFF ADIOS1 backend (.bp files up to version BP3)
openPMD_USE_ADIOS2 AUTO/ON/OFF ADIOS2 backend (.bp files in BP3, BP4 or higher)
openPMD_USE_PYTHON AUTO/ON/OFF Enable Python bindings
openPMD_USE_INVASIVE_TESTS ON/OFF Enable unit tests that modify source code 1
openPMD_USE_VERIFY ON/OFF Enable internal VERIFY (assert) macro independent of build type 2
PYTHON_EXECUTABLE (first found) Path to Python executable

1 e.g. changes C++ visibility keywords, breaks MSVC 2 this includes most pre-/post-condition checks, disabling without specific cause is highly discouraged

Additionally, the following libraries are shipped internally. The following options allow to switch to external installs:

CMake Option Values Library Version
openPMD_USE_INTERNAL_VARIANT ON/OFF MPark.Variant 1.4.0+
openPMD_USE_INTERNAL_CATCH ON/OFF Catch2 2.6.1+
openPMD_USE_INTERNAL_PYBIND11 ON/OFF pybind11 2.3.0+
openPMD_USE_INTERNAL_JSON ON/OFF NLohmann-JSON 3.7.0+

By default, this will build as a shared library (libopenPMD.[so|dylib|dll]) and installs also its headers. In order to build a static library, append -DBUILD_SHARED_LIBS=OFF to the cmake command. You can only build a static or a shared library at a time.

By default, the Release version is built. In order to build with debug symbols, pass -DCMAKE_BUILD_TYPE=Debug to your cmake command.

By default, tests, examples and command line tools are built. In order to skip building those, pass -DBUILD_TESTING=OFF, -DBUILD_EXAMPLES=OFF, or -DBUILD_CLI_TOOLS=OFF to your cmake command.

Linking to your project

The install will contain header files and libraries in the path set with -DCMAKE_INSTALL_PREFIX.

CMake

If your project is using CMake for its build, one can conveniently use our provided openPMDConfig.cmake package which is installed alongside the library.

First set the following environment hint if openPMD-api was not installed in a system path:

# optional: only needed if installed outside of system paths
export CMAKE_PREFIX_PATH=$HOME/somepath:$CMAKE_PREFIX_PATH

Use the following lines in your project's CMakeLists.txt:

# supports:                       COMPONENTS MPI NOMPI HDF5 ADIOS1 ADIOS2
find_package(openPMD 0.9.0 CONFIG)

if(openPMD_FOUND)
    target_link_libraries(YourTarget PRIVATE openPMD::openPMD)
endif()

Alternatively, add the openPMD-api repository source directly to your project and use it via:

add_subdirectory("path/to/source/of/openPMD-api")

target_link_libraries(YourTarget PRIVATE openPMD::openPMD)

For development workflows, you can even automatically download and build openPMD-api from within a depending CMake project. Just replace the add_subdirectory call with:

include(FetchContent)
set(CMAKE_POLICY_DEFAULT_CMP0077 NEW)
set(BUILD_TESTING OFF)
set(BUILD_EXAMPLES OFF)
set(openPMD_USE_PYTHON OFF)
FetchContent_Declare(openPMD
  GIT_REPOSITORY "https://github.com/openPMD/openPMD-api.git"
  GIT_TAG        "dev")
FetchContent_MakeAvailable(openPMD)

Manually

If your (Linux/OSX) project is build by calling the compiler directly or uses a manually written Makefile, consider using our openPMD.pc helper file for pkg-config which are installed alongside the library.

First set the following environment hint if openPMD-api was not installed in a system path:

# optional: only needed if installed outside of system paths
export PKG_CONFIG_PATH=$HOME/somepath/lib/pkgconfig:$PKG_CONFIG_PATH

Additional linker and compiler flags for your project are available via:

# switch to check if openPMD-api was build as static library
# (via BUILD_SHARED_LIBS=OFF) or as shared library (default)
if [ "$(pkg-config --variable=static openPMD)" == "true" ]
then
    pkg-config --libs --static openPMD
    # -L/usr/local/lib -L/usr/lib/x86_64-linux-gnu/openmpi/lib -lopenPMD -pthread /usr/lib/libmpi.so -pthread /usr/lib/x86_64-linux-gnu/openmpi/lib/libmpi_cxx.so /usr/lib/libmpi.so /usr/lib/x86_64-linux-gnu/hdf5/openmpi/libhdf5.so /usr/lib/x86_64-linux-gnu/libsz.so /usr/lib/x86_64-linux-gnu/libz.so /usr/lib/x86_64-linux-gnu/libdl.so /usr/lib/x86_64-linux-gnu/libm.so -pthread /usr/lib/libmpi.so -pthread /usr/lib/x86_64-linux-gnu/openmpi/lib/libmpi_cxx.so /usr/lib/libmpi.so
else
    pkg-config --libs openPMD
    # -L${HOME}/somepath/lib -lopenPMD
fi

pkg-config --cflags openPMD
# -I${HOME}/somepath/include

Author Contributions

openPMD-api is developed by many people. It was initially started by the Computational Radiation Physics Group at HZDR as successor to libSplash, generalizing the successful HDF5 & ADIOS1 implementations in PIConGPU. The following people and institutions contributed to openPMD-api:

Further thanks go to improvements and contributions from:

Grants

The openPMD-api authors acknowledge support via the following programs. This project has received funding from the European Unions Horizon 2020 research and innovation programme under grant agreement No 654220. Supported by the Consortium for Advanced Modeling of Particles Accelerators (CAMPA), funded by the U.S. DOE Office of Science under Contract No. DE-AC02-05CH11231. Supported by the Exascale Computing Project (17-SC-20-SC), a collaborative effort of two U.S. Department of Energy organizations (Office of Science and the National Nuclear Security Administration).

Transitive Contributions

openPMD-api stands on the shoulders of giants and we are grateful for the following projects included as direct dependencies:

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Files for openPMD-api, version 0.11.1a1
Filename, size File type Python version Upload date Hashes
Filename, size openPMD_api-0.11.1a1-cp35-cp35m-macosx_10_9_x86_64.whl (4.6 MB) File type Wheel Python version cp35 Upload date Hashes View
Filename, size openPMD_api-0.11.1a1-cp35-cp35m-manylinux2010_x86_64.whl (5.4 MB) File type Wheel Python version cp35 Upload date Hashes View
Filename, size openPMD_api-0.11.1a1-cp36-cp36m-macosx_10_13_x86_64.whl (4.5 MB) File type Wheel Python version cp36 Upload date Hashes View
Filename, size openPMD_api-0.11.1a1-cp36-cp36m-macosx_10_15_x86_64.whl (4.6 MB) File type Wheel Python version cp36 Upload date Hashes View
Filename, size openPMD_api-0.11.1a1-cp36-cp36m-macosx_10_9_x86_64.whl (4.6 MB) File type Wheel Python version cp36 Upload date Hashes View
Filename, size openPMD_api-0.11.1a1-cp36-cp36m-manylinux2010_x86_64.whl (5.4 MB) File type Wheel Python version cp36 Upload date Hashes View
Filename, size openPMD_api-0.11.1a1-cp37-cp37m-macosx_10_13_x86_64.whl (4.5 MB) File type Wheel Python version cp37 Upload date Hashes View
Filename, size openPMD_api-0.11.1a1-cp37-cp37m-macosx_10_15_x86_64.whl (4.6 MB) File type Wheel Python version cp37 Upload date Hashes View
Filename, size openPMD_api-0.11.1a1-cp37-cp37m-macosx_10_9_x86_64.whl (4.6 MB) File type Wheel Python version cp37 Upload date Hashes View
Filename, size openPMD_api-0.11.1a1-cp37-cp37m-manylinux2010_x86_64.whl (5.4 MB) File type Wheel Python version cp37 Upload date Hashes View
Filename, size openPMD_api-0.11.1a1-cp38-cp38-macosx_10_13_x86_64.whl (4.6 MB) File type Wheel Python version cp38 Upload date Hashes View
Filename, size openPMD_api-0.11.1a1-cp38-cp38-macosx_10_15_x86_64.whl (4.7 MB) File type Wheel Python version cp38 Upload date Hashes View
Filename, size openPMD_api-0.11.1a1-cp38-cp38-macosx_10_9_x86_64.whl (4.7 MB) File type Wheel Python version cp38 Upload date Hashes View
Filename, size openPMD_api-0.11.1a1-cp38-cp38-manylinux2010_x86_64.whl (5.4 MB) File type Wheel Python version cp38 Upload date Hashes View
Filename, size openPMD_api-0.11.1a1-pp36-pypy36_pp73-macosx_10_13_x86_64.whl (4.6 MB) File type Wheel Python version pp36 Upload date Hashes View
Filename, size openPMD_api-0.11.1a1-pp36-pypy36_pp73-macosx_10_15_x86_64.whl (4.7 MB) File type Wheel Python version pp36 Upload date Hashes View
Filename, size openPMD_api-0.11.1a1-pp36-pypy36_pp73-macosx_10_9_x86_64.whl (4.7 MB) File type Wheel Python version pp36 Upload date Hashes View
Filename, size openPMD_api-0.11.1a1-pp36-pypy36_pp73-manylinux2010_x86_64.whl (5.4 MB) File type Wheel Python version pp36 Upload date Hashes View
Filename, size openPMD-api-0.11.1a1.tar.gz (613.0 kB) File type Source Python version None Upload date Hashes View

Supported by

Pingdom Pingdom Monitoring Google Google Object Storage and Download Analytics Sentry Sentry Error logging AWS AWS Cloud computing DataDog DataDog Monitoring Fastly Fastly CDN DigiCert DigiCert EV certificate StatusPage StatusPage Status page