iowarp-core 0.6.4

IOWarp Core: High-performance distributed I/O and task execution runtime

IOWarp Core

A Comprehensive Platform for Context Management in Scientific Computing

Overview · Components · Getting Started · Documentation · Contributing

---

Overview

IOWarp Core is a unified framework that integrates multiple high-performance components for context management, data transfer, and scientific computing. Built with a modular architecture, IOWarp Core enables developers to create efficient data processing pipelines for HPC, storage systems, and near-data computing applications.

IOWarp Core provides:

• High-Performance Context Management: Efficient handling of computational contexts and data transformations
• Heterogeneous-Aware I/O: Multi-tiered, dynamic buffering for accelerated data access
• Modular Runtime System: Extensible architecture with dynamically loadable processing modules
• Advanced Data Structures: Shared memory compatible containers with GPU support (CUDA, ROCm)
• Distributed Computing: Seamless scaling from single node to cluster deployments

Architecture

IOWarp Core follows a layered architecture integrating five core components:

┌──────────────────────────────────────────────────────────────┐
│                      Applications                            │
│          (Scientific Workflows, HPC, Storage Systems)        │
└──────────────────────────────────────────────────────────────┘
                              │
        ┌─────────────────────┼─────────────────────┐
        │                     │                     │
┌───────────────┐   ┌──────────────────┐   ┌────────────────┐
│   Context     │   │    Context       │   │   Context      │
│  Exploration  │   │  Assimilation    │   │   Transfer     │
│    Engine     │   │     Engine       │   │    Engine      │
└───────────────┘   └──────────────────┘   └────────────────┘
        │                     │                     │
        └─────────────────────┼─────────────────────┘
                              │
                    ┌─────────────────┐
                    │  Chimaera       │
                    │  Runtime        │
                    │  (ChiMod System)│
                    └─────────────────┘
                              │
                ┌─────────────────────────┐
                │  Context Transport      │
                │  Primitives             │
                │  (Shared Memory & IPC)  │
                └─────────────────────────┘

Components

IOWarp Core consists of five integrated components, each with its own specialized functionality:

1. Context Transport Primitives

Location: context-transport-primitives/

High-performance shared memory library containing data structures and synchronization primitives compatible with shared memory, CUDA, and ROCm.

Key Features:

• Shared memory compatible data structures (vector, list, unordered_map, queues)
• GPU-aware allocators (CUDA, ROCm)
• Thread synchronization primitives
• Networking layer with ZMQ transport
• Compression and encryption utilities

Read more →

2. Chimaera Runtime

Location: context-runtime/

High-performance modular runtime for scientific computing and storage systems with coroutine-based task execution.

Key Features:

• Ultra-high performance task execution (< 10μs latency)
• Modular ChiMod system for dynamic extensibility
• Coroutine-aware synchronization (CoMutex, CoRwLock)
• Distributed architecture with shared memory IPC
• Built-in storage backends (RAM, file-based, custom block devices)

Read more →

3. Context Transfer Engine

Location: context-transfer-engine/

Heterogeneous-aware, multi-tiered, dynamic I/O buffering system designed to accelerate I/O for HPC and data-intensive workloads.

Key Features:

• Programmable buffering across memory/storage tiers
• Multiple I/O pathway adapters
• Integration with HPC runtimes and workflows
• Improved throughput, latency, and predictability

Read more →

4. Context Assimilation Engine

Location: context-assimilation-engine/

High-performance data ingestion and processing engine for heterogeneous storage systems and scientific workflows.

Key Features:

• OMNI format for YAML-based job orchestration
• MPI-based parallel data processing
• Binary format handlers (Parquet, CSV, custom formats)
• Repository and storage backend abstraction
• Integrity verification with hash validation

Read more →

5. Context Exploration Engine

Location: context-exploration-engine/

Interactive tools and interfaces for exploring scientific data contents and metadata.

Key Features:

• Model Context Protocol (MCP) for HDF5 data
• HDF Compass viewer (wxPython-4 based)
• Interactive data exploration interfaces
• Metadata browsing capabilities

Read more →

Getting Started

Quick Start with uv (Fastest)

uv is a fast Python package installer and runner. Once IOWarp Core is published to PyPI:

# Install from PyPI (coming soon!)
uv pip install iowarp-core

# Or run the main tool directly without installation
uvx iowarp-core --help

# Run other tools (requires --from flag)
uvx --from iowarp-core wrp_start --help
uvx --from iowarp-core wrp_stop --help
uvx --from iowarp-core wrp_compose --help

After installation, all tools are available directly:

# Main entry point
iowarp-core --help

# User-friendly aliases (recommended)
wrp_start              # Start IOWarp runtime
wrp_stop               # Stop IOWarp runtime
wrp_compose            # Compose cluster configuration
wrp_refresh            # Refresh repository
wrp_cae                # CAE OMNI processor

# Original names (backwards compatible)
chimaera_start_runtime
chimaera_stop_runtime
chimaera_compose
chi_refresh_repo
wrp_cae_omni

Note: Build from source takes 10-30 minutes on first install (compiles C++ dependencies).

Quick Install with pip (Easiest)

The easiest way to install IOWarp Core is using pip. All dependencies are automatically built and installed into your Python environment - no system packages required!

Prerequisites: Only Python 3.8+ and a C++17 compiler

# Ubuntu/Debian
sudo apt-get update
sudo apt-get install -y build-essential python3-dev python3-pip

# macOS (Xcode command line tools)
xcode-select --install

Install IOWarp Core:

# Clone the repository (pip install requires local clone)
git clone https://github.com/iowarp/core.git
cd core

# Basic installation - builds and installs everything automatically
pip install .

# Or install in editable mode for development
pip install -e .

Customization with Environment Variables:

You can customize the pip installation using CMake environment variables:

# Enable tests and benchmarks
WRP_CORE_ENABLE_TESTS=ON WRP_CORE_ENABLE_BENCHMARKS=ON pip install .

# Enable MPI support (requires MPI to be installed)
WRP_CORE_ENABLE_MPI=ON pip install .

# Enable specific components only
WRP_CORE_ENABLE_CTE=ON WRP_CORE_ENABLE_CAE=OFF WRP_CORE_ENABLE_CEE=OFF pip install .

# Enable compression and encryption support
WRP_CORE_ENABLE_COMPRESS=ON WRP_CORE_ENABLE_ENCRYPT=ON pip install .

# Enable GPU support (CUDA or ROCm)
WRP_CORE_ENABLE_CUDA=ON pip install .
WRP_CORE_ENABLE_ROCM=ON pip install .

# Enable advanced networking (libfabric/Thallium)
WRP_CORE_ENABLE_LIBFABRIC=ON WRP_CORE_ENABLE_THALLIUM=ON pip install .

# Full customization example
WRP_CORE_ENABLE_TESTS=ON \
WRP_CORE_ENABLE_BENCHMARKS=ON \
WRP_CORE_ENABLE_MPI=ON \
WRP_CORE_ENABLE_COMPRESS=ON \
WRP_CORE_ENABLE_OPENMP=ON \
pip install .

Available CMake Options (for pip install):

Component Control:

• WRP_CORE_ENABLE_RUNTIME: Enable runtime component (default: ON)
• WRP_CORE_ENABLE_CTE: Enable Context Transfer Engine (default: ON)
• WRP_CORE_ENABLE_CAE: Enable Context Assimilation Engine (default: ON)
• WRP_CORE_ENABLE_CEE: Enable Context Exploration Engine (default: ON)

Build Features:

• WRP_CORE_ENABLE_TESTS: Enable tests (default: OFF)
• WRP_CORE_ENABLE_BENCHMARKS: Enable benchmarks (default: OFF)
• WRP_CORE_ENABLE_PYTHON: Enable Python bindings (default: OFF, automatically ON for pip)

Distributed Computing:

• WRP_CORE_ENABLE_MPI: Enable MPI support (default: OFF)
• WRP_CORE_ENABLE_ZMQ: Enable ZeroMQ transport (default: ON)
• WRP_CORE_ENABLE_LIBFABRIC: Enable libfabric transport (default: OFF)
• WRP_CORE_ENABLE_THALLIUM: Enable Thallium RPC (default: OFF)

Data Processing:

• WRP_CORE_ENABLE_CEREAL: Enable serialization (default: ON)
• WRP_CORE_ENABLE_COMPRESS: Enable compression libraries (default: OFF)
• WRP_CORE_ENABLE_ENCRYPT: Enable encryption (default: OFF)
• WRP_CORE_ENABLE_HDF5: Enable HDF5 support (default: ON)

Performance:

• WRP_CORE_ENABLE_OPENMP: Enable OpenMP (default: OFF)
• WRP_CORE_ENABLE_CUDA: Enable CUDA support (default: OFF)
• WRP_CORE_ENABLE_ROCM: Enable ROCm support (default: OFF)

Development/Debugging:

• WRP_CORE_ENABLE_ASAN: Enable AddressSanitizer (default: OFF)
• WRP_CORE_ENABLE_COVERAGE: Enable code coverage (default: OFF)
• WRP_CORE_ENABLE_DOXYGEN: Enable documentation checks (default: OFF)

Verify Installation:

import wrp_cte  # Context Transfer Engine
import wrp_cee  # Context Exploration Engine
print("IOWarp Core successfully installed!")

Note: First installation takes 10-15 minutes as dependencies build from source. Everything is installed to your Python environment - no manual environment variable configuration needed!

Alternative: Install Using install.sh

For system-wide installations or when you need more control over the build configuration, use the install.sh script:

Install IOWarp Core:

# Clone the repository
git clone https://github.com/iowarp/core.git
cd core

# Install to /usr/local (requires sudo for final install step)
./install.sh

# Or install to custom location (no sudo required)
INSTALL_PREFIX=$HOME/iowarp ./install.sh

Customization with Environment Variables:

The install.sh script accepts environment variables to customize the build:

# Install with tests and benchmarks enabled
WRP_CORE_ENABLE_TESTS=ON WRP_CORE_ENABLE_BENCHMARKS=ON ./install.sh

# Install with MPI support (checks for MPI installation first)
WRP_CORE_ENABLE_MPI=ON ./install.sh

# Install only dependencies (useful for development)
DEPS_ONLY=TRUE ./install.sh

# Custom install prefix with parallel build jobs
INSTALL_PREFIX=/opt/iowarp BUILD_JOBS=8 ./install.sh

# Full customization example
INSTALL_PREFIX=$HOME/iowarp \
WRP_CORE_ENABLE_TESTS=ON \
WRP_CORE_ENABLE_BENCHMARKS=ON \
WRP_CORE_ENABLE_MPI=ON \
BUILD_JOBS=16 \
./install.sh

install.sh-Specific Environment Variables:

• INSTALL_PREFIX: Installation directory (default: /usr/local)
• BUILD_JOBS: Number of parallel build jobs (default: $(nproc))
• DEPS_ONLY: Only build dependencies, skip IOWarp Core (default: FALSE)
• WRP_CORE_ENABLE_TESTS: Enable building tests (default: OFF)
• WRP_CORE_ENABLE_BENCHMARKS: Enable building benchmarks (default: OFF)
• WRP_CORE_ENABLE_MPI: Enable MPI support (default: OFF)

All CMake Options Also Work:

You can use ANY of the CMake options listed in the pip section above with install.sh:

# Enable compression and encryption
WRP_CORE_ENABLE_COMPRESS=ON WRP_CORE_ENABLE_ENCRYPT=ON ./install.sh

# Disable specific components
WRP_CORE_ENABLE_CAE=OFF WRP_CORE_ENABLE_CEE=OFF ./install.sh

# Enable CUDA support
WRP_CORE_ENABLE_CUDA=ON ./install.sh

# Enable debugging tools
WRP_CORE_ENABLE_ASAN=ON WRP_CORE_ENABLE_COVERAGE=ON ./install.sh

# Combined example with both install.sh and CMake options
INSTALL_PREFIX=$HOME/iowarp \
BUILD_JOBS=16 \
WRP_CORE_ENABLE_MPI=ON \
WRP_CORE_ENABLE_COMPRESS=ON \
WRP_CORE_ENABLE_CUDA=ON \
WRP_CORE_ENABLE_OPENMP=ON \
./install.sh

Set Environment Variables After Installation:

After installation, add these to your ~/.bashrc or ~/.zshrc:

export INSTALL_PREFIX=/usr/local  # Or your custom path
export CMAKE_PREFIX_PATH="$INSTALL_PREFIX:$CMAKE_PREFIX_PATH"
export LD_LIBRARY_PATH="$INSTALL_PREFIX/lib:$LD_LIBRARY_PATH"
export PKG_CONFIG_PATH="$INSTALL_PREFIX/lib/pkgconfig:$PKG_CONFIG_PATH"
export PYTHONPATH="$INSTALL_PREFIX/lib/python$(python3 -c 'import sys; print(".".join(map(str, sys.version_info[:2])))')/site-packages:$PYTHONPATH"

Note: First installation takes 10-15 minutes as dependencies build from source.

For detailed installation instructions and troubleshooting, see INSTALL.md or QUICKSTART.md.

Prerequisites

IOWarp Core requires the following dependencies:

Required Dependencies

These dependencies must be installed on your system:

Build Tools:

• C++17 compatible compiler (GCC >= 9, Clang >= 10)
• CMake >= 3.20
• pkg-config

Core Libraries:

• Boost >= 1.70 (components: context, fiber, system)
• libelf (ELF binary parsing for adapter functionality)
• ZeroMQ (libzmq) (distributed communication)
• yaml-cpp - YAML configuration library (git submodule in external/yaml-cpp)
• cereal - Serialization library (git submodule in external/cereal)
• Threads (POSIX threads library)

Compression Libraries (if HSHM_ENABLE_COMPRESS=ON):

• bzip2
• lzo2
• libzstd
• liblz4
• zlib
• liblzma
• libbrotli (libbrotlicommon, libbrotlidec, libbrotlienc)
• snappy
• blosc2

Encryption Libraries (if HSHM_ENABLE_ENCRYPT=ON):

• libcrypto (OpenSSL)

Optional Dependencies

These dependencies enable additional features:

Testing:

• Catch2 >= 3.0.1 (if WRP_CORE_ENABLE_TESTS=ON) - git submodule in external/Catch2

Documentation:

• Doxygen (if HSHM_ENABLE_DOXYGEN=ON)
• Perl (required by Doxygen)

Distributed Computing:

• MPI (MPICH, OpenMPI, or compatible) (if HSHM_ENABLE_MPI=ON)
• libfabric (high-performance networking) (if HSHM_ENABLE_LIBFABRIC=ON)
• Thallium (RPC framework) (if HSHM_ENABLE_THALLIUM=ON)

Parallel Computing:

• OpenMP (if HSHM_ENABLE_OPENMP=ON)

GPU Support:

• CUDA Toolkit >= 11.0 (if HSHM_ENABLE_CUDA=ON)
• ROCm/HIP >= 4.0 (if HSHM_ENABLE_ROCM=ON)

Context Assimilation Engine (CAE):

• HDF5 with C components (if CAE_ENABLE_HDF5=ON, default: ON)
• POCO (Net, NetSSL, Crypto, JSON components) (if CAE_ENABLE_GLOBUS=ON)
• nlohmann_json (if CAE_ENABLE_GLOBUS=ON)

Python Bindings:

• Python 3 with development headers (if WRP_CORE_ENABLE_PYTHON=ON)
• nanobind - Python bindings library (git submodule in external/nanobind)

Installation Commands

Ubuntu/Debian:

# Required dependencies
sudo apt-get update
sudo apt-get install -y \
  build-essential cmake pkg-config \
  libboost-context-dev libboost-fiber-dev libboost-system-dev \
  libelf-dev libzmq3-dev

# Optional: Compression libraries
sudo apt-get install -y \
  libbz2-dev liblzo2-dev libzstd-dev liblz4-dev \
  zlib1g-dev liblzma-dev libbrotli-dev libsnappy-dev libblosc2-dev

# Optional: HDF5 support (for CAE)
sudo apt-get install -y libhdf5-dev

# Optional: MPI support
sudo apt-get install -y libmpich-dev

# Optional: Testing framework (git submodule, no need to install separately)

# Optional: YAML library (git submodule, no need to install separately)

# Optional: Serialization library (git submodule, no need to install separately)

Docker Container (Recommended): All dependencies are pre-installed in our Docker container:

docker pull iowarp/iowarp-build:latest

Quick Installation

# Clone the repository
git clone https://github.com/iowarp/iowarp-core.git
cd iowarp-core

# Configure with CMake preset (debug mode)
cmake --preset=debug

# Build all components
cmake --build build --parallel $(nproc)

# Install to system or custom prefix
cmake --install build --prefix /usr/local

Component Build Options

The unified build system provides options to enable/disable components:

cmake --preset=debug \
  -DWRP_CORE_ENABLE_RUNTIME=ON \
  -DWRP_CORE_ENABLE_CTE=ON \
  -DWRP_CORE_ENABLE_CAE=ON \
  -DWRP_CORE_ENABLE_CEE=ON

Available Options:

• WRP_CORE_ENABLE_RUNTIME: Enable runtime component (default: ON)
• WRP_CORE_ENABLE_CTE: Enable context-transfer-engine (default: ON)
• WRP_CORE_ENABLE_CAE: Enable context-assimilation-engine (default: ON)
• WRP_CORE_ENABLE_CEE: Enable context-exploration-engine (default: ON)

Quick Start Example

Here's a simple example using the Chimaera runtime with the bdev ChiMod:

#include <chimaera/chimaera.h>
#include <chimaera/bdev/bdev_client.h>
#include <chimaera/admin/admin_client.h>

int main() {
  // Initialize Chimaera (client mode with embedded runtime)
  chi::CHIMAERA_INIT(chi::ChimaeraMode::kClient, true);

  // Create admin client (always required)
  chimaera::admin::Client admin_client(chi::PoolId(7000, 0));
  admin_client.Create(HSHM_MCTX, chi::PoolQuery::Local());

  // Create bdev client for high-speed RAM storage
  chimaera::bdev::Client bdev_client(chi::PoolId(8000, 0));
  bdev_client.Create(HSHM_MCTX, chi::PoolQuery::Local(),
                    chimaera::bdev::BdevType::kRam, "", 1024*1024*1024); // 1GB RAM

  // Allocate and use a block
  auto block = bdev_client.Allocate(HSHM_MCTX, 4096);  // 4KB block
  std::vector<hshm::u8> data(4096, 0xAB);
  bdev_client.Write(HSHM_MCTX, block, data);
  auto read_data = bdev_client.Read(HSHM_MCTX, block);
  bdev_client.Free(HSHM_MCTX, block);

  return 0;
}

Build and Link:

# Unified package includes everything - HermesShm, Chimaera, and all ChiMods
find_package(iowarp-core REQUIRED)

target_link_libraries(my_app
  chimaera::admin_client  # Admin ChiMod (always available)
  chimaera::bdev_client   # Block device ChiMod (always available)
  # Optional: Add hshm modular targets if needed
  # hshm::configure    # For YAML configuration
  # hshm::serialize    # For object serialization
  # hshm::mpi          # For MPI support
)

What find_package(iowarp-core) provides:

Core Components:

• All hshm::* modular targets (cxx, configure, serialize, interceptor, lightbeam, thread_all, mpi, compress, encrypt)
• chimaera::cxx (core runtime library)
• ChiMod build utilities

Core ChiMods (Always Available):

• chimaera::admin_client, chimaera::admin_runtime
• chimaera::bdev_client, chimaera::bdev_runtime

Optional ChiMods (if enabled at build time):

• wrp_cte::core_client, wrp_cte::core_runtime (Context Transfer Engine)
• wrp_cae::core_client, wrp_cae::core_runtime (Context Assimilation Engine)

Testing

IOWarp Core includes comprehensive test suites for each component:

# Run all unit tests
cd build
ctest -VV

# Run specific component tests
ctest -R context_transport  # Transport primitives tests
ctest -R chimaera           # Runtime tests
ctest -R cte                # Context transfer engine tests
ctest -R omni               # Context assimilation engine tests

Documentation

Comprehensive documentation is available for each component:

• CLAUDE.md: Unified development guide and coding standards
• Context Transport Primitives: Shared memory data structures
• Chimaera Runtime: Modular runtime system and ChiMod development
  • MODULE_DEVELOPMENT_GUIDE.md: Complete ChiMod development guide
• Context Transfer Engine: I/O buffering and acceleration
  • CTE API Documentation: Complete API reference
• Context Assimilation Engine: Data ingestion and processing
• Context Exploration Engine: Interactive data exploration

Docker Deployment

IOWarp Core can be deployed using Docker containers for distributed deployments:

# Build and start 3-node cluster
cd docker
docker-compose up -d

# Check status
docker-compose ps

# View logs
docker-compose logs -f iowarp-node1

# Stop cluster
docker-compose down

See CLAUDE.md for detailed Docker deployment configuration.

Use Cases

Scientific Computing:

• High-performance data processing pipelines
• Near-data computing for large datasets
• Custom storage engine development
• Computational workflows with context management

Storage Systems:

• Distributed file system backends
• Object storage implementations
• Multi-tiered cache and storage solutions
• High-throughput I/O buffering

HPC and Data-Intensive Workloads:

• Accelerated I/O for scientific applications
• Data ingestion and transformation pipelines
• Heterogeneous computing with GPU support
• Real-time streaming analytics

Performance Characteristics

IOWarp Core is designed for high-performance computing scenarios:

• Task Latency: < 10 microseconds for local task execution (Chimaera Runtime)
• Memory Bandwidth: Up to 50 GB/s with RAM-based storage backends
• Scalability: Single node to multi-node cluster deployments
• Concurrency: Thousands of concurrent coroutine-based tasks
• I/O Performance: Native async I/O with multi-tiered buffering

Contributing

We welcome contributions to the IOWarp Core project!

Development Workflow

1. Fork the repository
2. Create a feature branch: git checkout -b feature/amazing-feature
3. Follow the coding standards in CLAUDE.md
4. Test your changes: ctest --test-dir build
5. Submit a pull request

Coding Standards

• Follow Google C++ Style Guide
• Use semantic naming for IDs and priorities
• Always create docstrings for new functions (Doxygen compatible)
• Add comprehensive unit tests for new functionality
• Never use mock/stub code unless explicitly required - implement real, working code

See CLAUDE.md for complete coding standards and workflow guidelines.

License

IOWarp Core is licensed under the BSD 3-Clause License. See LICENSE file for complete license text.

Copyright (c) 2024, Gnosis Research Center, Illinois Institute of Technology

---

Acknowledgements

IOWarp Core is developed at the GRC lab at Illinois Institute of Technology as part of the IOWarp project. This work is supported by the National Science Foundation (NSF) and aims to advance next-generation scientific computing infrastructure.

For more information:

• IOWarp Project: https://grc.iit.edu/research/projects/iowarp
• IOWarp Organization: https://github.com/iowarp
• Documentation Hub: https://grc.iit.edu/docs/category/iowarp

---

Built with ❤️ by the GRC Lab at Illinois Institute of Technology