CLIO Core

A Comprehensive Platform for Context Management in Scientific Computing

Overview · Installation · Components · Quickstart · Documentation · Contributing

Overview

CLIO Core is a unified framework that integrates multiple high-performance components for context management, data transfer, and scientific computing. CLIO Core enables developers to build efficient data processing pipelines for HPC, storage systems, and near-data computing applications.

It provides:

• High-performance context management for computational contexts and data transformations.
• Heterogeneous-aware I/O with multi-tiered, dynamic buffering.
• A modular runtime with dynamically loadable processing modules.
• Shared-memory data structures that work across host, CUDA, and ROCm.
• Distributed-by-construction scaling from single node to clusters.

Architecture

┌──────────────────────────────────────────────────────────────┐
│                      Applications                            │
│          (Scientific Workflows, HPC, Storage Systems)        │
└──────────────────────────────────────────────────────────────┘
                              │
        ┌─────────────────────┼─────────────────────┐
        │                     │                     │
┌───────────────┐   ┌──────────────────┐   ┌────────────────┐
│   Context     │   │    Context       │   │   Context      │
│  Exploration  │   │  Assimilation    │   │   Transfer     │
│    Engine     │   │     Engine       │   │    Engine      │
└───────────────┘   └──────────────────┘   └────────────────┘
        │                     │                     │
        └─────────────────────┼─────────────────────┘
                              │
                    ┌─────────────────┐
                    │  CLIO Runtime   │
                    │  (Module System)│
                    └─────────────────┘
                              │
                ┌─────────────────────────┐
                │  Context Transport      │
                │  Primitives             │
                │  (Shared Memory & IPC)  │
                └─────────────────────────┘

Installation

Pip (recommended)

The pip wheel ships a portable, self-contained build with all dependencies statically linked. No system installs are required beyond glibc 2.28+ (manylinux_2_28: RHEL/Rocky/AlmaLinux 8 and newer, Ubuntu 20.04+, Debian 11+) and Python 3.10+.

pip install iowarp-core

The wheel includes the CLIO runtime, the clio_run CLI, the CTE, CAE, and CEE engines, and the clio_cee Python bindings. A default configuration is seeded at ~/.clio/clio.yaml on first import.

Extra features

The pip wheel covers the common case. Build from source if you need any of:

• NVIDIA GPU (CUDA) or AMD GPU (ROCm) acceleration
• MPI for distributed multi-node deployment
• HDF5 / ADIOS2 adapters
• FUSE adapter
• Compression backends (LibPressio, Blosc, etc.)
• Custom ChiMods built against the C++ headers
• Sanitizer or debug builds

git clone --recurse-submodules https://github.com/iowarp/clio-core.git
cd clio-core

# Common presets: release, debug, cuda-release, rocm-release
cmake --preset=release
cmake --build build -j"$(nproc)"
sudo cmake --install build

For the per-feature apt / dnf dependency list and the complete CLIO_*_ENABLE_* flag checklist, see INSTALL.md. Other install methods (Conda, Docker, Spack) are documented in docs/getting-started/installation.

Components

Quickstart

Start the runtime

Installation seeds a default configuration at ~/.clio/clio.yaml, so the runtime works out of the box:

clio_run start          # foreground
clio_run start &        # background

To override the configuration, point CLIO_SERVER_CONF at your own YAML file:

export CLIO_SERVER_CONF=/path/to/my_config.yaml
clio_run start

FUSE filesystem

clio_cte_fuse exposes the Context Transfer Engine as a POSIX filesystem, so unmodified applications read and write IOWarp-managed data through ordinary file I/O. The wheel ships this binary on Linux and Windows; macOS wheels do not (macOS has no FUSE3 API).

1. Install the FUSE runtime. libfuse3 is a system dependency and is deliberately not bundled in the wheel, so install your distribution's package:

sudo apt install fuse3 libfuse3-3      # Debian / Ubuntu
sudo dnf install fuse3 fuse3-libs      # Fedora / RHEL

On Windows, install WinFsp instead. The clio_cte_fuse console script prepends WinFsp's bin directory to PATH itself, so winfsp-x64.dll resolves without a system-wide PATH edit.

2. Mount. Start the runtime, then point the daemon at a mountpoint. The daemon create-or-binds the filesystem pool and the CTE pool underneath it, so no separate compose step is required:

clio_run start &
mkdir -p ~/clio-mnt
CLIO_WITH_RUNTIME=0 clio_cte_fuse ~/clio-mnt -f

CLIO_WITH_RUNTIME=0 attaches to the runtime you just started rather than spawning an embedded one — without it the daemon brings up its own runtime and the data will not be visible to other clients. -f keeps the daemon in the foreground, which is the configuration CI exercises. Every remaining argument is handed to fuse_main, so standard libfuse options apply (-o allow_other, -d for protocol tracing). On Windows the mountpoint is a drive letter: clio_cte_fuse Z:.

The mount then behaves like any other filesystem:

cp big_input.h5 ~/clio-mnt/
python analysis.py ~/clio-mnt/big_input.h5

3. Unmount:

fusermount3 -u ~/clio-mnt      # or: fusermount -u ~/clio-mnt

To size the storage tiers explicitly instead of taking the defaults, compose a CTE pool before mounting — see context-transfer-engine/test/integration/fuse-manual/cte_compose.yaml for a working single-tier example:

clio_run compose start my_cte.yaml

Two semantics worth knowing: writes are write-through, so each write() reaches the chimod synchronously and the logical file size is always exact; and the kernel attribute and entry caches are disabled, so metadata that another client changed is never served stale.

Python: bundle, query, retrieve

import clio_cee as cee

ctx_interface = cee.ContextInterface()

# Assimilate inline strings into IOWarp storage.
# src="string::<blob_name>" names the blob; src_data carries the payload.
docs = [
    ("climate_report",   "Arctic sea ice extent fell to a record low in 2023."),
    ("ocean_temps",      "Ocean surface temperatures rose 0.3°C above the 20-year mean."),
    ("co2_levels",       "Atmospheric CO₂ reached 421 ppm, the highest in 800,000 years."),
]
bundle = [
    cee.AssimilationCtx(
        src=f"string::{name}",
        dst="iowarp::climate_docs",
        format="string",
        src_data=text,
    )
    for name, text in docs
]
ctx_interface.context_bundle(bundle)

# Query for blob names matching a regex.
blobs = ctx_interface.context_query("climate_docs", ".*")

# Query the top-2 most relevant blob names via BM25 semantic search.
blobs = ctx_interface.context_query("climate_docs", ".*",
                                    max_results=2,
                                    prompt="temperature anomaly over Arctic")

# Retrieve blob payloads (regex).
data = ctx_interface.context_retrieve("climate_docs", ".*")

# Retrieve the top-2 most relevant blobs via BM25 semantic search.
data = ctx_interface.context_retrieve("climate_docs", ".*",
                                      max_results=2,
                                      prompt="temperature anomaly over Arctic")

# Clean up.
ctx_interface.context_destroy(["climate_docs"])

C++ (CTE, direct)

For direct CTE put/get from C++, see the canonical example and operation reference in the Context Transfer Engine README.

Testing

cd build/release
ctest -VV                    # all unit tests
ctest -R context_transport   # CTP tests
ctest -R runtime             # runtime tests
ctest -R cte                 # CTE tests
ctest -R omni                # CAE tests
ctest -R context             # CEE tests

Benchmarking

CLIO Core ships two main benchmarks; pass --help to either for the full parameter list.

• clio_run_thrpt_bench — runtime task throughput and latency (bdev_io, bdev_allocation, bdev_task_alloc, latency test cases).
• clio_cte_bench — CTE Put / Get / PutGet throughput across threads, async depth, I/O size, and key-space cardinality.

Example:

clio_run_thrpt_bench --test-case bdev_io --threads 8 --duration 30
clio_cte_bench       --op PutGet --threads 8 --depth 16 --io-size 1m --io-count 200

Documentation

• AGENTS.md — unified development guide and coding standards.
• INSTALL.md — bare-metal source-build instructions.
• Context Transport Primitives
• CLIO Runtime
• Context Transfer Engine — canonical C++ CTE API reference.
• Context Assimilation Engine
• Context Exploration Engine
• Full documentation site: https://grc.iit.edu/docs/category/iowarp

Use Cases

Scientific computing: data processing pipelines, near-data computing, custom storage engines, workflows with context management.

Storage systems: distributed file system backends, object storage, multi-tiered caches, high-throughput I/O buffering.

HPC and data-intensive workloads: accelerated I/O, ingestion and transformation pipelines, heterogeneous computing with GPU support, real-time streaming analytics.

Performance Characteristics

• Task latency: < 10 µs for local task execution.
• Memory bandwidth: up to 50 GB/s with the RAM bdev backend.
• Scalability: single node to multi-node clusters.
• Concurrency: thousands of concurrent coroutine-based tasks.

Contributing

1. Fork the repository.
2. Create a feature branch: git checkout -b feature/amazing-feature.
3. Follow the standards in AGENTS.md.
4. ctest --test-dir build/release before opening a PR.
5. Submit a pull request against iowarp/clio-core.

License

CLIO Core is licensed under the BSD 3-Clause License. See LICENSE for the full text.

Copyright (c) 2024, Gnosis Research Center, Illinois Institute of Technology

Acknowledgements

CLIO Core is developed at the GRC lab at Illinois Institute of Technology as part of the IOWarp project, supported by the National Science Foundation (NSF) to advance next-generation scientific computing infrastructure.

• IOWarp project: https://grc.iit.edu/research/projects/iowarp
• IOWarp organization: https://github.com/iowarp
• Documentation hub: https://grc.iit.edu/docs/category/iowarp