CAPIO-CL — Cross-Application Programmable I/O Coordination Language

Platform support

OS / Arch
	YES	YES	YES
	No CI/CD support	YES	N.A.

Documentation

• 
• 
• 

CAPIO-CL is a novel I/O coordination language that enables users to annotate file-based workflow data dependencies with synchronization semantics for files and directories. Designed to facilitate transparent overlap between computation and I/O operations, CAPIO-CL allows multiple producer–consumer application modules to coordinate efficiently using a JSON-based syntax.

For detailed documentation and examples, please visit:

---

Overview

The CAPIO Coordination Language (CAPIO-CL) allows applications to declare:

• Data objects, I/O dependencies, and access modes
• Synchronization semantics across different processes
• Commit policies for I/O objects

At runtime, CAPIO-CL’s parser and engine components analyze, track, and manage these declared relationships, enabling * transparent data sharing* and cross-application optimizations.

---

Building

Requirements & dependencies

• C++17 or greater
• Cmake 3.15 or newer
• Python3 to bundle CAPIO-CL json schemas into target binaries
• danielaparker/jsoncons to parse, serialize and validate CAPIO-CL JSON config files
• GoogleTest for automated testing
• pybind11 when building python wheels
• CALF for both logs and CLI messages

jsoncons, GoogleTest and pybind11 are fetched automatically by CMake — no manual setup required.

Steps

Clone
git clone https://github.com/High-Performance-IO/CAPIO-CL.git


mkdir -p CAPIO-CL/build && cd CAPIO-CL/build
cmake ..
make 

By default, this will:

• Build the "libcapio_cl" static library
• Build the "CAPIO_CL_tests" executable (GoogleTest-based)
• Build the "py_capio_cl" python bindings (pybind11)

---

Integration as a Subproject

CAPIO-CL can be included directly into another CMake project using:

include(FetchContent)

#####################################
# External projects
#####################################
FetchContent_Declare(
        capio_cl
        GIT_REPOSITORY https://github.com/High-Performance-IO/CAPIO-CL.git
        GIT_TAG main
)
FetchContent_MakeAvailable(capio_cl)

#####################################
# Include files and directories
#####################################
target_include_directories(${TARGET_NAME} PRIVATE
        ${capio_cl_SOURCE_DIR}
)

#####################################
# Link libraries
#####################################
target_link_libraries(${PROJECT_NAME} PRIVATE
        libcapio_cl
)

When included this way, unit tests and python bindings are not built, keeping integration clean for external projects.

---

Python Bindings

CAPIO-CL now provides native Python bindings built using pybind11.
These bindings expose the core C++ APIs (Engine, Parser and Serializer), directly to Python, allowing the CAPIO-CL logic to be used within python projects.

Install from PyPI

CAPIO-CL is available on PyPI! Simply run

pip install py_capio_cl

Building the Bindings

You can build and install the Python bindings directly from the CAPIO-CL source tree using:

pip install --upgrade pip
pip install -r build-requirements.txt
python -m build
pip install dst/*.whl

This will build the Python wheel and install it into your current environment using an ad-hoc build environment, which is downloaded, installed, and configured in isolation. A faster way to build and install CAPIO-CL is to use native system packages and then run from within the CAPIO-CL root directory:

pip install .

This assumes that all build dependencies not fetched by cmake are available.

---

Runtime TOML Configuration

Runtime behavior is configured with a TOML file loaded into CapioClConfiguration. This is separate from the JSON coordination-language document: the TOML file selects the JSON document, monitor backends, metadata storage, and dynamic API settings.

Complete example

# Workflow and optional JSON coordination-language document
[capiocl]
workflow_name = "my-workflow"
config_path = "workflow.json"
resolve_path = "/data/run-42"
store_all_in_memory = false

[capiocl.dynamic_api]
enabled = false
ip = "224.224.224.3"
port = 11223

[capiocl.monitor.filesystem]
enabled = true
metadata_dir = "/shared/trusted/run-42"

[capiocl.monitor.mcast]
enabled = false
delay_ms = 300

[capiocl.monitor.mcast.commit]
ip = "224.224.224.1"
port = 12345

[capiocl.monitor.mcast.homenode]
ip = "224.224.224.2"
port = 12345

Options

Key	Type	Default	Description
capiocl.workflow_name	string	JSON name, or CAPIO without JSON	Overrides the workflow name.
capiocl.config_path	string/path	empty	JSON CAPIO-CL document to parse. An empty value creates a runtime-only engine.
capiocl.resolve_path	string/path	empty	Prefix applied to relative paths in the JSON document.
capiocl.store_all_in_memory	boolean	false	Marks every parsed data path for in-memory storage.
capiocl.dynamic_api.enabled	boolean	false	Starts the dynamic configuration API.
capiocl.dynamic_api.ip	string	224.224.224.3	Multicast address used by the dynamic API.
capiocl.dynamic_api.port	integer	11223	UDP port used by the dynamic API.
capiocl.monitor.filesystem.enabled	boolean	see below	Enables filesystem commit and home-node tokens.
capiocl.monitor.filesystem.metadata_dir	string/path	empty	Trusted metadata root for persistent counted ON_CLOSE state.
capiocl.monitor.mcast.enabled	boolean	see below	Enables multicast commit and home-node propagation.
capiocl.monitor.mcast.delay_ms	integer	300	Delay before multicast status operations, in milliseconds.
capiocl.monitor.mcast.commit.ip	string	224.224.224.1	Multicast group for commit state.
capiocl.monitor.mcast.commit.port	integer	12345	UDP port for commit state.
capiocl.monitor.mcast.homenode.ip	string	224.224.224.2	Multicast group for home-node state.
capiocl.monitor.mcast.homenode.port	integer	12345	UDP port for home-node state.

Every CAPIO-CL option is under the top-level capiocl table. Other top-level tables may coexist in the same TOML file and are ignored by CAPIO-CL. When parsing a user-provided configuration, omitted capiocl.monitor.filesystem.enabled and capiocl.monitor.mcast.enabled values are false. Engine() and CapioClConfiguration.loadDefaults() use the built-in configuration, which enables both monitors. Set both values explicitly in deployed TOML files to avoid ambiguity.

TOML booleans must be unquoted true or false, and ports/delays must be integers. Relative capiocl.config_path, capiocl.resolve_path, and capiocl.monitor.filesystem.metadata_dir values are interpreted from the process working directory. Unknown keys are retained but ignored by CAPIO-CL.

Filesystem metadata

capiocl.monitor.filesystem.metadata_dir is required only when an ON_CLOSE rule commits after more than one close. It must name a trusted, non-attacker-writable directory unique to the workflow run. CAPIO-CL creates and owns a capiocl subdirectory beneath it. Multi-process and multi-node producers must share that directory through storage providing coherent atomic exclusive file creation, rename, and unlink. Ordinary commit and home-node token operations do not require this option.

Loading configuration

#include "capiocl/configuration.h"
#include "capiocl/parser.h"

using capiocl::configuration::CapioClConfiguration;

CapioClConfiguration config;
config.load("runtime.toml");
std::unique_ptr<capiocl::engine::Engine> engine(capiocl::parser::Parser::parse(config));

import py_capio_cl

config = py_capio_cl.CapioClConfiguration()
config.load("runtime.toml")
engine = py_capio_cl.Parser.parse(config)

Configuration can also be constructed from a string map/dictionary. Values in that form must use their flattened keys and string representations, for example {"capiocl.monitor.filesystem.enabled": "true"}.

---

API Snapshot

A simplified example of CAPIO-CL usage in C++:

#include "capiocl.hpp"

int main() {
    capiocl::Engine engine;
    engine.newFile("Hello_World.txt")
    engine.print();
    
    // Dump engine to configuration file
    capiocl::Serializer::dump(engine, "my_workflow", "my_workflow.json")
    return 0;
}

The py_capio_cl module provides access to CAPIO-CL’s core functionality through a high-level Python interface.

from py_capio_cl import Engine, Serializer

engine = Engine()
engine.newFile("Hello_World.txt")
engine.print()

# Dump engine to configuration file
Serializer.dump(engine, "my_workflow", "my_workflow.json")

Notes

• All GET endpoints expect a JSON body containing the targeted file path.
• The API is intended for local control and orchestration, not public exposure.

---

Developing team

Name	Role	Contact
Marco Edoardo Santimaria	Designer and Maintainer	email | Homepage
Iacopo Colonnelli	Workflows Expert and Designer	email | Homepage
Massimo Torquati	Designer	email | Homepage
Marco Aldinucci	Designer	email | Homepage

Former Members

Name	Role	Contact
Alberto Riccardo Martinelli	Designer	email | Homepage