iops-benchmark 3.3.1

A generic benchmark orchestration framework for automated parametric experiments

IOPS

A generic benchmark orchestration framework for automated parametric experiments.

IOPS automates the generation, execution, and analysis of benchmark experiments. Instead of writing custom scripts for each benchmark study, you define a YAML configuration describing what to vary, what to run, and what to measure—IOPS handles the rest.

What is IOPS?

IOPS is a framework that transforms benchmark experiments from manual scripting into automated, reproducible workflows.

Without IOPS: Write bash scripts → Parse outputs → Aggregate data → Generate plots → Repeat for each parameter change

With IOPS: Write one YAML config → Run iops run config.yaml → Get interactive HTML reports

Originally designed for I/O performance studies (see our 2022 paper), IOPS has evolved into a generic framework for any parametric benchmark workflow.

Key Features

• Parameter Sweeping: Automatically generate and execute tests for all parameter combinations
• Multiple Search Strategies: Exhaustive, Bayesian optimization, or random sampling
• Execution Backends: Run locally or submit to SLURM clusters
• Smart Caching: Skip redundant tests with parameter-aware result caching
• Budget Control: Set core-hour limits to avoid exceeding compute allocations
• Automatic Reports: Generate interactive HTML reports with plots and statistical analysis
• Flexible Output: Export results to CSV, Parquet, or SQLite

Installation

Prerequisites

• Python 3.10 or later
• For benchmark execution: Required tools in PATH (e.g., ior, mpirun for I/O benchmarks)
• For SLURM clusters: Access to a SLURM scheduler

Quick Installation (from PyPI)

Install IOPS directly from PyPI:

pip install iops-benchmark

Installation with Spack (for HPC environments)

Spack is a package manager designed for HPC systems. To install IOPS with Spack:

# Add the IOPS Spack repository
spack repo add https://gitlab.inria.fr/lgouveia/iops-spack.git

# Install IOPS
spack install iops-benchmark

# Load the module
spack load iops-benchmark

# Verify installation
iops --version

Basic Installation (from source)

# Clone the repository
git clone https://gitlab.inria.fr/lgouveia/iops.git
cd iops

# Install the package with dependencies
pip install .

# Verify installation
iops --version

Development Installation

For development work, install in editable mode:

# Clone the repository
git clone https://gitlab.inria.fr/lgouveia/iops.git
cd iops

# Install in editable mode
pip install -e .

# Verify installation
iops --version

Using a Virtual Environment (Recommended)

Using a virtual environment keeps IOPS dependencies isolated from your system Python:

Option 1: Python venv

# Create virtual environment
python3 -m venv iops_env

# Activate it
source iops_env/bin/activate  # On Linux/Mac

# Install IOPS (from source)
pip install .

# Or for development
pip install -e .

# Verify installation
iops --version

Option 2: Conda

# Create conda environment
conda create -n iops python=3.10
conda activate iops

# Install IOPS (from source)
pip install .

# Or for development
pip install -e .

# Verify installation
iops --version

Quick Start

1. Create a Configuration

Generate a comprehensive YAML template with all options documented:

iops generate my_config.yaml

This creates a fully-commented template showing all available configuration options. Customize it for your needs.

Or start from an example:

cp docs/examples/ior/local/ior_simple.yaml my_config.yaml

2. Preview Your Benchmark

# Dry-run to see what will be executed
iops run my_config.yaml --dry-run

# Check configuration validity
iops check my_config.yaml

3. Run the Benchmark

# Basic execution
iops run my_config.yaml

# With caching (skip already-executed tests)
iops run my_config.yaml --use-cache

# With budget limit (SLURM only)
iops run my_config.yaml --max-core-hours 1000

# With verbose logging
iops run my_config.yaml --log-level DEBUG

4. Explore Executions

# List all executions with their parameters
iops find /path/to/workdir/run_001

# Filter executions by variable values
iops find /path/to/workdir/run_001 nodes=4 ppn=8

# Show details for specific execution
iops find /path/to/workdir/run_001/exec_042

5. Generate Analysis Report

# Generate HTML report with interactive plots
iops report /path/to/workdir/run_001

How It Works

IOPS follows a simple workflow:

1. Configuration: Define variables to sweep, commands to run, and metrics to measure in a YAML file
2. Planning: IOPS generates execution instances for parameter combinations
3. Execution: Runs tests locally or submits SLURM jobs
4. Parsing: Extracts metrics from output files using your parser script
5. Storage: Saves results to CSV, SQLite, or Parquet
6. Analysis: Generates HTML reports with interactive plots and statistics

Core Concepts

Variables: Parameters you want to vary

vars:
  nodes:
    type: int
    sweep:
      mode: list
      values: [4, 8, 16, 32]

Commands: What to execute (supports Jinja2 templating)

command:
  template: "ior -w -b {{ block_size }}mb -o {{ output_file }}"

Metrics: What to measure

metrics:
  - name: bandwidth_mbps
  - name: latency_ms

Search Methods:

• exhaustive: Test all combinations (thorough, complete)
• bayesian: Gaussian Process optimization (efficient, finds optima faster)
• random: Random sampling (useful for statistical analysis)

Example Configuration

benchmark:
  name: "My Benchmark Study"
  workdir: "./workdir"
  executor: "local"  # or "slurm" for clusters
  search_method: "exhaustive"
  repetitions: 3

vars:
  threads:
    type: int
    sweep:
      mode: list
      values: [1, 2, 4, 8]

  buffer_size:
    type: int
    sweep:
      mode: list
      values: [4, 16, 64]

command:
  template: "my_benchmark --threads {{ threads }} --buffer {{ buffer_size }}"

scripts:
  - name: "benchmark"
    parser:
      file: "{{ execution_dir }}/output.json"
      metrics:
        - name: throughput
      parser_script: scripts/parse_results.py

output:
  sink:
    type: csv
    path: "{{ workdir }}/results.csv"

SLURM Integration

IOPS provides native SLURM cluster support with automatic job submission, monitoring, and budget tracking:

benchmark:
  executor: "slurm"
  max_core_hours: 1000
  cores_expr: "{{ nodes * processes_per_node }}"

scripts:
  - name: "benchmark"
    submit: "sbatch"
    script_template: |
      #!/bin/bash
      #SBATCH --nodes={{ nodes }}
      #SBATCH --ntasks-per-node={{ processes_per_node }}
      #SBATCH --time=01:00:00

      module load mpi/openmpi
      {{ command.template }}

Features:

• Automatic job submission and status monitoring
• Core-hours budget tracking and enforcement
• Multi-node resource allocation
• Graceful handling of job failures

Advanced Features

Result Caching

IOPS caches execution results to avoid redundant tests. Enable caching by specifying a SQLite database in your config:

benchmark:
  cache_file: "/path/to/cache.db"

Then use --use-cache to skip tests with identical parameters:

iops run config.yaml --use-cache

Multi-Round Execution

Run experiments in stages with the rounds feature:

rounds:
  - name: "explore"
    sweep_vars: ["nodes"]
    repetitions: 1

  - name: "validate"
    sweep_vars: ["nodes", "processes_per_node"]
    repetitions: 5

Best results from each round propagate to the next.

Budget Control

Prevent exceeding compute allocations:

# Set budget limit from command line
iops run config.yaml --max-core-hours 1000

# Or in YAML config
benchmark:
  max_core_hours: 500
  cores_expr: "{{ nodes * ppn }}"

Documentation

For comprehensive documentation, examples, and tutorials, visit:

https://lgouveia.gitlabpages.inria.fr/iops/

The documentation includes:

• Complete YAML configuration reference
• User guides for all features
• Working examples for various scenarios
• Best practices and optimization tips

Examples

Check docs/examples/ for working configuration examples:

• ior/local/ior_simple.yaml - Basic local IOR benchmark
• ior/local/ior_bayesian.yaml - IOR with Bayesian optimization
• ior/slurm/ior_simple_slurm.yaml - SLURM cluster deployment
• ior/slurm/ior_bayesian_slurm.yaml - Cluster with Bayesian search
• mdtest/local/mdtest_local.yaml - Metadata operations benchmark

Command Reference

IOPS uses a subcommand-based CLI similar to git and docker:

# Run benchmark
iops run <config.yaml> [options]
  --dry-run              Preview without executing
  --use-cache            Skip cached tests
  --max-core-hours N     Budget limit (SLURM)
  --log-level LEVEL      Verbosity (DEBUG, INFO, WARNING)
  --no-log-terminal      Disable terminal logging (log to file only)

# Validate configuration
iops check <config.yaml>

# Find and explore executions
iops find <path> [filters...]
iops find <path> nodes=4 ppn=8    # Filter by parameters
iops find <path> --show-command   # Show command column

# Generate analysis report
iops report <workdir/run_NNN> [--report-config report.yaml]

# Generate configuration template
iops generate [output.yaml]

# Show version
iops --version

# Show help
iops --help
iops run --help      # Subcommand-specific help

License

This project is developed at Inria. See LICENSE file for details.