CompilerSutraPerfTool

Alpha performance experimentation for C/C++ programs and GPU kernels (OpenCL, HIP, Vulkan shaders).

Compile, run, profile, compare, and export structured JSON/CSV/XLSX artifacts — from the CLI or pip install compilersutra-perf.

Status: Alpha MVP (0.2.0). Suitable for local experimentation; not yet validated for production CI regression gates. See limitations and docs/METHODOLOGY.md.

Research goal: a unified CPU/GPU hardware & compiler observatory layer — orchestrate existing toolchains, normalize results, and keep static vs measured evidence honest. See docs/OBSERVATORY.md.

Install

pip install compilersutra-perf

Optional extras:

pip install 'compilersutra-perf[visualize]'   # HTML reports + Streamlit dashboard
pip install 'compilersutra-perf[excel]'       # XLSX result workbooks
pip install 'compilersutra-perf[pdf]'         # PDF reports (WeasyPrint)
pip install 'compilersutra-perf[dev]'         # pytest

Requirements: Python 3.11+. Linux wheels ship prebuilt native runners for CPU (and Vulkan, where available), so CPU workloads run out of the box. Other platforms (or exotic Python builds) fall back to building the runner on demand, which needs a C/C++ toolchain and cmake on PATH; perf is also recommended for full CPU profiling on Linux. GPU backends need their respective runtimes (ROCm, OpenCL, Vulkan) installed separately.

Platform wheels and old distributions: release wheels use a manylinux tag (e.g. manylinux_2_39_x86_64) for the glibc they were built on, so a wheel built on Ubuntu 24.04 may fail to load its bundled runner on Ubuntu 22.04 or older (newer glibc). The loader probes the bundled runner at startup and automatically falls back to an on-demand local build, so only an older toolchain is needed there. Hardware counter events are probed per-CPU, so vendor-specific events never break collection on other vendors (Intel vs AMD).

Native Windows: not yet supported. The runner uses POSIX APIs (fork, sched_setaffinity) and won't build on Windows natively — use WSL2. Result artifacts (JSON/CSV/XLSX/HTML reports) open fine on Windows.

Check that every system tool csperf needs is present, and install whatever is missing:

csperf quickstart --output-dir results/quickstart   # doctor + example + report
csperf doctor              # report missing tools; exit 1 if a required tool is missing
csperf doctor --install    # install missing required tools (apt/dnf/pacman/apk/brew)
csperf doctor --install --all   # also install optional GPU/profiling tools

See docs/TROUBLESHOOTING.md for perf, energy, and GPU fixes.

Develop from source (private checkout if you have access):

cd CompilerSutraPerfTool
python3 -m venv .venv && source .venv/bin/activate
pip install -e '.[dev]'
# Native runners are compiled on demand at first run; build them explicitly with:
cmake -S native/runtime -B build/native && cmake --build build/native

To produce a platform wheel that embeds prebuilt runners (used by the release pipeline):

CSPERF_BUNDLE_NATIVE=1 python -m build --wheel

Quick start

csperf list-backends
csperf quickstart --output-dir results/quickstart
csperf list-energy-backends
csperf run --input examples/cpp/matrix_traversal.cpp --backend cpu \
  --warmup-runs 1 --repeat-runs 3 --output results/cpu.json
csperf profile results/cpu.json
csperf diff results/run-a.json results/run-b.json --csv results/compare.csv
# Auto-compare -O0 / -O1 / -O2 / -O3 runtimes (writes results/optimize/ + comparison report):
csperf run --input examples/cpp/matrix_traversal.cpp --diff-optimize --no-perf --report-format both
# Energy (Linux RAPL / AMD GPU when available):
csperf run --input examples/cpp/matrix_traversal.cpp --backend cpu \
  --energy-backend amd-cpu --output results/cpu-energy.json

Each run writes results/<name>.json, .csv, and .xlsx (when openpyxl is installed).

Backends

Compiler selection and flags

Use --compiler to select the compiler explicitly, or select the toolchain with environment variables; pass optimization and language flags with --compiler-flag.

Choose compiler (CPU)

# Default: clang / clang++
csperf run --input program.cpp --backend cpu --output results/clang.json

# GCC
CC=gcc CXX=g++ csperf run --input program.cpp --backend cpu --output results/gcc.json

# Explicit compiler override env vars (highest priority after CXX_COMPILER)
CXX=g++-13 csperf run --input program.cpp --backend cpu --output results/gpp13.json

Resolution order: CXX_COMPILER / C_COMPILER → CXX / CC → default (clang++ / clang).

HIP always uses hipcc when available.

Add compiler flags

Repeat --compiler-flag for each option. -O3 is added by default unless you pass another -O* flag (for example --compiler-flag=-O0):

csperf run --input program.cpp --backend cpu \
  --compiler-flag=-march=native \
  --compiler-flag=-funroll-loops \
  --compiler-flag=-fopenmp \
  --output results/optimized.json

csperf run --input program.cpp --compiler-flag=-std=c++20 --output results/cpp20.json

CXXFLAGS / CFLAGS are not read — pass flags explicitly.

Compare two compilers (batch)

For a folder of .c/.cpp files, use the batch script with JSON compiler configs:

python scripts/compiler_diff_batch.py examples/cpp \
  --config1 configs/compiler_gcc.sample.json \
  --config2 configs/compiler_clang.sample.json

Sample config (configs/compiler_gcc.sample.json):

{
  "compiler_family": "gcc",
  "cc": "gcc",
  "cxx": "g++",
  "compiler_flags": ["-O3", "-march=native"]
}

Outputs: per-file JSON/CSV diffs, summary.csv, and summary.xlsx.

Compare two saved runs

csperf diff results/gcc.json results/clang.json \
  --csv results/gcc-vs-clang.csv \
  --derived-config configs/derived_metrics.sample.json

Common run options

OpenCL kernel arguments:

csperf run --input examples/opencl/saxpy.cl --backend opencl --device-index 0 \
  --kernel-name saxpy \
  --kernel-arg buffer:float:read:4096:1.0 \
  --kernel-arg scalar:uint32:4096 \
  --readback-arg 2

Inspect planned commands:

csperf run --input program.cpp --plan-only

Results and visualization

Artifacts per run:

• JSON — full structured result (metrics, execution with host_platform and metrics_availability, hardware, diagnostics)
• CSV — sectioned spreadsheet-friendly report with dotted metric paths and units
• XLSX — Results + Metadata sheets (requires [excel] extra)

csperf profile results/cpu.json
csperf visualize results/cpu.json results/gpu.json --output report.html
csperf visualize results/cpu.json --format pdf --output report.pdf
csperf visualize results/cpu.json --format both --output report
csperf list-report-formats
pip install 'compilersutra-perf[visualize]' && csperf dashboard results/cpu.json

Energy and power domains appear in the HTML/PDF report when the run collected a power block. See docs/ENERGY.md and docs/REPORT.md.

Failed runs exit non-zero (1) and include actionable hints when tooling is missing. See exit codes in USAGE.

Limitations

Read these before trusting numbers in papers or CI:

• Alpha — methodology and backend coverage are still evolving.
• CPU runs support --execution-timeout to bound workload execution.
• CPU runs support repeated --program-arg values and --stdin-file; manifests can define the same settings.
• --experiment / --opt-config — apply compile macros and optional extra flags; plugin entry points remain planned for 0.3.0.
• Optional extras track, tune, and distributed are reserved for later releases and are not wired into the CLI.
• OpenCL — program may recompile each trial; interpret kernel times carefully.
• CUDA / Metal — not implemented despite registry entries.
• Native runners — Linux wheels bundle prebuilt native runners; other platforms build on demand from the bundled sources (needs a C/C++ toolchain and cmake).

Best supported today: Linux CPU compile/run/profile with perf configured, plus HIP/OpenCL exploration with manual interpretation.

Documentation

Public getting started guide:

https://www.compilersutra.com/docs/project/compilersutra-perf/getting-started/

In-tree guides (source checkout):

License

Apache License 2.0 — see LICENSE.

Author: Priya Pandey · CompilerSutra