CompilerSutraPerfTool

CompilerSutraPerfTool is a modular performance experimentation framework for native programs, GPU kernels, and shader workloads. It is designed for compiler engineers, systems developers, GPU developers, and performance researchers who need reproducible experiments across CPU and GPU backends.

Installation

Install locally in editable mode:

cd CompilerSutraPerfTool
python3 -m venv .venv
source .venv/bin/activate
pip install -e .

Install optional extras:

pip install -e '.[dev]'
pip install -e '.[visualize]'

For a published package, the intended distribution name is:

pip install compilersutra-perf

For a fuller design explanation, see GETTING_STARTED.md, ARCHITECTURE.md, USAGE.md, TESTING.md, RELEASING.md, SECURITY.md, CONTRIBUTING.md, and CODE_OF_CONDUCT.md.

Project Metadata

• Website: https://compilersutra.com
• Author: Abhinav
• LinkedIn: https://www.linkedin.com/in/abhinavcompilerllvm/
• License: Apache-2.0, see LICENSE

Repository Hygiene

• Contribution guide: CONTRIBUTING.md
• Security policy: SECURITY.md
• Release procedure: RELEASING.md
• CI workflow: .github/workflows/ci.yml

This repository is structured as a production-style MVP:

• Stable CPU-oriented workflow with real compile-and-run support
• Native OpenCL execution, native HIP execution, and native Vulkan shader validation
• Structured result generation for JSON/CSV consumers
• Lightweight HTML visualization and Streamlit dashboard support with derived metrics, comparison tables, and comparison charts
• Dedicated CPU-vs-GPU comparison summaries with normalized runtime and GPU speedup reporting
• A native C++ runtime runner used by CPU execution, ready for LLVM/MLIR-backed expansion

Current Capabilities

Backend	Current State	Notes
CPU	Execute + profile	Native execution through C++ runner, LLVM IR emission, curated perf counters, warmup/repeat support, and optional CPU affinity
GPU	Auto-route	Generic --backend gpu alias resolves to HIP for .hip, OpenCL for .cl, and Vulkan for shader inputs, with vendor-aware device inventory
OpenCL	Execute	Native kernel build and launch with generalized buffer/scalar/local argument binding, warmup/repeat support, vendor-aware device listing, and device selection
Vulkan	Compile + validate	GLSL to SPIR-V, native device selection, vendor-aware device listing, shader-module validation
CUDA	Planned	No native runner yet
HIP	Execute	Native hipcc compile-and-run path for .hip inputs, ROCm device discovery, warmup/repeat support, device selection, and rocprof-backed profiling artifacts
Metal	Planned	Required for full macOS GPU support

Supported On This Machine

The current repository has been exercised on this Linux host:

• CPU: AMD Ryzen 7 9700X
• OpenCL GPU devices detected: AMD Radeon RX 9060 XT and AMD Radeon Graphics
• HIP/ROCm GPU devices detected: AMD Radeon RX 9060 XT and AMD Radeon Graphics
• Vulkan runtime detected and native shader validation verified
• Tooling detected: perf, clang, clang++, hipcc, hipconfig, glslangValidator, spirv-opt, clinfo, vulkaninfo

This means the following paths have been verified locally:

• CPU compile + native execution
• CPU perf-based profiling with curated counters
• HIP native compile + execution
• OpenCL native kernel execution
• Vulkan SPIR-V compilation + native shader-module validation

Example Commands

CPU

csperf run --input examples/cpp/matrix_traversal.cpp --backend cpu --warmup-runs 1 --repeat-runs 3 --output results/cpu-perf.json
csperf profile results/cpu-perf.json

CPU + HIP + OpenCL Flow

csperf list-devices --backend gpu
csperf list-devices --backend hip
csperf run --input examples/cpp/matrix_traversal.cpp --backend cpu --warmup-runs 1 --repeat-runs 3 --output results/cpu-perf.json
csperf run --input examples/hip/vector_add.hip --backend hip --device-index 0 --warmup-runs 1 --repeat-runs 2 --output results/hip-profiled.json
csperf run --input examples/opencl/saxpy.cl --backend gpu --device-index 0 --warmup-runs 1 --repeat-runs 3 --output results/gpu-opencl.json
csperf visualize results/cpu-perf.json results/hip-profiled.json --output results/cpu-vs-hip.html
csperf visualize results/cpu-perf.json results/gpu-opencl.json --output results/cpu-vs-gpu.html

OpenCL

csperf run --input examples/opencl/saxpy.cl --backend gpu --device-index 0 --warmup-runs 1 --repeat-runs 3 --output results/gpu-opencl.json
csperf profile results/gpu-opencl.json

HIP

csperf list-devices --backend hip
csperf run --input examples/hip/vector_add.hip --backend hip --device-index 0 --warmup-runs 1 --repeat-runs 2 --output results/hip.json
csperf run --input examples/hip/vector_add.hip --backend hip --device-index 0 --warmup-runs 1 --repeat-runs 2 --no-perf --output results/hip-no-prof.json
csperf run --input examples/hip/vector_add.hip --backend gpu --device-index 0 --warmup-runs 1 --repeat-runs 2 --output results/gpu-hip.json
csperf profile results/hip.json

Generic GPU Alias

csperf list-devices --backend gpu
csperf run --input examples/opencl/saxpy.cl --backend gpu --device-index 0 --warmup-runs 1 --repeat-runs 3 --output results/gpu-opencl.json
csperf profile results/gpu-opencl.json

Vulkan

csperf run --input examples/shaders/vector_add.comp --backend vulkan --device-index 0 --output results/vulkan-native.json
csperf profile results/vulkan-native.json

Comparison Report

csperf visualize results/cpu-perf.json results/gpu-opencl.json --output results/compare-charts.html
csperf dashboard results/cpu-perf.json results/gpu-opencl.json
csperf list-devices --backend gpu
csperf list-devices --backend opencl
csperf list-devices --backend vulkan

Result Diff

csperf diff results/gcc.json results/clang.json
csperf diff results/gcc.json results/clang.json --csv results/gcc-vs-clang.csv
csperf diff results/gcc.json results/clang.json --output results/gcc-vs-clang.json --csv results/gcc-vs-clang.csv
csperf diff results/gcc.json results/clang.json --derived-config configs/derived_metrics.sample.json --output results/gcc-vs-clang-derived.json --csv results/gcc-vs-clang-derived.csv

The diff command compares two stored result files and can export:

• raw metric differences
• percentage differences
• the exact compiler used for each result from CC / CXX
• config-driven derived metrics from derived_metrics.sample.json
• bottleneck analysis from config thresholds and rules

Compiler Diff Batch

For CPU-only compiler comparison across a folder of C/C++ files, use:

python3 scripts/compiler_diff_batch.py examples/cpp \
  --config1 configs/compiler_gcc.sample.json \
  --config2 configs/compiler_clang.sample.json

Recursive folder scan:

python3 scripts/compiler_diff_batch.py examples \
  --config1 configs/compiler_gcc.sample.json \
  --config2 configs/compiler_clang.sample.json \
  --recursive

What it does:

• scans the folder for .c, .cc, .cpp, and .cxx files
• runs every file with compiler config 1 and compiler config 2
• generates per-file diff CSV files
• generates derived diff CSV files unless --skip-derived is used
• writes a combined summary CSV
• writes a summary XLSX workbook containing the CSV data as spreadsheet sheets

Inputs:

• --config1 and --config2 must point to compiler config JSON files such as compiler_gcc.sample.json and compiler_clang.sample.json
• the script is CPU-only and uses csperf run --backend cpu internally
• use --results-dir to choose the output directory and --skip-derived to omit derived-metric sheets

Outputs:

• results/compiler-diff/summary.csv
• results/compiler-diff/summary.xlsx
• per-file result JSON artifacts
• per-file diff CSV and JSON artifacts
• per-file derived diff CSV and JSON artifacts when derived diff is enabled

CPU vs GPU Comparison

csperf visualize results/cpu-perf.json results/gpu-opencl.json --output results/cpu-vs-gpu.html
csperf dashboard results/cpu-perf.json results/gpu-opencl.json

The current comparison mode adds:

• normalized comparable runtime in milliseconds
• CPU-vs-GPU summary table
• GPU speedup versus CPU
• backend-aware throughput and cache charts

Goals

• Accept a source file and infer the correct compilation and execution pipeline
• Run workload experiments with explicit flags such as memory layout or tiling
• Collect hardware context and profiling-ready metadata
• Store reproducible results for later comparison and visualization
• Keep compiler, runtime, profiling, and visualization layers modular

Repository Layout

CompilerSutraPerfTool/
├── README.md
├── CMakeLists.txt
├── pyproject.toml
├── configs/
├── examples/
├── native/
│   └── runtime/
├── src/
│   └── csperf/
└── tests/

Key Modules

• src/csperf/cli.py: CLI entry point with run, profile, diff, visualize, list-backends, and list-experiments
• src/csperf/execution.py: Python orchestration layer that builds and invokes native runtime components, benchmark controls, and trial summaries
• src/csperf/hardware.py: GPU tooling detection and vendor classification helpers
• src/csperf/detector.py: Source type detection based on extension
• src/csperf/pipelines/: Compilation and execution pipeline planners
• src/csperf/backends/: Backend registry and backend capability metadata
• src/csperf/profiler/: Profiling abstractions and CPU-oriented metric definitions
• src/csperf/profile_summary.py: Shared derived-metric, unit-aware, and CPU-vs-GPU comparison summaries
• src/csperf/results.py: Result schema and JSON/CSV export
• src/csperf/visualize.py: HTML report generator with multi-result comparison tables, CPU-vs-GPU summaries, and bar-style charts
• native/runtime/: C++ runtime split into reusable libraries and thin native tools for low-overhead execution

Supported Workload Types

• C programs
• C++ programs
• OpenCL kernels
• GPU compute kernels
• Vulkan compute shaders
• Compute workloads and shader validation
• Memory access benchmarks

Current Stability Model

• Stable
  • CLI orchestration
  • Workload detection
  • CPU compile and native execution
  • CPU perf collection with curated default counters
  • Experiment configuration
  • Structured result export
  • Unit-aware profiling summaries
• Experimental
  • HIP native execution with hipcc compile-and-run
  • OpenCL native execution with generalized buffer/scalar/local binding
  • Vulkan native shader validation
  • Chart-heavy visualization
• Research
  • MLIR-based transformations
  • Architecture-aware autotuning
  • Cross-backend optimization studies

Quick Start

Detailed usage instructions are in USAGE.md.

1. Create a virtual environment

cd CompilerSutraPerfTool
python3 -m venv .venv
source .venv/bin/activate
pip install -e .

2. List available capabilities

csperf list-backends
csperf list-experiments
csperf cpuinfo
csperf gpuinfo
csperf deviceinfo

3. Run an example workload

csperf run --input examples/cpp/matrix_traversal.cpp --experiment row-major,column-major
csperf run --input examples/cpp/tiled_matmul.cpp --experiment tiled --tile-size 32
csperf run --input examples/shaders/vector_add.comp --backend gpu
csperf run --input examples/opencl/saxpy.cl --backend gpu
csperf run --input examples/hip/vector_add.hip --backend gpu
csperf run --input examples/opencl/saxpy.cl --backend opencl --vendor amd
csperf run --input examples/shaders/vector_add.comp --backend vulkan --vendor amd
csperf run --input examples/opencl/saxpy.cl --backend opencl --policy-config configs/policy_config.sample.json

For CPU workloads, csperf run compiles and executes the program by default when clang or clang++ is installed. For HIP workloads, the tool compiles .hip sources with hipcc and executes them against the ROCm runtime. For Vulkan shaders, the tool compiles GLSL to SPIR-V and then validates the shader natively against the Vulkan runtime. For OpenCL kernels, the tool builds and launches kernels natively through the C++ OpenCL runner.

4. Inspect or visualize results

csperf profile results/latest.json
csperf diff results/gcc.json results/clang.json --csv results/gcc-vs-clang.csv
csperf diff results/gcc.json results/clang.json --derived-config configs/derived_metrics.sample.json --output results/gcc-vs-clang-derived.json --csv results/gcc-vs-clang-derived.csv
csperf visualize results/latest.json --output results/report.html
csperf dashboard results/latest.json
csperf visualize results/cpu-perf.json results/gpu-opencl.json --output results/compare-charts.html
csperf dashboard results/cpu-perf.json results/gpu-opencl.json
csperf visualize results/cpu-perf.json results/gpu-opencl.json --output results/cpu-vs-gpu.html

CLI Overview

Run a workload

csperf run --input program.c
csperf run --input matmul.cpp --backend cpu
csperf run --input shader.comp --backend gpu
csperf run --input kernel.cl --backend gpu
csperf run --input kernel.hip --backend gpu

Useful execution flags:

csperf run --input program.cpp --plan-only
csperf run --input program.cpp --no-perf
csperf run --input program.cpp --build-dir build/debug
csperf run --input program.cpp --warmup-runs 1 --repeat-runs 5
csperf run --input examples/opencl/saxpy.cl --backend gpu --device-index 0
csperf run --input examples/opencl/saxpy.cl --backend opencl --vendor amd --device-index 0
csperf run --input examples/opencl/saxpy.cl --backend opencl --policy-config configs/policy_config.sample.json --device-index 0
csperf run --input examples/opencl/saxpy.cl --backend gpu --kernel-name saxpy --kernel-arg buffer:float:read:4096:1.0 --kernel-arg buffer:float:read:4096:2.0 --kernel-arg buffer:float:write:4096:0.0 --kernel-arg scalar:uint32:4096 --readback-arg 2
csperf run --input examples/hip/vector_add.hip --backend hip --device-index 0
csperf run --input examples/shaders/vector_add.comp --backend vulkan --vendor amd --device-index 0
csperf run --input examples/shaders/vector_add.comp --backend vulkan --policy-config configs/policy_config.sample.json --device-index 0

Memory layout experiments

csperf run --input matrix.cpp --experiment row-major
csperf run --input matrix.cpp --experiment column-major
csperf run --input matrix.cpp --experiment tiled --tile-size 32
csperf run --input matrix.cpp --experiment row-major,column-major,tiled --tile-size 32

Custom optimization experiment

csperf run \
  --input examples/cpp/tiled_matmul.cpp \
  --experiment custom \
  --opt-config configs/sample_tuning.json

Example Workloads

CPU examples

• examples/cpp/matrix_traversal.cpp: row-major vs column-major traversal
• examples/cpp/tiled_matmul.cpp: tiled matrix multiplication
• examples/c/memory_stride.c: cache locality and memory stride behavior

GPU / shader examples

• examples/shaders/vector_add.comp: Vulkan compute shader
• examples/opencl/vector_add.cl: OpenCL kernel
• examples/opencl/saxpy.cl: OpenCL kernel with scalar argument binding
• examples/hip/vector_add.hip: HIP vector-add example with JSON metric output

Each example is accompanied by simple commands and is intended as a starting point for backend experimentation.

GPU Status

• Vulkan: shader compilation to SPIR-V plus native device and shader-module validation is implemented
• HIP: native compile-and-run is implemented with ROCm device discovery
• HIP profiling: rocprof --stats --hip-trace integration is implemented and parsed into result metrics plus artifacts
• OpenCL: native kernel build and launch is implemented with configurable kernel argument binding
• CUDA / Metal: planning only

The next implementation step is workload matching plus baseline/regression analysis, then full Vulkan compute dispatch and broader backend coverage.

Native Runtime

The native/runtime directory now follows a more LLVM-like split:

• include/csperf/support/: shared support headers such as debug assertions and JSON helpers
• include/csperf/runtime/: backend-independent runtime interfaces such as process helpers
• include/csperf/native/: CPU-native application interfaces
• include/csperf/opencl/: OpenCL application and support interfaces
• include/csperf/vulkan/: Vulkan application and support interfaces
• lib/Support/, lib/Runtime/, lib/Native/, lib/OpenCL/, lib/Vulkan/: reusable implementation libraries
• tools/csperf-native-runner/, tools/csperf-opencl-runner/, tools/csperf-vulkan-runner/: thin executable entrypoints

The backend libraries are also split by responsibility:

• Native CPU: argument parser, process runner, and application entry
• OpenCL: argument parser, device catalog, kernel executor, and application entry
• Vulkan: argument parser, runtime loader, device catalog, shader validator, and application entry

Hardware-specific decisions are now intended to live behind policy objects instead of preprocessor branches:

• OpenCL uses device-filter policies
• Vulkan uses library-loading and queue-family-selection policies

This keeps vendor or hardware-specific behavior injectable and reusable without spreading #ifdef logic through backend code.

The native code is organized around namespaces that mirror the layout:

• csperf::support
• csperf::runtime
• csperf::native
• csperf::opencl
• csperf::vulkan

The public CLI behavior is unchanged, but the internal runtime is now split into libraries first and executables second instead of monolithic backend source files.

The intent is to grow this further into:

• low-overhead launcher APIs
• pinned-memory and buffer abstractions
• platform/backend adapters
• future LLVM JIT or ahead-of-time execution support

Visualization

• csperf profile prints unit-aware metric summaries and derived metrics such as IPC and miss-rate ratios.
• csperf diff compares two stored result files, can export CSV/JSON output, and supports config-driven derived metrics and bottleneck rules.
• csperf visualize generates an HTML report from one or more JSON result artifacts.
• csperf visualize shows derived metrics, units, comparison tables, CPU-vs-GPU summaries, and bar-style comparison charts.
• csperf dashboard launches a Streamlit dashboard against one or more result files and supports comparison tables, CPU-vs-GPU summaries, and charts.

HIP runs can also emit profiler artifacts such as:

• kernel stats CSV
• HIP API stats CSV
• copy stats CSV
• JSON trace

Install dashboard dependencies with:

pip install -e '.[visualize]'

Testing

Detailed validation steps are in TESTING.md.

python3 -m compileall src
python -m pytest

The included tests validate:

• source type detection
• experiment registry behavior
• CLI smoke behavior
• native runner build and backend verification are covered in the manual testing guide

CPU Execution Notes

• CPU runs use clang for C and clang++ for C++ when available.
• LLVM IR is emitted into the selected build directory alongside the native binary.
• CPU binary execution is handled by the native C++ runner built through CMake.
• perf collection is attempted by default on CPU runs using a curated event set from the local PMU list.
• CPU runs support warmup and repeated timing trials.
• CPU runs support optional CPU affinity through --cpu-affinity.
• CPU profile output includes units, derived metrics, and timing summaries across repeated runs.
• On systems with restrictive perf_event_paranoid settings, execution still succeeds and the result artifact records the profiling error in the execution.perf_error field.

HIP Execution Notes

• HIP runs use hipcc when available.
• HIP device discovery uses rocminfo.
• HIP profiling uses rocprof --stats --hip-trace when --no-perf is not set.
• HIP result artifacts can include parsed profiler metrics and generated profiler CSV/JSON files.
• On this ROCm stack, rocprof prints deprecation/support warnings but still produces usable profiling outputs.

Benchmark Controls

The current implementation supports:

• --warmup-runs N: warmup iterations before measurement
• --repeat-runs N: repeated measured trials with summary statistics
• --device-index N: explicit device selection for HIP execution, OpenCL execution, and Vulkan validation
• --cpu-affinity 0,1: pin CPU execution to specific logical cores through the native runner
• --backend gpu: generic GPU alias for currently supported GPU workload types
• --compiler-flag FLAG: append a compiler option; repeat the flag to pass multiple options

Example:

csperf run --input examples/cpp/matrix_traversal.cpp --backend cpu \
  --compiler-flag=-march=native --compiler-flag=-funroll-loops

Device inventory commands:

• csperf list-devices --backend cpu
• csperf list-devices --backend gpu
• csperf list-devices --backend hip
• csperf list-devices --backend opencl
• csperf list-devices --backend vulkan

Recommended Next Steps

If you are using the tool now:

1. Run matched CPU and GPU workloads and save the result JSON files.
2. Generate cpu-vs-hip.html and cpu-vs-gpu.html reports.
3. Inspect HIP profiler metrics and artifacts in the result JSON.
4. Use the dashboard for side-by-side comparison across CPU, HIP, and OpenCL runs.

If you are extending the tool next:

1. Add workload matching and baseline/regression support.
2. Add richer HIP metric extraction from newer ROCm profiler paths.
3. Add full Vulkan compute dispatch.
4. Add CUDA and Metal backends.

Roadmap

Phase 1

• Complete CPU compile-and-run path using Clang/LLVM
• Add perf integration for hardware counters
• Improve result schema and metric normalization

Phase 2

• Add workload matching and baseline/regression support
• Add Vulkan compute execution and profiling hooks
• Add CUDA and Metal adapters and deeper HIP profiling
• Introduce MLflow-backed experiment tracking

Phase 3

• Add MLIR-based transformation passes
• Add Optuna-based autotuning workflows
• Add distributed experiment scheduling with Ray

Technology Stack

See CompilerSutraPerfTool-Technology-Stack.md for the full stack recommendation that accompanies this repository.

License

CompilerSutraPerfTool is licensed under the Apache License 2.0. See LICENSE.

Author

• Abhinav
• CompilerSutra: https://compilersutra.com
• LinkedIn: https://www.linkedin.com/in/abhinavcompilerllvm/
• Maintainer details: AUTHORS.md