🚧 Aidge Benchmark

[!WARN] This repository is still under active construction.

This module benchmarks ML models across Eclipse Aidge modules (CPU, CUDA, exportCPP), as well as external libraries (Torch, ONNXRuntime). It measures:

• Correctness (output similarity)
• Precision
• Inference time (performance)

Useful links

• Eclipse Aidge main repository
• Installation guide
• Usage
  • CLI
  • Python API

Quick start

System Requirements

• python >= 3.10
• numpy >= 1.22.0
• aidge_core >= 0.7.0

Any benchmarked library might require additional dependencies, notably onnx for external libraries.

Plotting results requires matplolib package.

pip install aidge-benchmark
python -c "import aidge_core; import aidge_benchmark; create_benchmark_from_dict({})"

Optional backends (TVM, TFLite)

aidge_benchmark can benchmark against Apache TVM (backends tvm / tvm_tuned) and TensorFlow Lite (backend tflite), but their runtime dependencies are not installed automatically, you must install them yourself before passing any of these backend names to the CLI or Python API.

The exact version pins known to work in CI are tracked in docker/ci/requirements.txt. Use them as the reference if a fresh install misbehaves.

TVM

The TVM backend relies on INRIA's prebuilt wheels (LLVM is required by TVM's llvm target):

pip install "xtc-llvm-tools==21.1.2.6" "xtc-tvm-python-bindings==0.19.0.10"

These wheels are not available on Windows. Once installed, import tvm should succeed; the benchmark backends tvm and tvm_tuned will then be usable.

TFLite

The TFLite backend uses tensorflow's interpreter, the tflite schema package, and onnx2tf for the ONNX → TFLite conversion:

pip install "tensorflow==2.18.0" "tflite" "onnx2tf"

🛠 Build from Source

Prerequisite (in addition to previous one):

• Please review the global installation instructions before proceeding.
• If using a virtual environment, use the same one for all Aidge modules.

1. Python installation using setup scripts

[!TIP] Use Get-Help setup.ps1 (Windows) or ./setup.sh -h (Unix) for full documentation.

2. Python Installation using pip

Run these commands from the aidge_benchmark/ directory:

# Standard install
pip install . -v

# Install with testing dependencies
pip install .[test] -v && pytest

Editable Install (Experimental)

Use this for real-time development without re-installing.

pip install --no-build-isolation -ve . --config-settings=editable.rebuild=true -Cbuild-dir=build

Usage

Command-line interface (CLI)

This module comes with a callable script: aidge_benchmark

aidge_benchmark --config-file conv2d.json -c -t -m aidge_export_cpp aidge_backend_cpu aidge_backend_cuda torch onnxruntime --save-directory benchmark_results

aidge_benchmark --help

More details on the tutorials

Python API

Here is an example script to generate a result and time comparison of several libraries with a reference library for the convolution operation and save results as an svg image.

import numpy as np
import matplotlib.pyplot as plt
from pathlib import Path

import aidge_benchmark as bench
from aidge_onnx import convert_aidge_to_onnx
import aidge_backend_cpu  # ensures backend is registered
import aidge_export_cpp  # ensures backend is registered

# List of backends we want to test against the reference implementation
TESTED_LIBS: list[str] = ["torch", "aidge_backend_cpu", "aidge_export_cpp"]
REFERENCE_LIB: str = "onnxruntime"
# Benchmark test cases description file
BENCHMARK_FILE = Path("conv2d.json")
N_ITER = 10


# Helper function
def format_time_stats(times: list[float]) -> str:
    """Return mean ± std formatted in ms (scientific notation)."""
    mean, std = np.mean(times), np.std(times)
    return f"{mean*1e3:.2e} ± {std*1e3:.2e} ms"


def run_single_benchmark(cfg):
    """
    Run a benchmark for a single configuration.
    Returns a dictionary mapping backend name -> list of timings (seconds).
    """
    # Create random input tensors according to the spec
    inputs = cfg.generate_inputs()

    # Build ONNX and Aidge model description from 'cfg' described with ONNX format
    model_aidge = cfg.as_model(input_arrays=inputs)
    model = convert_aidge_to_onnx(
            model_aidge, f"test-model_{cfg.operation}_{label}", opset=21, ir_version=10
        )
    # update input names for ONNX
    for idx, named_tensor in enumerate(inputs):
        if idx < len(model.graph.input):
            for node in model.graph.node:
                for i, inp in enumerate(node.input):
                    if inp == model.graph.input[idx].name:
                        node.input[i] = named_tensor.name
            model.graph.input[idx].name = named_tensor.name


    # /!\ specific to ONNX format
    # Determine how many inputs are actual data (excluding initializers)
    nb_data_inputs = cfg.format.metadata["initializer_rank"]

    onnx_param = {"model": model, "inputs": inputs[:nb_data_inputs]}
    aidge_param = {"model": model_aidge, "inputs": inputs[:nb_data_inputs]}

    backend_params = {
        **{lib: onnx_param for lib in ["torch", "onnxruntime"]},
        **{lib: aidge_param for lib in ["aidge_backend_cpu", "aidge_export_cpp"]},
    }

    # Reference run
    ref_out = bench.compute_output(REFERENCE_LIB, **backend_params[REFERENCE_LIB])
    ref_times = bench.measure_inference_time(
        REFERENCE_LIB, **backend_params[REFERENCE_LIB], nb_iterations=N_ITER
    )
    print(f"\t{REFERENCE_LIB:<20}✔️\t{format_time_stats(ref_times)}")
    results = {REFERENCE_LIB: ref_times}

    # Test each backend and compare results
    for lib in TESTED_LIBS:
        out = bench.compute_output(lib, **backend_params[lib])
        is_equal = bench.utils.compare_tensors(ref_out, out, verbose=False)
        times = bench.measure_inference_time(
            lib, **backend_params[lib], nb_iterations=N_ITER
        )
        print(f"\t{lib:<20}{'✔️' if is_equal else '❌'}\t{format_time_stats(times)}")
        results[lib] = times
    return results

if __name__ == "__main__":
    scheme, default_cfg = bench.create_benchmark_from_json(str(BENCHMARK_FILE))
    timing_results = {}
    for  label, cfg in scheme:
        print(f"\n- {label}")

        # Merge test config with default settings
        full_cfg = default_cfg.override_with(cfg)
        timing_results[label] = run_single_benchmark(full_cfg)

    fig, _ = bench.visualize.ratio_plot(timing_results, REFERENCE_LIB)
    fig.savefig(f"{str(BENCHMARK_FILE)}_performances.svg")
    print(f"\nSaved: '{str(BENCHMARK_FILE)}_performances.svg'")

License

Aidge is licensed under Eclipse Public License 2.0, as found in the LICENSE.