ML SDK Scenario Runner

The Scenario Runner is an application that executes shader and neural network graph workloads through Vulkan® or the ML extensions for Vulkan®. The Scenario Runner acts as a validation and performance exploration vehicle. The Scenario Runner also acts as a mechanism to define test-cases called scenarios in a declarative way via a JSON description. The Scenario Runner can parse the JSON, load the input stimulus that is described in the JSON, execute the scenario and produce output artifacts.

Cloning the repository

To clone the ML SDK Scenario Runner as a stand-alone repository, you can use regular git clone commands. However, for better management of dependencies and to ensure everything is placed in the appropriate directories, we recommend using the git-repo tool to clone the repository as part of the ML SDK for Vulkan® suite. Repo tool.

For a minimal build and to initialize only the Scenario Runner and its dependencies, run:

repo init -u https://github.com/arm/ai-ml-sdk-manifest -g scenario-runner

Alternatively, to initialize the repo structure for the entire ML SDK for Vulkan®, including the Scenario Runner, run:

repo init -u https://github.com/arm/ai-ml-sdk-manifest -g all

Once the repo is initialized, you can fetch the contents:

repo sync --no-clone-bundle

Cloning on Windows®

To ensure nested submodules do not exceed the maximum long path length, you must enable long paths on Windows®, and you must clone close to the root directory or use a symlink. Make sure to use Git for Windows.

Using PowerShell:

Set-ItemProperty -Path "HKLM:\SYSTEM\CurrentControlSet\Control\FileSystem" -Name "LongPathsEnabled" -Value 1
git config --global core.longpaths true
git --version # Ensure you are using Git for Windows, for example 2.50.1.windows.1
git clone <git-repo-tool-url>
python <path-to-git-repo>\git-repo\repo init -u <manifest-url> -g all
python <path-to-git-repo>\git-repo\repo sync --no-clone-bundle

Using Git Bash:

cmd.exe "/c reg.exe add \"HKLM\System\CurrentControlSet\Control\FileSystem"" /v LongPathsEnabled /t REG_DWORD /d 1 /f"
git config --global core.longpaths true
git --version # Ensure you are using the Git for Windows, for example 2.50.1.windows.1
git clone <git-repo-tool-url>
python <path-to-git-repo>/git-repo/repo init -u <manifest-url> -g all
python <path-to-git-repo>/git-repo/repo sync --no-clone-bundle

After the sync command completes successfully, you can find the ML SDK Scenario Runner in <repo_root>/sw/scenario-runner/. You can also find all the dependencies required by the ML SDK Scenario Runner in <repo_root>/dependencies/.

Building Scenario Runner from source

The build system must have:

• C/C++ 17 compiler: GCC or Clang on Linux, Clang on Darwin, or MSVC on Windows®.
• CMake 3.25 or later.
• Ninja 1.8.2 or later.
• Python 3.10 through 3.14 and uv 0.9.26. Runtime, test, and tooling dependencies are declared in pyproject.toml and locked in uv.lock.

The following dependencies are also needed:

• Argument Parser for Modern C++.
• DirectXShaderCompiler.
• glslang.
• JSON for Modern C++.
• SPIRV-Headers.
• SPIRV-Tools.
• Vulkan-Headers.
• stb headers stb_image.h and stb_image_write.h (these are downloaded automatically during CMake configure)
• GoogleTest. Optional, for testing.

For the preferred dependency versions see the manifest file.

Building with the script

To build on Linux, run the following command:

SDK_PATH="path/to/sdk"
python3 ${SDK_PATH}/sw/scenario-runner/scripts/build.py -j $(nproc) \
    --flatbuffers-path ${SDK_PATH}/dependencies/flatbuffers \
    --argparse-path ${SDK_PATH}/dependencies/argparse \
    --json-path ${SDK_PATH}/dependencies/json \
    --vulkan-headers-path ${SDK_PATH}/dependencies/Vulkan-Headers \
    --glslang-path ${SDK_PATH}/dependencies/glslang \
    --dxc-path ${SDK_PATH}/dependencies/DirectXShaderCompiler \
    --spirv-headers-path ${SDK_PATH}/dependencies/SPIRV-Headers \
    --spirv-tools-path ${SDK_PATH}/dependencies/SPIRV-Tools \
    --vgf-lib-path ${SDK_PATH}/sw/vgf-lib \
    --gtest-path ${SDK_PATH}/dependencies/googletest

To build on Windows®, run the following command:

$env:SDK_PATH="path\to\sdk"
$cores = [System.Environment]::ProcessorCount
python "$env:SDK_PATH\sw\scenario-runner\scripts\build.py" -j $cores  `
    --flatbuffers-path "$env:SDK_PATH\dependencies\flatbuffers" `
    --argparse-path "$env:SDK_PATH\dependencies\argparse" `
    --json-path "$env:SDK_PATH\dependencies\json" `
    --vulkan-headers-path "$env:SDK_PATH\dependencies\Vulkan-Headers" `
    --glslang-path "$env:SDK_PATH\dependencies\glslang" `
    --dxc-path "$env:SDK_PATH\dependencies\DirectXShaderCompiler" `
    --spirv-headers-path "$env:SDK_PATH\dependencies\SPIRV-Headers" `
    --spirv-tools-path "$env:SDK_PATH\dependencies\SPIRV-Tools" `
    --vgf-lib-path "$env:SDK_PATH\sw\vgf-lib" `
    --gtest-path "$env:SDK_PATH\dependencies\googletest"

To build on Darwin from a Repo tool checkout, first configure the LunarG Vulkan® SDK for Darwin environment according to its documentation, then run:

export SDK_PATH=/absolute/path/to/ml-sdk
python3 "$SDK_PATH/sw/scenario-runner/scripts/build.py" \
    -j "$(sysctl -n hw.logicalcpu)"

At runtime, use the Vulkan® Loader to select either MoltenVK or KosmicKrisp and enable the ML Emulation Layer for Vulkan®. The ML SDK's Darwin support is experimental.

To cross compile for AArch64 architecture, you can add the following option:

SDK_PATH="path/to/sdk"
python3 $SDK_PATH/sw/scenario-runner/scripts/build.py -j $(nproc) \
    --flatbuffers-path ${SDK_PATH}/dependencies/flatbuffers \
    --argparse-path ${SDK_PATH}/dependencies/argparse \
    --json-path ${SDK_PATH}/dependencies/json \
    --vulkan-headers-path ${SDK_PATH}/dependencies/Vulkan-Headers \
    --glslang-path ${SDK_PATH}/dependencies/glslang \
    --dxc-path ${SDK_PATH}/dependencies/DirectXShaderCompiler \
    --spirv-headers-path ${SDK_PATH}/dependencies/SPIRV-Headers \
    --spirv-tools-path ${SDK_PATH}/dependencies/SPIRV-Tools \
    --vgf-lib-path ${SDK_PATH}/sw/vgf-lib \
    --gtest-path ${SDK_PATH}/dependencies/googletest \
    --target-platform aarch64

(Experimental) To cross compile for Android™, and build an APK package, you must target Android™, provide the Android™ CMake toolchain, and request the apk package type.

SDK_PATH="path/to/sdk"
ANDROID_NDK_HOME="path/to/android-ndk"
python3 ${SDK_PATH}/sw/scenario-runner/scripts/build.py -j $(nproc) \
    --target-platform android \
    --cmake-toolchain-for-android ${ANDROID_NDK_HOME}/build/cmake/android.toolchain.cmake \
    --flatbuffers-path ${SDK_PATH}/dependencies/flatbuffers \
    --argparse-path ${SDK_PATH}/dependencies/argparse \
    --json-path ${SDK_PATH}/dependencies/json \
    --vulkan-headers-path ${SDK_PATH}/dependencies/Vulkan-Headers \
    --glslang-path ${SDK_PATH}/dependencies/glslang \
    --dxc-path ${SDK_PATH}/dependencies/DirectXShaderCompiler \
    --spirv-headers-path ${SDK_PATH}/dependencies/SPIRV-Headers \
    --spirv-tools-path ${SDK_PATH}/dependencies/SPIRV-Tools \
    --vgf-lib-path ${SDK_PATH}/sw/vgf-lib \
    --package-type apk

The APK build uses the gradle command, so Gradle 8.4 or later must be available on your PATH. The Android™ SDK installation pointed to by ANDROID_HOME should include build-tools;34.0.0 and platforms;android-34, or other compatible versions.

To build with RenderDoc support, add --enable-rdoc. If RenderDoc is installed outside the default search locations, also pass --renderdoc-path <renderdoc-install-root>.

At runtime, pass --capture-frame and enable the RenderDoc Vulkan® layer. When capturing a workload through the ML Emulation Layer for Vulkan®, order RenderDoc after the Graph and Tensor layers so it records the translated Vulkan® commands:

export VK_INSTANCE_LAYERS=VK_LAYER_ML_Graph_Emulation:VK_LAYER_ML_Tensor_Emulation:VK_LAYER_RENDERDOC_Capture
scenario-runner --capture-frame \
    --scenario scenario.json

Some RenderDoc configurations may not support VK_KHR_portability_enumeration. If RenderDoc reports that the extension is unsupported and the Vulkan® device does not require it, disable it with:

--disable-extension VK_KHR_portability_enumeration

To enable and run tests, use the --test flag. To lint the tests, use the --lint flag. To enable tests and documentation building python dependencies must be installed:

uv sync --locked --no-install-project --no-default-groups --group test --group tooling

Run build commands with uv run --no-sync python scripts/build.py to use the locked environment. To build documentation, pass --doc; Doxygen must also be installed on your machine.

You can install the build artifacts for this project into a specified location. To install the build artifacts, pass the --install option with the required path.

To create an archive with the build artifacts, you must add --package-type. The archive is stored in the provided location.

For more command line options, consult the program help:

python3 $SDK_PATH/sw/scenario-runner/scripts/build.py --help

Code coverage

Sync the locked test and tooling dependencies, then run the C++ and Python unit tests on Linux with GCC coverage instrumentation:

uv sync --locked --no-install-project --no-default-groups --group test --group tooling
uv run --no-sync python scripts/build.py --coverage --build-dir build-coverage

The coverage command writes a detailed HTML report to build-coverage/coverage/index.html and a machine-readable JSON summary to build-coverage/coverage/summary.json. Coverage is collected for Scenario Runner sources and public headers, including native code exercised through pytest. Python source-line coverage is not collected.

Usage

To run a scenario file, use the following command:

./scenario-runner --scenario ${SCENARIO_JSON_FILE}

Where:

• --scenario: File to load the scenario from. The file must be in JSON format. If the resources in the SCENARIO_JSON_FILE are not specified with absolute paths, their relative paths will be resolved against the parent directory of the SCENARIO_JSON_FILE.

For more details, see the help output:

./scenario-runner --help

For the Android APK package, install the generated APK and start the foreground service with adb:

adb install -r build/scenario-runner-debug.apk

adb shell am start-foreground-service \
    -n com.arm.ai_ml_sdk_scenario_runner/.Main \
    --esa args --scenario,/data/user/0/com.arm.ai_ml_sdk_scenario_runner/scenario.json

PyPI

The ML SDK Scenario Runner is available on PyPI as the ai-ml-sdk-scenario-runner package.

Install the published package:

pip install ai-ml-sdk-scenario-runner

Python bindings

Python bindings are available through the scenario_runner package:

import numpy as np
import scenario_runner

builder = scenario_runner.ScenarioBuilder()
image_id = builder.add_image(
    [1, 2, 2, 1],
    scenario_runner.Format.R8Uint,
    is_input=True,
    is_sampled=True,
    tiling=scenario_runner.Tiling.Linear,
)
scenario = builder.build()

image = np.array([1, 2, 3, 4], dtype=np.uint8).reshape(1, 2, 2, 1)
scenario.upload(image_id, image)
scenario.run()
result = scenario.download(image_id)

Install from source with pip

From the repository root, install the package with:

pip install .

This command builds the host executable and Python bindings, then installs the resulting Python package. Ensure the prerequisites listed in Building Scenario Runner from source are installed and the repository dependencies have been initialized.

Graph Shape Inference

Graph shape inference is experimental. A scenario requests graph shape inference by marking graph output tensor dimensions with ?. Fully shaped graph outputs do not request shape inference. Input tensors must be fully shaped, and Scenario Runner does not inspect SPIR-V™ modules to detect unshaped graphs automatically.

Known Limitations

• Resources created with Optimal tiling cannot be used with memory aliasing.
• HLSL lacks support for VK_ARM_tensor at the moment, so tensor resources cannot be used with HLSL shaders.
• For graph shape inference, partially specified output shapes are not supported as constraints. Tensor dimensions must be either unranked or ranked with all dynamic shapes. Mixed dimensions such as [1, "?", 2, "?"] are not supported. Inferred output shapes replace the declared dimensions.
• A graph resource requiring shape inference cannot be reused by dispatches with different interface shapes. Declare a separate graph resource for each shape.

License

The ML SDK Scenario Runner is distributed under the software licenses in LICENSES directory.

Trademark notice

Arm® is a registered trademark of Arm Limited (or its subsidiaries) in the US and/or elsewhere.

Khronos® and Vulkan® are registered trademarks, and SPIR-V™ is a trademark of The Khronos Group Inc..