ML SDK Scenario Runner

The Scenario Runner is an application that executes shader and neural network graph workloads through Vulkan® or the ML extensions for Vulkan®. The Scenario Runner acts as a validation and performance exploration vehicle. The Scenario Runner also acts as a mechanism to define test-cases called scenarios in a declarative way via a JSON description. The Scenario Runner can parse the JSON, load the input stimulus that is described in the JSON, execute the scenario and produce output artifacts.

Cloning the repository

To clone the ML SDK Scenario Runner as a stand-alone repository, you can use regular git clone commands. However, for better management of dependencies and to ensure everything is placed in the appropriate directories, we recommend using the git-repo tool to clone the repository as part of the ML SDK for Vulkan® suite. Repo tool.

For a minimal build and to initialize only the Scenario Runner and its dependencies, run:

repo init -u https://github.com/arm/ai-ml-sdk-manifest -g scenario-runner

Alternatively, to initialize the repo structure for the entire ML SDK for Vulkan®, including the Scenario Runner, run:

repo init -u https://github.com/arm/ai-ml-sdk-manifest -g all

Once the repo is initialized, you can fetch the contents:

repo sync --no-clone-bundle

Cloning on Windows®

To ensure nested submodules do not exceed the maximum long path length, you must enable long paths on Windows®, and you must clone close to the root directory or use a symlink. Make sure to use Git for Windows.

Using PowerShell:

Set-ItemProperty -Path "HKLM:\SYSTEM\CurrentControlSet\Control\FileSystem" -Name "LongPathsEnabled" -Value 1
git config --global core.longpaths true
git --version # Ensure you are using Git for Windows, for example 2.50.1.windows.1
git clone <git-repo-tool-url>
python <path-to-git-repo>\git-repo\repo init -u <manifest-url> -g all
python <path-to-git-repo>\git-repo\repo sync --no-clone-bundle

Using Git Bash:

cmd.exe "/c reg.exe add \"HKLM\System\CurrentControlSet\Control\FileSystem"" /v LongPathsEnabled /t REG_DWORD /d 1 /f"
git config --global core.longpaths true
git --version # Ensure you are using the Git for Windows, for example 2.50.1.windows.1
git clone <git-repo-tool-url>
python <path-to-git-repo>/git-repo/repo init -u <manifest-url> -g all
python <path-to-git-repo>/git-repo/repo sync --no-clone-bundle

After the sync command completes successfully, you can find the ML SDK Scenario Runner in <repo_root>/sw/scenario-runner/. You can also find all the dependencies required by the ML SDK Scenario Runner in <repo_root>/dependencies/.

Building Scenario Runner from source

The build system must have:

• CMake 3.25 or later.
• C/C++ 17 compiler: GCC, or optionally Clang on Linux and MSVC on Windows®.
• Python 3.10 or later. Required python libraries for building are listed in tooling-requirements.txt.
• Ninja 1.10 or later.

The following dependencies are also needed:

• Argument Parser for Modern C++.
• DirectXShaderCompiler.
• glslang.
• JSON for Modern C++.
• SPIRV-Headers.
• SPIRV-Tools.
• Vulkan-Headers.
• stb headers stb_image.h and stb_image_write.h (these are downloaded automatically during CMake configure)
• GoogleTest. Optional, for testing.

For the preferred dependency versions see the manifest file.

Building with the script

To build on Linux, run the following command:

SDK_PATH="path/to/sdk"
python3 ${SDK_PATH}/sw/scenario-runner/scripts/build.py -j $(nproc) \
    --flatbuffers-path ${SDK_PATH}/dependencies/flatbuffers \
    --argparse-path ${SDK_PATH}/dependencies/argparse \
    --json-path ${SDK_PATH}/dependencies/json \
    --vulkan-headers-path ${SDK_PATH}/dependencies/Vulkan-Headers \
    --glslang-path ${SDK_PATH}/dependencies/glslang \
    --dxc-path ${SDK_PATH}/dependencies/DirectXShaderCompiler \
    --spirv-headers-path ${SDK_PATH}/dependencies/SPIRV-Headers \
    --spirv-tools-path ${SDK_PATH}/dependencies/SPIRV-Tools \
    --vgf-lib-path ${SDK_PATH}/sw/vgf-lib \
    --gtest-path ${SDK_PATH}/dependencies/googletest

To build on Windows®, run the following command:

$env:SDK_PATH="path\to\sdk"
$cores = [System.Environment]::ProcessorCount
python "$env:SDK_PATH\sw\scenario-runner\scripts\build.py" -j $cores  `
    --flatbuffers-path "$env:SDK_PATH\dependencies\flatbuffers" `
    --argparse-path "$env:SDK_PATH\dependencies\argparse" `
    --json-path "$env:SDK_PATH\dependencies\json" `
    --vulkan-headers-path "$env:SDK_PATH\dependencies\Vulkan-Headers" `
    --glslang-path "$env:SDK_PATH\dependencies\glslang" `
    --dxc-path "$env:SDK_PATH\dependencies\DirectXShaderCompiler" `
    --spirv-headers-path "$env:SDK_PATH\dependencies\SPIRV-Headers" `
    --spirv-tools-path "$env:SDK_PATH\dependencies\SPIRV-Tools" `
    --vgf-lib-path "$env:SDK_PATH\sw\vgf-lib" `
    --gtest-path "$env:SDK_PATH\dependencies\googletest"

To cross compile for AArch64 architecture, you can add the following option:

SDK_PATH="path/to/sdk"
python3 $SDK_PATH/sw/scenario-runner/scripts/build.py -j $(nproc) \
    --flatbuffers-path ${SDK_PATH}/dependencies/flatbuffers \
    --argparse-path ${SDK_PATH}/dependencies/argparse \
    --json-path ${SDK_PATH}/dependencies/json \
    --vulkan-headers-path ${SDK_PATH}/dependencies/Vulkan-Headers \
    --glslang-path ${SDK_PATH}/dependencies/glslang \
    --dxc-path ${SDK_PATH}/dependencies/DirectXShaderCompiler \
    --spirv-headers-path ${SDK_PATH}/dependencies/SPIRV-Headers \
    --spirv-tools-path ${SDK_PATH}/dependencies/SPIRV-Tools \
    --vgf-lib-path ${SDK_PATH}/sw/vgf-lib \
    --gtest-path ${SDK_PATH}/dependencies/googletest \
    --target-platform aarch64

(Experimental) To cross compile for Android™, and build an APK package, you must target Android™, provide the Android™ CMake toolchain, and request the apk package type.

SDK_PATH="path/to/sdk"
ANDROID_NDK_HOME="path/to/android-ndk"
python3 ${SDK_PATH}/sw/scenario-runner/scripts/build.py -j $(nproc) \
    --target-platform android \
    --cmake-toolchain-for-android ${ANDROID_NDK_HOME}/build/cmake/android.toolchain.cmake \
    --flatbuffers-path ${SDK_PATH}/dependencies/flatbuffers \
    --argparse-path ${SDK_PATH}/dependencies/argparse \
    --json-path ${SDK_PATH}/dependencies/json \
    --vulkan-headers-path ${SDK_PATH}/dependencies/Vulkan-Headers \
    --glslang-path ${SDK_PATH}/dependencies/glslang \
    --dxc-path ${SDK_PATH}/dependencies/DirectXShaderCompiler \
    --spirv-headers-path ${SDK_PATH}/dependencies/SPIRV-Headers \
    --spirv-tools-path ${SDK_PATH}/dependencies/SPIRV-Tools \
    --vgf-lib-path ${SDK_PATH}/sw/vgf-lib \
    --package-type apk

The APK build uses the gradle command, so Gradle 8.4 or later must be available on your PATH. The Android™ SDK installation pointed to by ANDROID_HOME should include build-tools;34.0.0 and platforms;android-34, or other compatible versions.

To build with RenderDoc support, add --enable-rdoc. If RenderDoc is installed outside the default search locations, also pass --renderdoc-path <renderdoc-install-root>.

To enable and run tests, use the --test flag. To lint the tests, use the --lint flag. To enable tests and documentation building python dependencies must be installed:

pip install -r requirements.txt
pip install -r tooling-requirements.txt

To build the documentation, use the --doc flag. To build the documentation, you must have sphinx and doxygen installed on your machine.

You can install the build artifacts for this project into a specified location. To install the build artifacts, pass the --install option with the required path.

To create an archive with the build artifacts, you must add --package-type. The archive is stored in the provided location.

For more command line options, consult the program help:

python3 $SDK_PATH/sw/scenario-runner/scripts/build.py --help

Usage

To run a scenario file, use the following command:

./scenario-runner --scenario ${SCENARIO_JSON_FILE}

Where:

• --scenario: File to load the scenario from. The file must be in JSON format. If the resources in the SCENARIO_JSON_FILE are not specified with absolute paths, their relative paths will be resolved against the parent directory of the SCENARIO_JSON_FILE.

For more details, see the help output:

./scenario-runner --help

For the Android APK package, install the generated APK and start the foreground service with adb:

adb install -r build/scenario-runner-debug.apk

adb shell am start-foreground-service \
    -n com.arm.ai_ml_sdk_scenario_runner/.Main \
    --esa args --scenario,/data/user/0/com.arm.ai_ml_sdk_scenario_runner/scenario.json

PyPI

The ML SDK Scenario Runner is available on PyPI as the ai-ml-sdk-scenario-runner package.

Install:

pip install ai-ml-sdk-scenario-runner

Known Limitations

• Resources created with Optimal tiling cannot be used with memory aliasing.
• HLSL lacks support for VK_ARM_tensor at the moment, so tensor resources cannot be used with HLSL shaders.

License

The ML SDK Scenario Runner is distributed under the software licenses in LICENSES directory.

Trademark notice

Arm® is a registered trademarks of Arm Limited (or its subsidiaries) in the US and/or elsewhere.

Khronos®, Vulkan® and SPIR-V™ are registered trademarks of the Khronos® Group.