mlia 0.3.0

ML Inference Advisor

ML Inference Advisor

Introduction

This tool is used to help AI developers design and optimize neural network models for efficient inference on Arm® targets (e.g. Cortex®-M55 and Ethos™-U55/Ethos™-U65, Cortex®-M85 and Ethos™-U55) by enabling performance analysis and providing actionable advice early in the model development cycle. The final advice can cover the operator list, performance analysis and suggestions for model inference run on certain hardware before/after applying model optimization (e.g. pruning, clustering, etc.).

Prerequisites and dependencies

It is recommended to use virtual environments for MLIA installation, and a typical setup for MLIA requires:

• Ubuntu® 20.04.03 LTS (other OSs may work, the ML Inference Advisor has been tested on this one specifically)
• Python® >= 3.8
• Ethos™-U Vela dependencies (Linux® only)
  • For more details, please refer to the prerequisites of Vela

Backend installation

The ML Inference Advisor is designed to support multiple performance estimators (backends) that could generate performance analysis for individual types of hardware.

The backend command is used to manage the installation of new backends. The install sub-command can be used to either

• install a backend installed locally already (option --path) or
• (if available) automatically download the necessary components and dependencies, install them and configure them properly (option --download).

The usage is:

mlia backend install --help

and the result looks like:

positional arguments:

• name: Name of the backend to install

optional arguments:

• -h/--help: Show this help message and exit
• --path PATH: Path to the installed backend
• --download: Download and install a backend
• --noninteractive: Non interactive mode with automatic confirmation of every action

Example:

# Use this command to see what backends can be downloaded.
mlia backend install --download

After a successful installation of the backend(s), start using mlia in your virtual environment.

Please note: Backends cannot be removed once installed. Consider creating a new environment and reinstall backends when needed.

Using Corstone™-300

To install Corstone™-300 as a backend for Ethos™-U both options (--download and --path) can be used:

# To download and install Corstone-300 automatically
mlia backend install --download Corstone-300
# To point MLIA to an already locally installed version of Corstone-300
mlia backend install --path YOUR_LOCAL_PATH_TO_CORSTONE_300

Please note: Corstone™-300 used in the example above is available only on the Linux® platform.

Using Corstone™-310

Corstone™-310 is available as Arm® Virtual Hardware (AVH).

• For access to AVH for Corstone™-310 please refer to: https://developer.arm.com/Processors/Corstone-310
• Please use the examples of MLIA using Corstone™-310 here to get started: https://github.com/ARM-software/open-iot-sdk

Usage

After the initial setup, you can start the program by opening your terminal and typing the following command:

mlia [command] [arguments]

Choices of commands:

• "operators": show the model's operator list
• "optimization": run the specified optimizations
• "performance": measure the performance of inference on hardware
• "all_tests": have a full report

To get a list of all available options, use:

mlia --help

To get help on a specific command, use:

mlia [command] --help

Most commands accept the name of the target profile name as input parameter. There are a number of predefined profiles with following attributes:

+--------------------------------------------------------------------+
| Profile name  | MAC | System config               | Memory mode    |
+=====================================================================
| ethos-u55-256 | 256 | Ethos_U55_High_End_Embedded | Shared_Sram    |
+---------------------------------------------------------------------
| ethos-u55-128 | 128 | Ethos_U55_High_End_Embedded | Shared_Sram    |
+---------------------------------------------------------------------
| ethos-u65-512 | 512 | Ethos_U65_High_End          | Dedicated_Sram |
+--------------------------------------------------------------------+

Operators (ops)

Description

Prints the model's operator list.

Example

# List operator compatibility for a specific target profile
mlia operators --target-profile ethos-u55-256 ~/models/mobilenet_v1_1.0_224_quant.tflite

Arguments

TFLite model options

• model: Input model in TFLite format [required].

Target profile options

• --target-profile: Target profile that will set the target options such as target, MAC value, memory mode, etc ...
  • default: ethos-u55-256
  • options:
    • ethos-u55-256
    • ethos-u55-128
    • ethos-u65-512

Output options

• --output: Name of the file where the report will be saved. The report is also displayed the standard output, as plain text. Valid file extensions (formats) are {.txt,.json,.csv}, anything else will be formatted as plain text.

Optional arguments

• -h/--help: Show the general help document and exit.
• --supported-ops-report: Generate the SUPPORTED_OPS.md file in the current working directory and exit.

Performance (perf)

Description

Prints the model's performance statistics.

Example

# Explicitly specify backend(s) to use with --evaluate-on
mlia performance ~/models/mobilenet_v1_1.0_224_quant.tflite \
--evaluate-on "Vela" "Corstone-310"

Arguments

TFLite model options

• model: Input model in TFLite format [required].

Target profile options

• --target-profile: Target profile that will set the target options such as target, MAC value, memory mode, etc ...
  • default: ethos-u55-256
  • options:
    • ethos-u55-256
    • ethos-u55-128
    • ethos-u65-512

Output options

• --output: Name of the file where the report will be saved. The report is also displayed the standard output, as plain text. Valid file extensions (formats) are {.txt,.json,.csv}, anything else will be formatted as plain text.

Debug options

• --verbose: Produce verbose output (for debugging purposes).

optional arguments

• -h/--help: Show the general help document and exit.
• --supported-ops-report: Generate the SUPPORTED_OPS.md file in the current working directory and exit.

Model optimization (opt)

Description

Shows the performance improvements after applying optimizations to the model.

Example

# Custom optimization parameters: pruning=0.6, clustering=16
mlia optimization \
--optimization-type pruning,clustering \
--optimization-target 0.6,16 \
~/models/ds_cnn_l.h5

Arguments

Keras™ model options

• model: Input model in Keras™ (.h5 or SavedModel) format [required].

optimization options

• --optimization-type: Type of optimization to apply to the model [required].
  • options:
    • pruning
    • clustering
• --optimization-target: Target for optimization (for pruning this is sparsity between (0,1), for clustering this is the number of clusters (positive integer)) [required].
• --layers-to-optimize: Name of the layers to optimize (separated by space). Example: conv1 conv2 conv3
  • default: every layer

Target profile options

• --target-profile: Target profile that will set the target options such as target, MAC value, memory mode, etc ...
  • default: ethos-u55-256
  • options:
    • ethos-u55-256
    • ethos-u55-128
    • ethos-u65-512

Debug options

• --verbose: Produce verbose output (for debugging purposes).

optional arguments

• -h/--help: Show the general help document and exit.
• --supported-ops-report: Generate the SUPPORTED_OPS.md file in the current working directory and exit.

All tests (all)

Description

Generates a full report on the input model's operator list, runs the specified optimizations and lists the performance improvements.

Example

# Create full report and save it as JSON file
mlia all_tests --output ./report.json ~/models/ds_cnn_l.h5

Arguments

Keras™ model options

• model: Input model in Keras™ (.h5 or SavedModel) format [required].

Optimization options

• --optimization-type: List of the optimization types separated by comma
  • default: pruning, clustering
• --optimization-target: List of the optimization targets separated by comma, (for pruning this is sparsity between (0,1), for clustering this is the number of clusters (positive integer))
  • default: 0.5, 32

Target profile options

• --target-profile: Target profile that will set the target options such as target, MAC value, memory mode, etc ...
  • default: ethos-u55-256
  • options:
    • ethos-u55-256
    • ethos-u55-128
    • ethos-u65-512

Output options

• --output: Name of the file where the report will be saved. The report is also displayed the standard output, as plain text. Valid file extensions (formats) are {.txt,.json,.csv}, anything else will be formatted as plain text.

Debug options

• --verbose: Produce verbose output (for debugging purposes).

Optional arguments

• -h/--help: show this help message and exit

Resources

Additional useful information:

• Corstone™-300
• Corstone™-310

License

ML Inference Advisor is licensed under Apache License 2.0.

Trademarks and Copyrights

Arm®, Ethos™-U, Cortex®-M, Corstone™ are registered trademarks or trademarks of Arm® Limited (or its subsidiaries) in the U.S. and/or elsewhere. TensorFlow™ is a trademark of Google® LLC. Keras™ is a trademark by François Chollet. Linux® is the registered trademark of Linus Torvalds in the U.S. and elsewhere. Python® is a registered trademark of the PSF. Ubuntu® is a registered trademark of Canonical.