ethos-u-vela 4.3.0

Neural network model compiler for Arm Ethos-U NPUs

Vela

Vela is a compiler for Arm Ethos-U NPU edge AI devices. It is used to convert neural networks defined in TOSA or TensorFlow Lite/LiteRT format into Ethos-U command streams.

The input network must have quantised activations and quantised weights in order to be accelerated by the Ethos-U.

More details about Ethos-U devices:

• Ethos-U55, product and developer information
• Ethos-U65, product and developer information
• Ethos-U85, product and developer information

Installation

Vela runs on:

• Apple macOS
• Linux
• Microsoft Windows

The simplest way to obtain Vela is to install it directly from PyPi. The following will install the latest released version:

pip3 install ethos-u-vela

Requirements

If your system does not match one of the pre-built binary wheels on PyPi then the installation process will attempt to build Vela from its source code. To do this it requires the following:

• Development version of Python 3 containing the Python/C API header files
  • e.g. apt install python3.10-dev
• C99 and C++17 capable compiler and toolchain
  • For Apple macOS; XCode or Command Line Tools are recommended, see https://developer.apple.com/xcode/resources/ or xcode-select --install
  • For Linux operating systems; a GNU or Clang toolchain is recommended, e.g. apt install build-essential
  • For Microsoft Windows 10; the Microsoft Visual C++ 14.2 Build Tools are recommended, see https://wiki.python.org/moin/WindowsCompilers
• CMake
  • See https://cmake.org/download/

Manually Building

The source code is available from Arm's GitLab Instance, see Building From Source for more details.

Getting Started

Once installed, Vela can be run from the command line by specifying just two settings:

• Target Ethos-U device; --accelerator-config
• Neural network file; .tflite or .tosa

vela --accelerator-config ethos-u55-128  my_network.tflite

However, it is highly recommended that you specify some additional settings that describe the Ethos-U system and what optimisation strategy the compiler should use. The three additional settings are:

• System configuration; --system-config
• Memory mode; --memory-mode
• Optimisation strategy; --optimise

The system configuration and memory mode refer to definitions in a configuration file. These are specific to the target system. However an example configuration (Arm/vela.ini) containing some generic reference systems is provided as part of the installation.

The optimisation strategy indicates whether the compiler should minimise inference time or runtime memory usage.

See CLI Options for more information.

Example of how to set the system configuration and memory mode:

vela --config Arm/vela.ini --system-config Ethos_U85_SYS_DRAM_High --memory-mode Dedicated_Sram_384KB --accelerator-config ethos-u85-256  my_network.tosa

Example of how to set the optimisation strategy:

vela --optimise Size --accelerator-config ethos-u55-64  my_network.tflite

Command to list all known configuration files:

vela --list-config-files

Command to list all configurations in a configuration file:

vela --list-configs Arm/vela.ini

Output

The result of the compilation is an optimised network in either TFLite or Raw format depending upon the input network. This can be overridden using the --output-format option.

TFLite output contains TensorFlow Lite Custom operators for those parts of the network that can be accelerated by the Ethos-U NPU. Parts of the network that cannot be accelerated are left unchanged.

Raw output contains the command stream and weight data required to run Ethos-U parts of the optimised network. This is stored in .npz format. See the Output format CLI Option for more information. This does not contain CPU parts of the network and so cannot be used for converting TFLite to TOSA.

Warnings

It is important to thoroughly review and understand all warning messages generated by the compiler as they could indicate that not all operators were optimally converted to run on the Ethos-U.

Performance information

A performance estimation summary is reported after compilation. The cycle and bandwidth numbers should not be taken as accurate representations of real performance numbers and they should not be compared against those from other compilations that use different settings or configurations. The numbers reported allow the compiler to make its optimisation decisions only. For accurate performance numbers the network should be run and profiled on an FPGA. For approximate performance numbers the network can be run on a Fixed Virtual Platform (FVP) Model.

Additional Vela Information

• External APIs
• Bug Reporting
• Building From Source
• Contributing
• CLI Options
• Debug Database
• Performance Estimations
• Operator Support
• Changelog
• Security Vulnerabilities
• Testing

Inclusive language commitment

This product conforms to Arm’s inclusive language policy and, to the best of our knowledge, does not contain any non-inclusive language. If you find something that concerns you then email [e-mail].

License

Vela is licensed under Apache License 2.0.