Mobilint SDK qb Runtime Library : qbruntime

qbruntime is runtime library for Mobilint NPU products. Runtime Library is required to run inference on Mobilint NPUs using precompiled or custom models.

IMPORTANT: Please refer to official documentation for installing driver before using runtime library.

Prerequisites

Currently, pip installation is limited to below Ubuntu environments. For other environment, please refer to official documentation.

• Ubuntu: 20.04 / 22.04 / 24.04
• Python: 3.8 / 3.9 / 3.10 / 3.11 / 3.12
• numpy

Installation

You may install python runtime library with following command

pip install mobilint-qb-runtime

For C++ library installation or downloading library from mobilint distribution site, please refer to official documentation.

Inference process

Running inference on Mobilint's NPU using the qbruntime runtime library involves four essential steps:

1. Initialize the NPU device.
2. Load the compiled model (MXQ file).
3. Upload model to the NPU device.
4. Run inference with user input.

Step 1: Accelerator

The Accelerator object represents the NPU device to be used. It abstracts a single device identified by the number appended to the device name (e.g., /dev/aries0 -> acc1, /dev/aries2 -> acc2). If no specific number is provided, NPU device 0 is used by default.

acc = qbruntime.Accelerator(0)

Step 2: Model

The Model object represents a model contained in an MXQ file. Upon creation, it immediately reads the MXQ file and stores the relevant information.

model = qbruntime.Model(MXQ_FILE_PATH)

Step 3: Pass Model information to the Accelerator

Pass the information with the {doxylink}mobilint::Model::launch(Accelerator & acc) method of Model object. It then uses the Accelerator object to run inference using this model.

Step 4: Run inference using passed Model information and input data

Get the input data and run inference using the mobilint::Model::infer() method of Model object.

model = qbruntime.Model(MXQ_FILE_PATH)
result = model.infer([INPUT])

Inference Scenario

## Python example
import qbruntime

MXQ_PATH = "path/to/mxq.mxq"

acc = qbruntime.Accelerator()               ## Step 1
model = qbruntime.Model(MXQ_PATH)           ## Step 2
model.launch(acc)                           ## Step 3

## Some preprocessing for input data

result = model.infer(preprocessed_input)    ## Step 4