ibm-aiu-smi 1.3.0

A tool to collect performance metrics from IBM Spyre during an execution of AI workload

aiu-smi

aiu-smi is a tool to collect performance metrics during an execution of AI workload and print them or save them to a file for further analyses, like nvidia-smi dmon. It uses AIU's H/W performance counter to capture the performance metrics, as well as software counters for full-stack performance analysis. This version uses new API to access metric files, served by spyremetrics repo.

How to install

Prerequisite: Install spyremetrics package

spyremetrics package is a Python library to access metric files created by the Spyre runtime software. We are preparing to publish this package from PyPI.

Meanwhile, please clone spyre-metrics-api repo and install it as:

git clone [e-mail]:ibm-aiu/spyre-metrics-api.git
cd spyre-metrics-api
pip install --editable .

Install aiu-smi package

We are also preparing to make aiu-smi package available from PyPI.

Meanwhile, install it from GHE repo:

git clone [e-mail]:ibm-aiu/aiu-smi.git
cd aiu-smi
pip install --editable .

PF-mode support: Install aiu-monitor RPM package

NOTE: This step is only needed to use aiu-smi in PF mode.

In PF mode, AIU runtime software creates performance metric files in the old format. spyremetrics library uses a shared library aiu-monitor.so to read data from metric files in the old format. Use aiu-monitor RPM package to install the shared library.

$ rpm -i ibm-aiu-monitor-1.3.0-1.x86_64.rpm
$ pip3 install aiu-smi

TIP Installing an RPM package usually needs root access. To install by a general user, use following options of rpm command:

$ rpm -i --nodeps --dbpath=$HOME/.local/lib/rpm --relocate /usr/lib=$HOME/.local/lib \
    --relocate /opt/ibm/spyre/aiu-monitor=$HOME/.local --badreloc ibm-aiu-monitor-1.3.0-1.x86_64.rpm
$ ls $HOME/.local/lib/
libaiumonitor.cpython-312-x86_64-linux-gnu.so  rpm/

# Specify --dbpath option to access the user-installed package info
$ rpm -q --dbpath=$HOME/.local/lib/rpm ibm-aiu-monitor
ibm-aiu-monitor-1.3.0-1.x86_64

How to use aiu-smi

Overview of operations

As shown in the following figure, some environment variables need to be set in each of two terminals: one for running workload and the other for running aiu-smi:

Note on VF mode

In VF mode, performance counters are configured when PFs are initialized (typically when the system starts up) and Spyre workload users cannot change the settings, unlike in PF mode. Since the older default configurations of performance counters are not compatible with what aiu-smi expects, the output of aiu-smi will be incorrect, unless the default system config is updated. Specifically, with the old configs, rdmem, wrmem, and rxpci become always 0, and txpci may show smaller values than expected. busy% may be inaccurate because it is derived from rdmem, wrmem, rxpci, and txpci. Sample output showing the difference between old and updated configs is here.

For OpenShift environment, Spyre operator version 1.3 or later will support the new default that is compatible with aiu-smi.

Setting up environment variables

Mandatory settings

1. aiu-smi terminal

   There is no mandatory environment variable to run aiu-smi, but the optional setting below is recommended.

2. Workload terminal – tell Spyre driver to create metric files

   For PF mode, if installed from the wheel package:

   cp <cloned-aiu-smi>/etc/senlib_config_aiusmi.json $HOME/.senlib.json
   

   If you already have $HOME/.senlib.json, merge the settings for aiu-smi into the existing config:

   mv $HOME/.senlib.json $HOME/.senlib.json.orig
   jq -s add $HOME/.senlib.json.orig <cloned-aiu-smi>/etc/senlib_config_aiusmi.json > $HOME/.senlib.json
   

   For VF mode,

   export AIUPTI_ENABLE_METRICS=true
   

The default metric file format in VF mode as of June 16, 2026 is the old format. To create a metric file in the new format, export following environment variable, too:

export AIUPTI_USE_NEW_FORMAT=true

This setting will not be needed when the default is changed to the new format.

Optional settings to enable actmem and peakmem metrics (for torch-spyre only)

Workload terminal – Enable memory tracking in Flex:

export FLEX_ENABLE_MEMORY_TRACKING=1

Note on memory tracking files

The memory values are stored in `/tmp/memtrack/memory_{device_ID}.log` file by default. This can be changed by setting the `MEMORY_TRACKING_FOLDER` env variable to the same value, in *both* the workload and aiu-smi terminals. If `MEMORY_TRACKING_FOLDER` is set to `xyz`, the file which the code looks for would be `xyz/memory_{device_ID}.log`, where the device_ID is filled by the code using the LOCAL_RANK for each available device.

Run AIU workload with enabling performance counters

senlib loads config files during its initialization phase. In PF mode, you can tell the path of your senlib config file to senlib by copying configs/senlib_config_aiusmi.json as ~/.senlib.json.

In VF mode, flex loads libaiupti.so and starts the metrics sampler thread by setting an environment variable AIUPTI_ENABLE_METRICS=true. libaiupti creates a metrics file /tmp/metrics.<busid>, in the same manner as PF mode.

Until the new format becomes the default, export AIUPTI_USE_NEW_FORMAT=true is also needed.

Note on modifying `senlib_config_aiusmi.json` file

Current version of `aiu-smi` can collect metrics from multiple AIU devices. For multi-aiu runs, make sure the senlib's metric file path contains a magic string `%BUSID` to avoid overwriting the same metric file by multiple processes. The magic string will be replaced by the actual PCIe bus ID when senlib creates a metric file.

{
  "METRICS": {
    "general": {
      "enable": true,
      "path": "/tmp/metrics.%BUSID",
    }
  }
}

Run aiu-smi to print performance summary

Simultaneously run aiu-smi in another terminal. Note that this command needs to run in the same AIU container as the AIU workload.

In the case of distributed workload launched by torchrun, the Spyre driver layer allocates an AIU device for each of child processes based on the hardware configuration specified by the OpenShift AIU operator. This configuration is shown in .GENERAL.sen_busid[] of /etc/aiu/senlib_config.json and copied to environment variables PCIDEVICE_IBM_COM_AIU_PF and/or AIU_WORLD_RANK_0, AIU_WORLD_RANK_1, etc.

aiu-smi shows a list of metric files for each device with ID, and then periodically show metric data. Each line shows the metrics of a single device (for the ID at the first column). For example, if the container has two devices. aiu-smi prints two lines every second.

$ echo $PCIDEVICE_IBM_COM_AIU_PF
0000:13:00.0,0000:12:00.0
$ env | grep AIU_WORLD_RANK_
AIU_WORLD_RANK_0=0000:13:00.0
AIU_WORLD_RANK_1=0000:12:00.0
$ /tmp/aiu-perf-toolkit/bin/aiu-smi  # or $HOME/.local/bin/aiu-smi
#MetricFiles
# 0 /tmp/metrics.0000:13:00.0
# 1 /tmp/metrics.0000:12:00.0
#ID Date      Time     hostcpu hostmem    pwr  gtemp   busy    rdmem    wrmem    rxpci    txpci   rdrdma   wrrdma   pt_act
#   YYYYMMDD  HH:MM:SS       %       %      W      C      %     GB/s     GB/s     GB/s     GB/s     GB/s     GB/s        %
  0 20250617  21:14:01   276.0     4.9      -      -      -        -        -        -        -        -        -        -
  1 20250617  21:14:01   276.0     4.9      -      -      -        -        -        -        -        -        -        -
  0 20250617  21:14:02  1245.3     4.9      -      -      -        -        -        -        -        -        -        -
  1 20250617  21:14:02  1245.3     4.9      -      -      -        -        -        -        -        -        -        -
  ...
  0 20250617  21:14:06    34.1     5.0      -      -      -        -        -        -        -        -        -        -
  1 20250617  21:14:06    34.1     5.0    0.2   49.4      0    0.000    0.000    0.000    0.000    0.000    0.000     0.00
  0 20250617  21:14:07  1068.8     5.0   20.8   50.0      1    0.001    0.000    0.001    0.000    0.000    0.000     0.00
  1 20250617  21:14:07  1068.8     5.0   21.9   49.1      0    0.000    0.000    0.000    0.000    0.000    0.000     0.00
  0 20250617  21:14:08  1216.7     5.0   21.6   50.0      0    0.000    0.000    0.000    0.000    0.000    0.000     0.00
  1 20250617  21:14:08  1216.7     5.0   21.8   49.4      0    0.000    0.000    0.000    0.000    0.000    0.000     0.00
  0 20250617  21:14:09   464.4     5.0   23.2   50.6     38    7.631    0.035    0.032    0.004    0.007    0.000     0.00
  1 20250617  21:14:09   464.4     5.0   23.2   49.1     39    7.633    0.037    0.032    0.004    0.007    0.000     0.00
  0 20250617  21:14:10   295.1     5.0   26.7   50.6    100   22.767    0.103    0.003    0.013    0.021    0.000     0.00
  1 20250617  21:14:10   295.1     5.0   26.3   49.4    100   22.753    0.111    0.003    0.013    0.021    0.000     0.00
  0 20250617  21:14:11   326.6     5.0   24.5   50.0     84   18.431    0.084    0.002    0.011    0.017    0.000     0.00
  1 20250617  21:14:11   326.6     5.0   24.7   49.1     84   18.436    0.089    0.002    0.011    0.017    0.000     0.00
  0 20250617  21:14:12   345.6     4.9   22.1   50.0      0    0.000    0.000    0.000    0.000    0.000    0.000     0.00
  1 20250617  21:14:12   345.6     4.9   21.8   49.4      0    0.000    0.000    0.000    0.000    0.000    0.000     0.00

Description of metrics

Metric group	Metric name	Description
	hostcpu	CPU usage in the host (100% = 1 core, not container-aware)
	hostmem	Memory usage in the host (not container-aware)
	pwr	Card power consumption
	gtemp	Maximum temperature of nine sensors
	busy	Time-based card utilization (number of 10ms internal sampling intervals that perform memory and/or data transfer activities)
D	rdmem, wrmem	Average bandwidth to read/write from device memory to RaPid cores and vice versa
D	rxpci, txpci	Average bandwidth to read/write from/to host PCIe
D	rdrdma, wrrdma	Average bandwidth to read/write data via AIU-to-AIU RDMA
U	peakmem, actmem	(torch-spyre only) Peak and actual device memory usage (in GiB) allocated by flex for the workload.
P	pt_active	Estimated PT-array utilization calculated from power consumption, based on empirical measurement of their correlation

Notes on metric values

• Until senlib (device driver) initializes AIU, AIU-related metrics (i.e., metrics except hostcpu and hostmem) are shown as -.
• After finishing an AIU workload, the same metric values are printed repeatedly for a few seconds because metric files are not updated any more, and then - are shown again for AIU-related metrics when aiu-smi detects metric files became stale.

Command line options

aiu-smi --help shows usage:

$ /tmp/aiu-perf-toolkit/bin/aiu-smi --help

Usage: aiu-smi [dmon] [-s] [-i <device-ids>] [-g <metric-list>] [-d <delay>] [-f <output-file>]
                      [--idle-power <watts>] [--llm-type <model-type>] [-v] [<metric-files>...]

options:
  -s | --csv:           Output csv format
  -i | --id:            Comma-separated device IDs to monitor (e.g., 0,1,2) [default=all]
  -g | --metric-groups: Select metric groups to print [default=DP]:
                          D=device Data rate, U=actual and peak device memory Usage (torch-spyre only),
                          P=Pt_act, A=All metrics.
                        Always print dev. ID, timestamp, host cpu/mem, power, temperature, and busy.
  -d | --delay:         Console summary interval in seconds [default=1sec]. Accept floating-point value.
                        Minimum is 0.1 sec on s390x console or VF mode, otherwise 0.01 sec.
  -f | --filename:      Log to a specified file, rather than to stdout
  --idle-power          Idle power baseline (W) for pt_act estimation [default=20.0]
  --llm-type            LLM model type for pt_act estimation: encoder|decoder [default=decoder]
  -v | --version:       Show version
  -h | -? | --help:     Show help message

Note: Currently, metric group 'U' implies 'D' because of implementation issue. Note: aiu-smi also takes options from an environment variable AIUSMI_OPTS. Command line options supersedes the options specified by the environment variable.

Monitoring specific devices

Monitor specific devices using -i or --id:

$ aiu-smi -i 0,2              # Devices 0 and 2
$ aiu-smi --id=0,1 -d 2       # With 2-second interval

Format: Comma-separated IDs (e.g., 0,1,2). Supports -i 0,1 or --id=0,1.

Example:

$ aiu-smi -i 0,2
INFO: Monitoring 2 device(s): 0 2
#ID Date      Time     ...
  0 20250617  21:14:01 ...
  2 20250617  21:14:01 ...

IDs are zero-indexed from AIU_WORLD_RANK_* environment variables (or PCIDEVICE_IBM_COM_AIU_PF in OpenShift environments). Out-of-range IDs skipped with warning.

Note: aiu-smi removes an existing memory-mapped files if there is no process that opens the files because the existing file should be created by a workload ran in the past and it should not be needed any more. Be careful not to run aiu-smi after finishing the workload if you still need it, otherwise aiu-smi removes the memory-mapped file.

Additional Tools

PT Active Modeling Tools

This toolset processes FLEX job trace JSON files and builds predictive models for PT Active based on Power consumption. The workflow supports splitting a multi-job trace into individual job files for separate analysis, and automatically analyzes idle power characteristics.

See ../pt_active_model/README.md for usage details.

LLM Memory Calculator

For LLM inference workloads, the LLM Memory Calculator provides theoretical calculations to help validate aiu-smi measurements:

• Expected memory footprint of model weights and KV Cache
• Theoretical bandwidth requirements (validates rdmem/wrmem)

See ../llm_calc/README.md for usage details.