spyremetrics 0.5.0

Python library for reading Spyre metric files

IBM Spyre Metrics API

Overview

This repository provides API to access Spyre performance metric files. It supports both the old and new format.

MetricFile class corresponds to a metric file. It provides a static method open_metrics() to open metric files, with expanding %BUSID keyword. An iterator read_metrics() reads performance metrics from the file and returns a pair of MetricDataType object and the value read from the file. Use set_filters() to limit the types of metrics read from a file if needed. It receives a list of metric names. The default behavior without using set_filters() is to return all metrics stored in a metric file.

MetricDataType is a frozen data class object to show metadata of each metric data, such as its name and ID, data type, unit, and scaling. The ID is the value stored in metric files to specify the data type of each metric data. MetricDataType objects are singleton in a process.

A typical usage looks like:

from spyremetrics import MetricFile

# Expand "%BUSID" keyword and open metric files. Return a list of MetricFile objects.
metric_file_list = MetricFile.open_metrics(metric_file_path)
for mf in metric_file_list:
    mf.set_filters(['pwr', 'tempr', 'rdmem', 'wrmem', 'avgmem', 'peakmem'])
    for metric, val in mf.read_metrics():
        match metric.name:
            case 'pwr':     pwr     = val
            case 'tempr':   tempr   = val
            case 'rdmem':   rdmem   = val / 1024 / 1024  # convert to GiB
            case 'wrmem':   wrmem   = val / 1024 / 1024  # convert to GiB
            case 'avgmem':  avgmem  = val / 1024 / 1024  # convert to GiB
            case 'peakmem': peakmem = val / 1024 / 1024  # convert to GiB

Installation

Prerequisite

• Python >= 3.12
• numpy
• psutil >= 5.9.6

Install from package

pip3 install spyremetrics

Editable installation

# From the repository root
pip install -e .

Build and install wheel package

# Build a wheel package
$ python3 -m build
$ ls -1 dist
spyremetrics-0.5.0-py3-none-any.whl
spyremetrics-0.5.0.tar.gz

# Install the wheel package
$ pip3 install dist/spyremetrics-0.5.0-py3-none-any.whl

...
Successfully installed spyremetrics-0.5.0
$ pip3 list | grep spyremetrics
spyremetrics            0.5.0

Usage

Example Script

Run the example script to see the package in action:

# Need a copy of metric file in new format
python3 example.py tests/test_metric.bin

The sample metric file was generated by gen_test_metrics.sh:

scripts/gen_test_metrics.sh > tests/test_metric.bin

Iterating Over Metric Data Words

A typical usage to retrieve all available metrics:

from spyremetrics import metric_file

# Expand "%BUSID" keyword and open metric files. Return a list of MetricFile objects.
metric_file_list = MetricFile.open_metrics(metric_file_path)
for mf in metric_file_list:
    for metric, val in mf.read_metrics():
        match metric.name:
            case 'pwr':     pwr     = val
            case 'tempr':   tempr   = val
            case 'rdmem':   rdmem   = val / 1024 / 1024  # convert to GiB
            case 'wrmem':   wrmem   = val / 1024 / 1024  # convert to GiB
            case 'avgmem':  avgmem  = val / 1024 / 1024  # convert to GiB
            case 'peakmem': peakmem = val / 1024 / 1024  # convert to GiB

set_filters() restricts metric types to be loaded:

from spyremetrics import metric_file

metric_file_list = MetricFile.open_metrics(metric_file_path)
for mf in metric_file_list:
    mf.set_filters(['pwr', 'avgmem'])
    for metric, val in mf.read_metrics():
        match metric.name:
            case 'pwr':     pwr     = val
            case 'avgmem':  avgmem  = val / 1024 / 1024  # convert to GiB

Development

API classes

• Class: MetricFile
  • Represent a single metric file. The main class of this API. Use open_metrics() static method to open metric files, and read_metrics() to iterate through the data in the file. set_filters() selects metrics to be read from a metric file.
• Class: MetricSection
  • Represent a single section in a metric file. MetricFile.sections() iterates through the sections in a metric file and returns MetricSection object.
• Class: MetricDataType
  • Specify the data type and how to summarize the data. Its ID is the value stored in metric files. Each instance is singleton.
• Class: SectionType
  • Specify the section type. Its ID is the value stored in metric files. Each instance is singleton.
• Class: ValueType
  • Specify the data type and unit of a metric data. This ID is only referred from MetricDataType and never shown in metric files. Each instance is singleton.
• Class: SummarizerType
  • Specify how to summarize data if multiple words are stored in a metric file. This ID is only referred from MetricDataType and never shown in metric files. Each instance is singleton.

Definition of Metric type, Section type, etc. in section_types.json

Supported metric types, section types, etc. are defined in a JSON file: section_types.json. This is the master definition of the data supported by Spyre metric files.

This JSON file has five types of definitions:

• Section type: version
  • The version of this definition. Note that adding/deleting metric and section types will affect the layout of a metric file. The version must be updated in such case.
• Section type: section_type
  • List of supported section types.
• Section type: metric_type
  • List of supported metric types.
• Section type: value_type
  • List of definitions on the data types and units of values.
• Section type: summarizer_type
  • List of definitions how to summarize data if multiple data words are stored for a single metric type.

Converting section_types.json to generated_section_types.py

For reducing the time to start up, we provide generated source code in Python that are converted form the configuration JSON file spyremetrics/section_types.json as spyremetrics/generated_section_types.py in the GitHub repo.

When the JSON file is modified, the converted file needs to be re-generated by using convert_section_types.py:

# From the repository root
python3 scripts/convert_section_types.py --output-dir spyremetrics spyremetrics/section_types.json

Metric File Format

Brief description is in the beginning of spyremetrics/metric_file.py.

Type Annotations

All functions include type annotations for better IDE support and type checking.

License

Apache 2.0