Skip to main content

Sonar: Tool to profile usage of HPC resources by regularly probing processes

Project description


Tool to profile usage of HPC resources by regularly probing processes using ps.


The code can do two things: take snapshots (sonar probe, typically every 20 minutes or so), and map them (sonar map, whenever you like) to applications/projects/users:

$ sonar --help

usage: sonar [-h]  ...

Tool to profile usage of HPC resources by regularly probing processes using ps.

optional arguments:
  -h, --help  show this help message and exit


    probe     Take a snapshot of the system. Run this on every node and often
              (e.g. every 20 minutes).
    map       Parse the system snapshots and map applications. Run this only
              once centrally and typically once a day.

Run sonar <subcommand> -h to get more information about subcommands.


Design goals and design decisions

  • Pip installable
  • Minimal overhead for recording
  • Super quick reporting and dashboard, both stdout and csv for web postprocessing
  • Can be used as health check tool

Why ps instead of top? We started using top but it turned out that top is dependent on locale, so it displays floats with comma instead of decimal point in many non-English locales. ps always uses decimal points. In addition, ps is (arguably) more versatile/configurable and does not print the header that top prints. All these properties make the ps output easier to parse than the top output.


Ideally install into a virtual environment:

$ pip install sonar

If you develop sonar, you can install like this:

$ git clone
$ cd sonar
$ virtualenv venv
$ source venv/bin/activate
$ pip install -r requirements.txt
$ flit install --symlink

How to analyze sonar logs

For this run sonar map which will go through the logs, and map processes to applications:

$ sonar map --input-dir /home/user/folder/with/logs

By default you will see data for the past 7 days. But you can change this:

$ sonar map --input-dir /home/user/folder/with/logs --num-days 300

Sonar uses the following mapping files:

The mapping files (string_map.txt and regex_map.txt) contain a space-separated (does not matter how many spaces) mapping from process to application.

You can use your own mapping files instead:

$ sonar map --input-dir /home/user/folder/with/logs \
            --str-map-file /home/user/my-own-mapping/string_map.txt \
            --re-map-file /home/user/my-own-mapping/regex_map.txt

You are welcome to use your own but encouraged to contribute mappings to

You can also export daily, weekly, and monthly CPU load percentages in CSV format for further postprocessing, e.g. using

$ sonar map --input-dir /home/user/folder/with/logs --export-csv daily
$ sonar map --input-dir /home/user/folder/with/logs --export-csv weekly --num-days 200

Taking snapshots with sonar probe

This is me running sonar probe on a compute node:

$ sonar probe --output-delimiter ","


The columns are: - time stamp - hostname - number of cores on this node - user - process - CPU percentage (this is a 20-core node) - memory used in MB - Slurm project - Slurm job ID - Number of CPUs requested by the job - Minimum size of memory requested by the job

By default they are tab-separated but here I chose to display the result comma-separated. You can also change cutoffs or ignore users to not measure the tool itself (sonar probe --help).

It can be useful to redirect the result to a file:

$ sonar probe >> /home/user/tmp/example.tsv

This is how it looks when I run sonar probe on my laptop (without Slurm):

$ sonar probe --output-delimiter ","


Running sonar probe on a cluster

We let cron execute a script every 20 minutes:

10,30,50 * * * * /global/work/sonar/sonar/

The script creates a list of active nodes and executes on all of these nodes:



# get list of all available nodes
/usr/bin/sinfo -h -r -o '%n' > ${SONAR_ROOT}/tmp/list-of-nodes 2> ${SONAR_ROOT}/tmp/list-of-nodes.err

# run sonar probe on all available nodes
/usr/bin/pdsh -w \^${SONAR_ROOT}/tmp/list-of-nodes ${SONAR_ROOT}/sonar/ >> ${SONAR_ROOT}/tmp/pdsh.log 2>> ${SONAR_ROOT}/tmp/pdsh.err

In we load the Python environment and wrap around sonar probe:

#!/usr/bin/env bash

source /global/work/sonar/python/environment
pyenv shell 3.6.7

source /global/work/sonar/sonar/venv/bin/activate
current_year=$(date +'%Y')
mkdir -p /global/work/sonar/probe-outputs/${current_year}
sonar probe --ignored-users root >> /global/work/sonar/probe-outputs/${current_year}/${HOSTNAME}.tsv

This produces ca. 10 MB data per day.

Project details

Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

sonar-0.5.0.tar.gz (26.1 kB view hashes)

Uploaded source

Built Distribution

sonar-0.5.0-py2.py3-none-any.whl (25.5 kB view hashes)

Uploaded py2 py3

Supported by

AWS AWS Cloud computing Datadog Datadog Monitoring Facebook / Instagram Facebook / Instagram PSF Sponsor Fastly Fastly CDN Google Google Object Storage and Download Analytics Huawei Huawei PSF Sponsor Microsoft Microsoft PSF Sponsor NVIDIA NVIDIA PSF Sponsor Pingdom Pingdom Monitoring Salesforce Salesforce PSF Sponsor Sentry Sentry Error logging StatusPage StatusPage Status page