Skip to main content

A collection of models for HPC used in NWPC.

Project description

nwpc-hpc-model

Build Status codecov

A key-value model for HPC resources in NWPC. Including models for:

  • LoadLeveler's llq -l query
  • Slurm's sinfo and squeue -o %all query
  • Disk usage and disk space query

Installation

Download source code from Github releases or get the latest code from Github repo.

Run pip install . to install.

Getting started

The following example uses nwpc-hpc-model to extract job id and job owner from a llq -l query.

A config YAML file is used to create query categories.

category_list:
  -
    id: "llq.id"
    display_name: "Id"
    label: "Job Step Id"
    record_parser_class: "DetailLabelParser"
    record_parser_arguments:
      - "Job Step Id"
    value_saver_class: "StringSaver"
    value_saver_arguments: []
  -
    id: "llq.owner"
    display_name: "Owner"
    label: "Owner"
    record_parser_class: "DetailLabelParser"
    record_parser_arguments:
      - "Owner"
    value_saver_class: "StringSaver"
    value_saver_arguments: []

First create QueryCategoryList according to the config json file.

from nwpc_hpc_model.workload.loadleveler import QueryCategoryList, \
    QueryCategory, record_parser, value_saver
import yaml

with open('config_file_path', 'r') as f:
    config = yaml.load(f)

category_list = QueryCategoryList()
for an_item in config['category_list']:
    category = QueryCategory(
        category_id=an_item['id'],
        display_name=an_item['display_name'],
        label=an_item['display_name'],
        record_parser_class=getattr(record_parser, an_item['record_parser_class']),
        record_parser_arguments=tuple(an_item['record_parser_arguments']),
        value_saver_class=getattr(value_saver, an_item['value_saver_class']),
        value_saver_arguments=tuple(an_item['value_saver_arguments'])
    )
    category_list.append(category)

Get llq -l command output.

import subprocess

command = "/usr/bin/llq -l"
pipe = subprocess.Popen([command], stdout=subprocess.PIPE, shell=True)
output = pipe.communicate()[0]
output_string = output.decode()
output_lines = output_string.split("\n")

Build QueryModel from QueryCategoryList

from nwpc_hpc_model.workload.loadleveler import LoadLevelerQueryModel

model = LoadLevelerQueryModel.build_from_category_list(output_lines, category_list)

model contains data of all categories in the config file.

Test

Use pytest to run all tests.

License

Copyright © 2016-2019, Perilla Roc.

nwpc-hpc-model is licensed under The MIT License.

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Files for nwpc-hpc-model, version 0.3.2
Filename, size File type Python version Upload date Hashes
Filename, size nwpc_hpc_model-0.3.2-py3-none-any.whl (40.0 kB) File type Wheel Python version py3 Upload date Hashes View hashes
Filename, size nwpc-hpc-model-0.3.2.tar.gz (17.6 kB) File type Source Python version None Upload date Hashes View hashes

Supported by

Elastic Elastic Search Pingdom Pingdom Monitoring Google Google BigQuery Sentry Sentry Error logging AWS AWS Cloud computing DataDog DataDog Monitoring Fastly Fastly CDN SignalFx SignalFx Supporter DigiCert DigiCert EV certificate StatusPage StatusPage Status page