A collection of models for HPC used in NWPC.
Project description
# nwpc-hpc-model
[![Build Status](https://travis-ci.org/perillaroc/nwpc-hpc-model.svg?branch=master)](https://travis-ci.org/perillaroc/nwpc-hpc-model)
[![codecov](https://codecov.io/gh/perillaroc/nwpc-hpc-model/branch/master/graph/badge.svg)](https://codecov.io/gh/perillaroc/nwpc-hpc-model)
A collection of models for HPC used in NWPC. Including models for:
* LoadLeveler's `llq -l` query
* Slurm's `sinfo` and `squeue -o %all` query
* Disk usage and disk space query
## Installation
Download source code from Github releases or get the latest code from Github repo.
Run `pip install .` to install.
## Getting start
The following example uses `nwpc-hpc-model` to extract job id and job owner from a `llq -l` query.
A config YAML file is used to create query categories.
```yaml
category_list:
-
id: "llq.id"
display_name: "Id"
label: "Job Step Id"
record_parser_class: "DetailLabelParser"
record_parser_arguments:
- "Job Step Id"
value_saver_class: "StringSaver"
value_saver_arguments: []
-
id: "llq.owner"
display_name: "Owner"
label: "Owner"
record_parser_class: "DetailLabelParser"
record_parser_arguments:
- "Owner"
value_saver_class: "StringSaver"
value_saver_arguments: []
```
First create `QueryCategoryList` according to the config json file.
```python
from nwpc_hpc_model.workload.loadleveler import QueryCategoryList, \
QueryCategory, record_parser, value_saver
import yaml
with open('config_file_path', 'r') as f:
config = yaml.load(f)
category_list = QueryCategoryList()
for an_item in config['category_list']:
category = QueryCategory(
category_id=an_item['id'],
display_name=an_item['display_name'],
label=an_item['display_name'],
record_parser_class=getattr(record_parser, an_item['record_parser_class']),
record_parser_arguments=tuple(an_item['record_parser_arguments']),
value_saver_class=getattr(value_saver, an_item['value_saver_class']),
value_saver_arguments=tuple(an_item['value_saver_arguments'])
)
category_list.append(category)
```
Get `llq -l` command output.
```python
import subprocess
command = "/usr/bin/llq -l"
pipe = subprocess.Popen([command], stdout=subprocess.PIPE, shell=True)
output = pipe.communicate()[0]
output_string = output.decode()
output_lines = output_string.split("\n")
```
Build `QueryModel` from `QueryCategoryList`
```python
from nwpc_hpc_model.workload.loadleveler import LoadLevelerQueryModel
model = LoadLevelerQueryModel.build_from_category_list(output_lines, category_list)
```
`model` contains data of all categories in the config file.
## Test
Use `pytest` to run all tests.
## License
Copyright © 2016-2019, Perilla Roc.
`nwpc-hpc-model` is licensed under [The MIT License](https://opensource.org/licenses/MIT).
[![Build Status](https://travis-ci.org/perillaroc/nwpc-hpc-model.svg?branch=master)](https://travis-ci.org/perillaroc/nwpc-hpc-model)
[![codecov](https://codecov.io/gh/perillaroc/nwpc-hpc-model/branch/master/graph/badge.svg)](https://codecov.io/gh/perillaroc/nwpc-hpc-model)
A collection of models for HPC used in NWPC. Including models for:
* LoadLeveler's `llq -l` query
* Slurm's `sinfo` and `squeue -o %all` query
* Disk usage and disk space query
## Installation
Download source code from Github releases or get the latest code from Github repo.
Run `pip install .` to install.
## Getting start
The following example uses `nwpc-hpc-model` to extract job id and job owner from a `llq -l` query.
A config YAML file is used to create query categories.
```yaml
category_list:
-
id: "llq.id"
display_name: "Id"
label: "Job Step Id"
record_parser_class: "DetailLabelParser"
record_parser_arguments:
- "Job Step Id"
value_saver_class: "StringSaver"
value_saver_arguments: []
-
id: "llq.owner"
display_name: "Owner"
label: "Owner"
record_parser_class: "DetailLabelParser"
record_parser_arguments:
- "Owner"
value_saver_class: "StringSaver"
value_saver_arguments: []
```
First create `QueryCategoryList` according to the config json file.
```python
from nwpc_hpc_model.workload.loadleveler import QueryCategoryList, \
QueryCategory, record_parser, value_saver
import yaml
with open('config_file_path', 'r') as f:
config = yaml.load(f)
category_list = QueryCategoryList()
for an_item in config['category_list']:
category = QueryCategory(
category_id=an_item['id'],
display_name=an_item['display_name'],
label=an_item['display_name'],
record_parser_class=getattr(record_parser, an_item['record_parser_class']),
record_parser_arguments=tuple(an_item['record_parser_arguments']),
value_saver_class=getattr(value_saver, an_item['value_saver_class']),
value_saver_arguments=tuple(an_item['value_saver_arguments'])
)
category_list.append(category)
```
Get `llq -l` command output.
```python
import subprocess
command = "/usr/bin/llq -l"
pipe = subprocess.Popen([command], stdout=subprocess.PIPE, shell=True)
output = pipe.communicate()[0]
output_string = output.decode()
output_lines = output_string.split("\n")
```
Build `QueryModel` from `QueryCategoryList`
```python
from nwpc_hpc_model.workload.loadleveler import LoadLevelerQueryModel
model = LoadLevelerQueryModel.build_from_category_list(output_lines, category_list)
```
`model` contains data of all categories in the config file.
## Test
Use `pytest` to run all tests.
## License
Copyright © 2016-2019, Perilla Roc.
`nwpc-hpc-model` is licensed under [The MIT License](https://opensource.org/licenses/MIT).
Project details
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
nwpc-hpc-model-0.3.1.tar.gz
(16.9 kB
view hashes)
Built Distribution
Close
Hashes for nwpc_hpc_model-0.3.1-py3-none-any.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 9cbf14159ac756172768abd785afac469757adc0b1853a8a3111d6de239d0d17 |
|
MD5 | 4b4d530500a44c550e62cd4dc50525f1 |
|
BLAKE2b-256 | eda7d12485bc268cacd2edd5b7edd9c80c6f712b19c9b6d1f53d1a4eae05151a |