Compspec Modules
A compspec (Composition spec) is a specification and model for comparing things. Compspec modules is a plugin for extraction of environment modules subsystem metadata about software installed on a system. This can be for the entire system or a user-specific install. The plugin exports json graph format (JGF) that can be used for scheduling and other purposes. To learn more:
- Compspec: the Python library that discovers and loads this plugin.
Note that I'm currently parsing environment modules, and very simply. We can extend to more complex / LMOD if needed.
Usage
Install compspec and the plugin here:
pip install compspec
pip install compspec-modules
Here is an easy way (in the devcontainer) to get a bunch of testing modules.
git clone https://github.com/shawfdong/modulefiles /tmp/modulefiles
export MODULEPATH=$MODULEPATH:/tmp/modulefiles
export PATH=~/.local/bin:$PATH
Then run an extraction for your environment modules:
compspec extract modules
To save to file:
compspec extract --outfile module-subsystem.json modules
You can see the example example/module-subsystem.json.
compspec-modules output
{
"graph": {
"nodes": {
"environment-modules0": {
"metadata": {
"type": "environment-modules",
"basename": "environment-modules",
"name": "environment-modules0",
"id": 0,
"uniq_id": 0,
"containment": {
"paths": "/environment-modules0"
},
"size": 1,
"unit": "",
"rank": 0,
"exclusive": false
},
"label": "environment-modules0"
},
"module1": {
"metadata": {
"type": "module",
"basename": "module",
"name": "module0",
"id": 1,
"uniq_id": 1,
"containment": {
"paths": "/environment-modules0/module0"
},
"size": 1,
"unit": "",
"rank": 0,
"exclusive": false,
"attributes": {
"name": "dot",
"software": "dot"
}
},
"label": "module1"
},
"module2": {
"metadata": {
"type": "module",
"basename": "module",
"name": "module1",
"id": 2,
"uniq_id": 2,
"containment": {
"paths": "/environment-modules0/module1"
},
"size": 1,
"unit": "",
"rank": 0,
"exclusive": false,
"attributes": {
"name": "module-info",
"software": "module-info"
}
},
"label": "module2"
},
"module3": {
"metadata": {
"type": "module",
"basename": "module",
"name": "module2",
"id": 3,
"uniq_id": 3,
"containment": {
"paths": "/environment-modules0/module2"
},
"size": 1,
"unit": "",
"rank": 0,
"exclusive": false,
"attributes": {
"name": "modules",
"software": "modules",
"version": "5.4.0"
}
},
"label": "module3"
},
"module4": {
"metadata": {
"type": "module",
"basename": "module",
"name": "module3",
"id": 4,
"uniq_id": 4,
"containment": {
"paths": "/environment-modules0/module3"
},
"size": 1,
"unit": "",
"rank": 0,
"exclusive": false,
"attributes": {
"name": "module-git",
"software": "module-git"
}
},
"label": "module4"
},
"module5": {
"metadata": {
"type": "module",
"basename": "module",
"name": "module4",
"id": 5,
"uniq_id": 5,
"containment": {
"paths": "/environment-modules0/module4"
},
"size": 1,
"unit": "",
"rank": 0,
"exclusive": false,
"attributes": {
"name": "use.own",
"software": "use.own"
}
},
"label": "module5"
},
"module6": {
"metadata": {
"type": "module",
"basename": "module",
"name": "module5",
"id": 6,
"uniq_id": 6,
"containment": {
"paths": "/environment-modules0/module5"
},
"size": 1,
"unit": "",
"rank": 0,
"exclusive": false,
"attributes": {
"name": "null",
"software": "null"
}
},
"label": "module6"
},
"module7": {
"metadata": {
"type": "module",
"basename": "module",
"name": "module6",
"id": 7,
"uniq_id": 7,
"containment": {
"paths": "/environment-modules0/module6"
},
"size": 1,
"unit": "",
"rank": 0,
"exclusive": false,
"attributes": {
"name": "python/intelpython2",
"software": "python"
}
},
"label": "module7"
},
"module8": {
"metadata": {
"type": "module",
"basename": "module",
"name": "module7",
"id": 8,
"uniq_id": 8,
"containment": {
"paths": "/environment-modules0/module7"
},
"size": 1,
"unit": "",
"rank": 0,
"exclusive": false,
"attributes": {
"name": "python/anaconda3",
"software": "python"
}
},
"label": "module8"
},
"module9": {
"metadata": {
"type": "module",
"basename": "module",
"name": "module8",
"id": 9,
"uniq_id": 9,
"containment": {
"paths": "/environment-modules0/module8"
},
"size": 1,
"unit": "",
"rank": 0,
"exclusive": false,
"attributes": {
"name": "python/anaconda2",
"software": "python"
}
},
"label": "module9"
},
"module10": {
"metadata": {
"type": "module",
"basename": "module",
"name": "module9",
"id": 10,
"uniq_id": 10,
"containment": {
"paths": "/environment-modules0/module9"
},
"size": 1,
"unit": "",
"rank": 0,
"exclusive": false,
"attributes": {
"name": "python/intelpython3",
"software": "python"
}
},
"label": "module10"
},
"module11": {
"metadata": {
"type": "module",
"basename": "module",
"name": "module10",
"id": 11,
"uniq_id": 11,
"containment": {
"paths": "/environment-modules0/module10"
},
"size": 1,
"unit": "",
"rank": 0,
"exclusive": false,
"attributes": {
"name": "python/rh-python36",
"software": "python"
}
},
"label": "module11"
},
"module12": {
"metadata": {
"type": "module",
"basename": "module",
"name": "module11",
"id": 12,
"uniq_id": 12,
"containment": {
"paths": "/environment-modules0/module11"
},
"size": 1,
"unit": "",
"rank": 0,
"exclusive": false,
"attributes": {
"name": "PrgEnv-intel/2019.1.053",
"software": "PrgEnv-intel",
"version": "2019.1.053"
}
},
"label": "module12"
},
"module13": {
"metadata": {
"type": "module",
"basename": "module",
"name": "module12",
"id": 13,
"uniq_id": 13,
"containment": {
"paths": "/environment-modules0/module12"
},
"size": 1,
"unit": "",
"rank": 0,
"exclusive": false,
"attributes": {
"name": "PrgEnv-pgi/llvm_18.10",
"software": "PrgEnv-pgi",
"version": "18.10"
}
},
"label": "module13"
},
...
},
"metadata": {
"install_name": "compat-experiment"
}
}
Development
If you open the Development container in VSCode, you'll find environment modules are installed:
$ module avail
This allows us to easily develop and test the compatibility plugin.
License
HPCIC DevTools is distributed under the terms of the MIT license. All new contributions must be made under this license.
See LICENSE, COPYRIGHT, and NOTICE for details.
SPDX-License-Identifier: (MIT)
LLNL-CODE- 842614
1 maintainer
Metadata
1 maintainer
Release files for compspec-modules 0.0.13
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | |
|---|---|---|---|
| compspec_modules-0.0.13.tar.gz | 10.6 kB | Details |
Built distribution (wheel)
| File | Interpreter | ABI | Platform | Reset |
|---|---|---|---|---|
| compspec_modules-0.0.13-py3-none-any.whl | Python 3 | none | any | Details |
Total release size: 19.2 kB
Release files / compspec_modules-0.0.13.tar.gz
| Download URL | compspec_modules-0.0.13.tar.gz |
|---|---|
| Size | 10.6 kB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
8dadf0fd09ea3f53d75026dc502d99bccdc7906b1be16bc487d0e306d13a3d36
|
|
BLAKE2b-256 checksum How to use checksums |
d95c938bab084d40ae6338d9921d2a010315992eb9663d25e510d502aaf9e02d
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/5.1.1 CPython/3.12.2
|
Release files / compspec_modules-0.0.13-py3-none-any.whl
| Download URL | compspec_modules-0.0.13-py3-none-any.whl |
|---|---|
| Size | 8.6 kB |
| Tags | Python 3 |
|
SHA-256 checksum How to use checksums |
e7d87323b0bff97ed63c476a9eabb1a54699b10179308210deb96331a8e58b92
|
|
BLAKE2b-256 checksum How to use checksums |
45478f45b02aeda2e6a1565ddc42322e6b9e334205085ff6f1f6f75b0bd2ecd1
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/5.1.1 CPython/3.12.2
|
