cfin-hpc-tools 0.2.3b0

Some general grid-engine HPC tools, mainly geared for UCL systems but should be generalisable. Note this is not a UCL endorsed package.

Getting started with hpc-tools

version: 0.2.3b0

hpc-tools is a Python package with some additional bash scripts/configuration files designed to simplify submitting array jobs on a cluster. Whilst it is written with Grid Engine in mind it can probably be re-purposed for other systems (such as LSF) - although this has not been tested in any way. Also, this has only been tested on the University College London (UCL) systems, Myriad and the CS cluster.

There is online documentation for hpc-tools and offline PDF documentation can be downloaded here.

Installation instructions

At present, hpc-tools is undergoing development and no packages exist yet on PyPi. Therefore it is recommended that it is installed in either of the two ways listed below. First, clone this repository and then cd to the root of the repository.

Installation using pip

pip install cfin-hpc-tools

Installation using conda

A conda build is also available for Python v3.7, v3.8 and v3.9 on linux-64.

conda install -c cfin cfin-hpc-tools

Post Python install

After installation of hpc-tools, there are some manual setup steps.

Setup a mapping file

The main submission script, pyjasub works by building job submission commands from user arguments. Therefore there is a mapping step to map between the user aguments and the cluster arguments. This happens internally but how the mapping is performed is controlled by a mapping config file. Examples, can be seen in the ./resources/mappings directory of the repository and is also shown below:

[mappings]
array=-t {0}-{1}:{2}

# Multiple arguments are separated by commas, such as two memory arguments
mem=-l hmem={0}, -l vmem={0}
time=-l h_rt={0}
nodes=-l hostname={0}
shell=-S {0}
out=-o {0}
err=-e {0}
batch=-tc {0}
tmp_size=-l tmpfs={0}
scratch_size=-l tscratch={0}

This follows a .ini format and and the Python formatting characters {0}, {1} etc... will be substituted with arguments. The mappings should be under the section header [mappings].

As a single the mappings file is expected per system, the location of the mappings file should be set in your ~/.bashrc:

export QSUB_PYTHON_MAPPING="/path/to/mappings.conf"

As the mappings file is a potential security risk, is should only be readable by you. This is checked by pyjasub and if any other users have any of rwx on the mapping file then pyjasub will error out with a PermissionError.

The abstraction of the cluster arguments has some plusses and minuses, it allows similar commands and job files can be used in different setups. However, this is at the expense of granular control. Having said that pyjasub can handle most everyday cluster uses and can probably be extended easily. Currently, pyjasub has only really been tested using GridEngine, however, it should be able to be adapted for LSF.

Add the bin directory to your PATH

There are a number of bash scripts used by pyjasub. Whilst, they are not obligatory and the user has full control of the task script, they can be used to perform some setup and cleanup operations. If you make sure that the directory .resources/bin is added to your PATH, for example, in your ~/.bashrc file:

export "${PATH}:/path/to/cluster/resources/bin"