Skip to main content

Utilities to support interacting with multiple HPC clusters

Project description

MyCluster

Master Build Status Scrutinizer Code Quality

Develop Build Status Scrutinizer Code Quality

Utilities to support interacting with multiple HPC clusters

Provides the ability to interact with the most popular HPC job scheduling systems using a single interface and enables the creation of job submission scripts. The system also tracks statistics of each job and records the hardware details of the compute nodes used.

Data is stored in a local database in ~/.mycluster

Tested with SGE and slurm (LSF and PBS/TORQUE support under development)

Installation - pip install MyCluster or git clone https://github.com/zenotech/MyCluster.git

Dependencies: ZODB, Fabric

Example usage

Register details

mycluster --firstname Fred --lastname Bloggs --email fred.bloggs@email.com

List all queues

mycluster -q

Create job script

mycluster --create JOB_SCRIPT --jobqueue QUEUE --script SCRIPT --ntasks=TASKS --jobname=JOB_NAME 
          --project ACCOUNT_NAME --maxtime 12:00:00

Submit job

mycluster --submit JOB_SCRIPT

Delete job

mycluster --delete JOB_ID

Print job table

mycluster -p

Print help

mycluster --help

The SCRIPT to be executed by the JOB_SCRIPT can make use of the following predefined environment variables

export NUM_TASKS=
export TASKS_PER_NODE=
export THREADS_PER_TASK=
export NUM_NODES=

# OpenMP configuration
export OMP_NUM_THREADS=$THREADS_PER_TASK

# Default mpiexec commnads for each flavour of mpi
export OMPI_CMD="mpiexec -n $NUM_TASKS -npernode $TASKS_PER_NODE -bysocket -bind-to-socket"
export MVAPICH_CMD="mpiexec -n $NUM_TASKS -ppn $TASKS_PER_NODE -bind-to-socket"
export IMPI_CMD="mpiexec -n $NUM_TASKS -ppn $TASKS_PER_NODE"

In order to capture the relevant information it is recommended that the SCRIPT also exports the following environment variables

# Application name
export MYCLUSTER_APP_NAME=
# Data size that typifies application performance for this job (e.g number of points or number of cells)
export MYCLUSTER_APP_DATA=

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Files for MyCluster, version 1.0.1
Filename, size File type Python version Upload date Hashes
Filename, size MyCluster-1.0.1.tar.gz (19.7 kB) File type Source Python version None Upload date Hashes View

Supported by

Pingdom Pingdom Monitoring Google Google Object Storage and Download Analytics Sentry Sentry Error logging AWS AWS Cloud computing DataDog DataDog Monitoring Fastly Fastly CDN DigiCert DigiCert EV certificate StatusPage StatusPage Status page