Skip to main content

Enhancements to the gridengine scheduler

Project description

gridengine-goodies: Enhancements to gridengine

The gridengine-goodies package provides a few command line scripts that enhance the usability of the gridengine scheduling system.

qsubx: an improved qsub

qsubx allows to submit multiple jobs at a time while keeping at most N jobs in the queue. This way you can submit tons of jobs and still be polite to your fellow users.

qsubx -n 100 job_*.sh

Note that N=100 is the default. qsubx will not return until all the jobs have been submitted.

qdelx: delete jobs via regular expressions

To delete jobs by name

qdelx sim_T*_rho*

To delete all jobs waiting in the queue

qdelx qw

qstatx: an improved qstat

It provides better layout and full job names (adapted from a code of Jeet Sukumaran). The main benefit of qstatx is to provide full (or almost full) job names and better layout.

------------------------------------------------------------------------------------------------
id      name      owner  submitted            state  queue  node                    slots
------------------------------------------------------------------------------------------------
124905  sleep_10  coslo  2018-10-09 15:54:42  r      all.q  varda.coulomb.univ-mon  1    
124906  sleep_10  coslo  2018-10-09 15:54:42  r      all.q  varda.coulomb.univ-mon  1    
124907  sleep_10  coslo  2018-10-09 15:54:42  r      all.q  varda.coulomb.univ-mon  1    
124908  sleep_10  coslo  2018-10-09 15:54:42  r      all.q  varda.coulomb.univ-mon  1    
124909  sleep_10  coslo  2018-10-09 15:54:44  qw                                    1    
124910  sleep_10  coslo  2018-10-09 15:54:44  qw                                    1    
124911  sleep_10  coslo  2018-10-09 15:54:44  qw                                    1    
124912  sleep_10  coslo  2018-10-09 15:54:44  qw                                    1    
------------------------------------------------------------------------------------------------
Running  :   4 (slots:    4)
Hold     :   0 (slots:    0)
Pending  :   4 (slots:    4)
Deletion :   0 (slots:    0)
Error    :   0 (slots:    0)
------------------------------------------------------------------------------------------------

qexec: execute a command on the cluster

Execute a command on the computing nodes directly. The job file is created on the fly and stored in a temporary directory under the user .local/share folder. The behavior is similar to qsub -b y <command> but qexec is more flexible and provide persistent access to job files and logs if needed.

This job will sleep for 10 seconds

qexec -c sleep 10

Submit a script or a program named code.x and store log files in the output_dir folder

qexec -o output_dir -c code.x --verbose -param 1.0

qbarrier: halt execution until some jobs are over

It provides a simple means to create command-line-level dependencies between jobs. For finer control on job dependencies, see 'job arrays' in the gridengine documentation.

Installation

The above goodies are plain python and bash scripts. You can just copy them and put them somewhere in your path. However, to keep up with updates it is better to clone the code repository and install the scripts accordingly

git clone https://gitlab.info-ufr.univ-montp2.fr/daniele.coslovich/gridengine-goodies.git
cd gridengine-goodies
make user

The above procedure will put the scripts in ~/.local/bin so make sure it is in your PATH

export PATH=$HOME/.local/bin:$PATH

Add the above line to your .bashrc to make it persistent across sessions.

To update to the latest version

cd gridengine-goodies
git pull
make user

Authors

Daniele Coslovich: http://www.coulomb.univ-montp2.fr/perso/daniele.coslovich/

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Files for gridengine-goodies, version 1.0.4
Filename, size & hash File type Python version Upload date
gridengine_goodies-1.0.4-py2.py3-none-any.whl (24.3 kB) View hashes Wheel py2.py3

Supported by

Elastic Elastic Search Pingdom Pingdom Monitoring Google Google BigQuery Sentry Sentry Error logging AWS AWS Cloud computing DataDog DataDog Monitoring Fastly Fastly CDN SignalFx SignalFx Supporter DigiCert DigiCert EV certificate StatusPage StatusPage Status page