Skip to main content

Submit array jobs to a SGE cluster without all the suffering

Project description

psub

Submit array jobs without all the suffering

Stands for petko-submit, OG ernstlab member who came up with the idea.

New API (alpha):

There's a new (unstable) API that allows submitting jobs directly in Python.

Installation:

Must be installed under ~/utils like this:

mkdir ~/utils; cd ~/utils
git clone git@github.com:udincer/psub.git

Add alias:

# Optional: add psub as an alias
# Add the following to .bashrc
alias psub='/u/home/d/<your_username>/utils/psub/psub_main.sh '

How to use:

Make script containing each command you want to run in parallel in exactly one line:

# the_jobs.sh
python run_analysis.py theta1
python run_analysis.py theta2
python run_analysis.py theta3
python run_analysis.py theta4
python run_analysis.py theta5

Run psub like this, will batch 3 commands to one SGE job (for short jobs):

psub the_jobs.py 3

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Files for psub, version 0.0.3a0
Filename, size File type Python version Upload date Hashes
Filename, size psub-0.0.3a0-py3-none-any.whl (3.4 kB) File type Wheel Python version py3 Upload date Hashes View hashes
Filename, size psub-0.0.3a0.tar.gz (3.3 kB) File type Source Python version None Upload date Hashes View hashes

Supported by

Elastic Elastic Search Pingdom Pingdom Monitoring Google Google BigQuery Sentry Sentry Error logging AWS AWS Cloud computing DataDog DataDog Monitoring Fastly Fastly CDN SignalFx SignalFx Supporter DigiCert DigiCert EV certificate StatusPage StatusPage Status page