Submit HPC jobs via ssh from Python
Project description
A Python-based and jupyter notebook-friendly job manegement system for remote job management systems.
pyjobs allows to submit/list/detele/track jobs on Univa Grid Engine (UGE) or Sun Grid Engine (SGE). It enables to remotely execute qsub, qstat, qdel (via ssh) and show the log/error files on local jupyter notebook (or jupyter lab). It can also make it possible to execute jobs on other machines via ssh and nohup (under developments).
Requirements
ssh environment to remote machine. pyjobs is designed to be run on local machine. ssh connection to remote machines (such as ABCI) must be prerequired.
Usage
pyjobs is designed to be run on local Jupyter notebook environment.
On jupyter notebooks, run
from pyjobs import *
import subprocess,json
jobs=pyjobs()
jobs,type(jobs),json.dumps(jobs)
to initialize jobs instance of pyjobs.
jobs instance is used to manage your jobs on a specific remote (or local) machine. jobs.qsub() jobs.show() jobs.kill() jobs.rm() jobs.bye()
Run job
jobs.qsub(s2o,"s2o.sh",jc="+gpu,g1,72h",n=0,docker="nvcr-pytorch-1903")
Show job list
jobs.show(depth=1,no_update=False)
Assign a job to the jobs instance later
jobs[3059261]=pyjob(jobid="3059261",jobname="s2o.sh",jobfile=`cat s2o.sh`) #jobs.dump()
– Delete a latest job and its log
jobs.kill()+jobs.rm()
jobs.bye(-1)
shell support with ssh
def shellrun(commandline,server,cd,ssh_bash_profile=True,nohup=False)
def run_via_ssh(commandline,server,cd,ssh_bash_profile=True,nohup=False)
when nohup=True, commandline shuould be a shell file.
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.