Skip to main content

Submit HPC jobs via ssh from Python

Project description

A Python-based and jupyter notebook-friendly job manegement system for remote job management systems.

pyjobs allows to submit/list/detele/track jobs on Univa Grid Engine (UGE) or Sun Grid Engine (SGE). It enables to remotely execute qsub, qstat, qdel (via ssh) and show the log/error files on local jupyter notebook (or jupyter lab). It can also make it possible to execute jobs on other machines via ssh and nohup (under developments).

Requirements

  • ssh environment to remote machine. pyjobs is designed to be run on local machine. ssh connection to remote machines (such as ABCI) must be prerequired.

Usage

pyjobs is designed to be run on local Jupyter notebook environment.

On jupyter notebooks, run

from pyjobs import *
import subprocess,json
jobs=pyjobs()
jobs,type(jobs),json.dumps(jobs)

to initialize jobs instance of pyjobs.

jobs instance is used to manage your jobs on a specific remote (or local) machine. jobs.qsub() jobs.show() jobs.kill() jobs.rm() jobs.bye()

  • Run job

    jobs.qsub(s2o,"s2o.sh",jc="+gpu,g1,72h",n=0,docker="nvcr-pytorch-1903")
  • Show job list

    jobs.show(depth=1,no_update=False)
  • Assign a job to the jobs instance later

    jobs[3059261]=pyjob(jobid="3059261",jobname="s2o.sh",jobfile=`cat s2o.sh`)
    #jobs.dump()

– Delete a latest job and its log

jobs.kill()+jobs.rm()

jobs.bye(-1)
  • shell support with ssh

def shellrun(commandline,server,cd,ssh_bash_profile=True,nohup=False)
def run_via_ssh(commandline,server,cd,ssh_bash_profile=True,nohup=False)
  • when nohup=True, commandline shuould be a shell file.

Project details


Release history Release notifications | RSS feed

Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

sshjob-0.0.dev4.tar.gz (12.8 kB view hashes)

Uploaded Source

Supported by

AWS AWS Cloud computing and Security Sponsor Datadog Datadog Monitoring Fastly Fastly CDN Google Google Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Sentry Sentry Error logging StatusPage StatusPage Status page