SumsJob 0.6.0

A simple Linux command-line utility which submits a job to one of the multiple servers

ΣΣ_Job

ΣΣ_Job or Sums_Job (Simple Utility for Multiple-Servers Job Submission) is a simple Linux command-line utility which submits a job to one of the multiple servers each with limited GPUs. ΣΣ_Job provides similar key functions for multiple servers as Slurm Workload Manager for supercomputers and computer clusters. It provides the following key functions:

• show the status of GPUs on all servers,
• submit a job to servers in noninteractive mode, i.e., the job will be running in the background of the server,
• submit a job to servers in interactive mode, just as the job is running in your local machine,
• display all running jobs,
• cancel running jobs.

Motivation

Assume you have a few GPU servers: server1, server2, ... When you need to run a code from your computer, you will

1. Select one server and log in

   $ ssh LAN (You may need to first log in a local area network)
   $ ssh server1
   

2. Check GPU status. If no free GPU, go to step 1

   $ nvidia-smi or $ gpustat

3. Copy the code from your computer to the server

   $ scp -r codes server1:~/project/codes
   

4. Run the code in the server

   $ cd ~/project/codes
   $ CUDA_VISIBLE_DEVICES=0 python main.py
   

5. Transfer back the results

   $ scp server1:~/project/codes/results.dat .
   

These steps are boring. ΣΣ_Job makes all these steps automatic.

Features

• Simple to use
• Two modes: noninteractive mode, and interactive mode
• Noninteractive mode: the job will be running in the background of the server
  • You can turn off your local machine
• Interactive mode: just as the job is running in your local machine
  • Display the output of the program in the terminal of your local machine in real time
  • Kill the job by Ctrl-C

Usage

$ gpuresource

Show the status of GPUs on all servers. For example,

$ submit jobfile [jobname]

Submit a job to GPU servers. Automatically do the following steps:

1. Find a GPU with low utilization and sufficient memory (the criterion is in the configuration file). You can also specify the server and GPU by -s SERVER and --gpuid GPUID.
2. Copy the code to the server.
3. Run the job on it in noninteractive mode (default) or interactive mode (with -i).
4. Save the output in a log file.
5. For interactive mode, when the code finishes, transfer back the result files and the log file.

• jobfile : File to be run
• jobname : Job name, and also the folder name of the job. If not provided, a random number will be used.

Options:

• -h, --help : Show this help message and exit
• -i, --interact : Submit as an interactive job
• -s SERVER, --server SERVER : Server host name
• --gpuid GPUID : GPU ID to be used; -1 to use CPU only

$ sacct

Display all running jobs ordered by the start time. For example,

$ scancel jobname

Cancel a running job.

• jobname : Job name.

Installation

Install Sums_Job with pip:

$ pip install sumsjob

You also need to do the following:

• Make sure you can ssh to each server, ideally without typing the password by SSH keys.
• Install gpustat in each server.
• Have a configuration file at ~/.sumsjob/config.py. Use config.py as a template, and modify the values to your configurations.
• Make sure ~/.local/bin is in your $PATH.

Then run gpuresource to check if everything works.

License

GNU GPLv3