SumsJob 0.2.0

A simple Linux command-line utility which submits a job to one of the multiple servers

ΣΣ_Job

ΣΣ_Job or Sums_Job (Simple Utility for Multiple-Servers Job Submission) is a simple Linux command-line utility which submits a job to one of the multiple servers each with limited resources. It will first look for servers with available resources, such as GPUs, and then run the job in that server interactively just as the job is running in your local machine.

Documentation: Slides

Motivation

Assume you have a few GPU servers: server1, server2, ... When you need to run a code from your computer, you will

1. Select one server and log in

   $ ssh LAN (You may need to first log in a local area network)
   $ ssh server1
   

2. Check GPU status. If no free GPU, go to step 1

   $ nvidia-smi or $ gpustat

3. Copy the code from your computer to the server

   $ scp -r codes server1:~/project/codes
   

4. Run the code in the server

   $ cd ~/project/codes
   $ CUDA_VISIBLE_DEVICES=0 python main.py
   

5. Transfer back the results

   $ scp server1:~/project/codes/results.dat .
   

These steps are boring. ΣΣ_Job makes all these steps automatic.

Features

• Simple to use: commands gpuresource and submit are all your need
• Two modes: noninteractive mode, and interactive mode
• Noninteractive mode: the job will be running in the background of the server
  • You can turn off your local machine
• Interactive mode: just as the job is running in your local machine
  • Display the output of the program in the terminal of your local machine in real time
  • Kill the job by Ctrl-C

Usage

$ gpuresource

Show the status of GPUs on all servers. For example,

$ submit jobfile [jobname]

Submit a job to (GPU) servers. Automatically do the following steps:

1. Find a server with free GPU. You can specify the server and GPU ID by -s SERVER and --gpuid GPUID.
2. Copy the code to the server.
3. Run the job on it in noninteractive mode (default) or interactive mode (with -i).
4. Save the output in a log file.
5. For interactive mode, when the code finishes, transfer back the result files and the log file.

• jobfile : File to be run
• jobname : Job name, and also the folder name of the job. If not provided, a random number will be used.

Options:

• -h, --help : Show this help message and exit
• -i, --interact : Submit as an interactive job
• -s SERVER, --server SERVER : Server host name
• --gpuid GPUID : GPU ID to be used; -1 to use CPU only

Installation

Install Sums_Job with pip:

$ pip install sumsjob

You also need to do the following:

• Make sure you can ssh to each server, ideally without typing the password by SSH keys.
• Install gpustat in each server.
• Have a configuration file at ~/.sumsjob/config.py. Use config.py as a template, and modify the values to your configurations.
• Make sure ~/.local/bin is in your $PATH.

Then run gpuresource to check if everything works.

License

GNU GPLv3