Spork: Helping You Run Slurm Jobs

Spork is a quality of life package that enables rapid deployment of code on a slurm cluster with singularity installed and password enabled login. Running experiments on your cluster is as simple as a single command in the terminal using spork. See below for an example and install instructions.

Installation

Spork can be installed using the pip package.

pip install spork-cli

Usage

You may configure spork to remember the ssh credentials to your cluster using the following example.

spork set --ssh-username username --ssh-password password --ssh-host compute.example.com

Running your first command on your cluster is then as simple as one line in the terminal.

spork remote echo "my first command"

Certain workloads require uploading certain data files into the singularity image on the remote machine before running experiments. This can be done with the following command.

spork upload --recursive --exclude "*.pkl" ./local_dir remote_dir/in/image

Additionally, you can download files from the remote machine with a single command. The following command will download the results folder inside the remote singularity image to a location in the current local working directory. The remote path is always taken with respect to the singularity image path.

spork download --recursive --exclude "*.pkl" results ./

Experimentation

A typical experiment creation pipeline involved working on code locally, testing the code locally, then running in on a server. This can be done easily using spork.

python do_my_experiment.py

Once your code is ready for deployment, tell spork how to install your code from github.

spork set --post-env-commands "git clone https://github.com/username/repo /code/repo" \
          --post-env-commands "pip install -e /code/repo"

Then, point spork to where your code working directory is stored locally.

spork set --sync-with ./ --sync-target /code/repo --exclude-from-sync "*.sif"

Then test your code in a local copy of the singularity to make sure it works as expected.

spork local python /code/repo/do_my_experiment.py

Then run it on the cluster.

spork remote python /code/repo/do_my_experiment.py

In this example, spork will automatically copy code from your repository on the local disk to code folder in your remote singularity image. This is especially helpful when changes aren't committed.