jobs-supervisor 0.3.5

Tools to supervise slurm jobs

Introduction

A tool to supervise jobs status via Feishu bot messages.

Currently supportted job states are : 'fail', 'complete', 'unavailable', 'cancel', 'running', 'pending'. If your job does not belong to those states, the reportted status will be 'unknown'.

Once you init a supervisor, it will be stopped if all its jobs are one of ['fail', 'complete', 'unavailable', 'cancel', 'unknown'], and will report all its jobs' status to you via your Feishu bot.

Setup Guide

1. Get webhook from your Feishu Bot.

2. Export and set webhook 'JOBS_SUPERVISOR_WEBHOOK' in your ~/.basrc. For example: export JOBS_SUPERVISOR_WEBHOOK=https://open.feishu.cn/your-webhook

3. Export and set the directory 'JOBS_SUPERVISOR_LOGDIR' to save your sbatch output in your ~/.basrc. For example: export JOBS_SUPERVISOR_LOGDIR=path_to_log_files

4. Run source ~/.basrc

5. (Optional) Create an enviornment for the tools by conda create -n env_tools python=3.8

6. Activate your enviornment by conda activate env_tools

7. Install this python package in the enviornment by pip3 install -i https://pypi.org/project/ jobs-supervisor==0.3.5

Usage

Assume 'jobid_start' and 'jobid_end' are int.

1. Activate your enviornment by conda activate env_tools

2. In terminal: supervise_jobs jobid_start jobid_end

3. You can run supervise_jobs --help for more configs. By default,

• it will skip to report 'unavailable' jobs (jobs that you cannot get status). Set the option --show_unavailable to report unavailable jobs.

• it will only report once all jobs are stopped. If you want to report every time a job stops, set the option --show_intermedia