slurm-longrun 0.1.2

No project description provided

Slurm Longrun

Slurm Longrun is a Python package that wraps Slurm’s sbatch command to automatically resubmit jobs that time out, allowing you to run workloads that exceed a single‐job walltime without manual intervention. It supports optional terminal detachment (so your monitor survives after you log out), configurable retry limits, and built-in logging via Loguru.

---

Installation

Prerequisites

• Python 3.10+
• Slurm workload manager (sbatch, sacct, scontrol in your PATH)

Install from PyPI:

pip install slurm-longrun

---

Quickstart

Instead of calling sbatch directly, use the sbatch_longrun wrapper:

sbatch_longrun [OPTIONS] [SBATCH_ARGS…]

Everything after -- is passed directly to sbatch.

Example: your job runs longer than 30 minutes, so you give it a 30 min walltime and let Longrun resubmit on timeout:

sbatch_longrun --max-restarts 5 --time=00:30:00 --job-name=my_job -- my_script.sbatch

This will:

1. Submit my_script.sbatch with a 30 min limit.
2. When it hits the 30 min walltime (TIMEOUT), automatically resubmit (appending to the same log file).
3. Repeat up to 5 times or until the job completes successfully.

---

Command-Line Interface

Usage

sbatch_longrun [OPTIONS] [SBATCH_ARGS…]

Options

• --use-verbosity [DEFAULT|VERBOSE|SILENT]
   Logging level (DEFAULT = INFO, VERBOSE = DEBUG, SILENT = WARNING).
• --detached / --no-detached
   Run the monitor loop in background (detached from your terminal).
• --max-restarts INTEGER
   Maximum number of resubmissions on TIMEOUT. Default: 99.
• -h, --help
   Show help and exit.

All other flags after -- are forwarded to sbatch.

Examples

1. Basic, retry up to 3 times, verbose logging:

   slurm-longrun --use-verbosity VERBOSE --max-restarts 3 -- \
     --time=02:00:00 --job-name=deep_train train.sbatch
   

2. Detach the monitor so it survives logout:

   slurm-longrun --detached -- \
     --time=01:00:00 --job-name=data_proc data_pipeline.sbatch
   # → prints “Monitor running in background PID: ”
   

---

How It Works

1. Submit
   Calls sbatch with your arguments; parses the returned job ID.
2. Monitor
   • Polls sacct + scontrol until the job reaches a terminal state.
   • If TIMEOUT and you haven’t exceeded --max-restarts, it immediately resubmits with --open-mode=append to preserve logs.
3. Detach (optional)
   If --detached is passed, the process forks twice, detaches from the terminal (setsid), redirects stdio to /dev/null, and continues monitoring in background.

---

Environment Variables

SLURM_LONGRUN_INITIAL_JOB_ID

• Set internally to the first submission’s job ID.
• You can read it in your job script (e.g., to name checkpoints).

---

Dependencies

• click
• loguru

These are installed automatically via pip.

---

Summary of CLI Options

Option	Default	Description
--use-verbosity	DEFAULT	Logging verbosity: DEFAULT (INFO), VERBOSE, SILENT (WARNING)
--detached / --no-detached	--no-detached	Detach monitoring loop into background process
--max-restarts	99	Max auto-resubmissions on TIMEOUT
-- [SBATCH_ARGS…]	–	All subsequent flags passed directly to sbatch

---

Enjoy uninterrupted long‐running Slurm jobs! 🐢→🚀