Slurm Job Pruning

A maintenance tool for canceling stale jobs on Slurm clusters.

A command-line utility for Slurm administrators that identifies and cancels jobs which have been stuck in the PENDING state for longer than a configurable threshold. Affected users are notified by email, with all of their canceled jobs batched into a single message, and a dry-run mode is provided for previewing changes before applying them.

Install and Setup

Install the utility from the CRCD Python repository using pipx:

pipx install crc-prune-stale

Confirm the prune-stale utility is available in your runtime environment:

prune-stale --help

Optionally, create the application log directory to enable persistent file logging. If the directory is not present, the application will fall back to console-only logging:

sudo mkdir -p /var/log/prune_stale

Usage

prune-stale [OPTIONS]

Pruning options

Notification options

Pass --help to see all options with their current defaults.

Examples

Cancel all jobs that have been pending for more than 10 days (no email notification):

prune-stale --threshold 10

Cancel jobs and issue an email notification to the job submitter:

prune-stale --threshold 10 --smtp-host mailrelay.pitt.edu

Preview what would be canceled without making any changes:

prune-stale --dry-run