Slurm Job Pruning

A maintenance tool for canceling stale jobs on Slurm clusters.

A command-line utility for Slurm administrators that identifies and cancels jobs which have been stuck in the PENDING state for longer than a configurable threshold. Affected users are notified by email, with all of their canceled jobs batched into a single message, and a dry-run mode is provided for previewing changes before applying them.

Install and Setup

Install the utility from the CRCD Python repository using pipx:

pipx install crc-prune-stale

Confirm the prune-stale utility is available in your runtime environment:

prune-stale --help

Optionally, create the application log directory to enable persistent file logging. If the directory is not present, or the log file cannot be opened, the application emits a warning and falls back to console-only logging:

mkdir -p /var/log/prune_stale

Logs are written to /var/log/prune_stale/prune_stale.log.

Usage

prune-stale [OPTIONS]

Targeting options

Control which cluster and partitions are queried for jobs.

The default cluster is resolved at startup from the local node's Slurm configuration (scontrol show config). If no cluster name can be determined, the application exits with an error.

Pruning options

Control which jobs are selected for cancellation.

Job age is measured against the job's submit time as reported by squeue. Jobs with malformed or unparsable squeue output are skipped and logged as warnings.

Notification options

Control outbound email notifications.

Notifications are only sent when --smtp-host is provided and --dry-run is not set. SMTP failures are logged and do not abort the run.

Pass --help to see all options with their current defaults.

Examples

Cancel all jobs that have been pending for more than 10 days (no email notification):

prune-stale --threshold 10

Cancel jobs and issue an email notification to the job submitter:

prune-stale --threshold 10 --smtp-host mailrelay.pitt.edu

Restrict the run to specific partitions on a specific cluster:

prune-stale --cluster htc --partition standard high-mem

Preview what would be canceled without making any changes:

prune-stale --dry-run