turboblast

Purpose

turboblast is a Python library for submitting high-throughput job arrays to a Slurm cluster using submitit. It is designed for workflows where you have a large list of command-line tasks (e.g. processing satellite files) that need to be distributed across many compute nodes in parallel.

The core idea is simple: you provide a text file where each line is a set of arguments, and turboblast dispatches each line as an independent Slurm task running a bash script of your choice. Large input lists are automatically split into chunks of 1000 to stay within Slurm array limits.

Dependencies

Installation

pip install turboblast

Or with conda:

conda install -c conda-forge turboblast

Usage

Prepare your inputs

Create a plain text file where each line contains the arguments for one task:

# inputs.txt
--input /data/file_001.nc --output /results/
--input /data/file_002.nc --output /results/
--input /data/file_003.nc --output /results/

Write your bash script

turboblast will call bash your_script.sh <args> for each line. Example:

#!/bin/bash
# process.sh
python my_processor.py "$@"

Submit the job array

turboblaster \
  --listing-input inputs.txt \
  --bash-slurm-exec process.sh \
  --slurm-partition gpu \
  --timeout-min 60 \
  --mem-gb 8 \
  --cpus-per-task 4 \
  --slurm-array-parallelism 50 \
  --output-dir submitit_logs

Full CLI reference

usage: turboblaster [-h] [--num-tasks NUM_TASKS] [--timeout-min TIMEOUT_MIN]
                    [--mem-gb MEM_GB] [--cpus-per-task CPUS_PER_TASK]
                    [--slurm-partition SLURM_PARTITION]
                    --listing-input LISTING_INPUT
                    --bash-slurm-exec BASH_SLURM_EXEC
                    [--output-dir OUTPUT_DIR]
                    [--slurm-array-parallelism SLURM_ARRAY_PARALLELISM]

options:
  --listing-input            Path to a file containing input lines (one task per line) [required]
  --bash-slurm-exec          Path to the bash script to execute for each task [required]
  --num-tasks                Number of tasks (unused if reading from file) [default: 20]
  --timeout-min              Timeout in minutes for each task [default: 20]
  --mem-gb                   Memory in GB for each task [default: 2]
  --cpus-per-task            Number of CPUs per task [default: 1]
  --slurm-partition          Slurm partition to use [default: cpu]
  --output-dir               Directory to store submitit logs [default: submitit_logs_array]
  --slurm-array-parallelism  Max number of tasks running concurrently [default: 20]

Submitit logs (.out / .err files) are written to a timestamped subdirectory under --output-dir:

submitit_logs/
└── 20260309T143000/
    ├── 12345_0_0.out
    ├── 12345_1_0.out
    └── ...

Monitor a specific task with:

tail -f submitit_logs/20260309T143000/12345_0_0.out

Throughput & tuning

The progress bar is not a measure of task speed. It shows an aggregate rate, s/task = elapsed / completed, not the per-task wall time. With P = --slurm-array-parallelism tasks running in parallel, a task that actually takes T seconds shows as T / P s/task.

Example: 1000 tasks, each ~83 s (apptainer launch + work), P = 20:

164/1000 [12:00<57:41, 4.14s/task, running=20, pending=816]

4.14 s/task is 12:00 / 164; the real per-task time is 4.14 × 20 ≈ 83 s. Nothing is stuck — 20 tasks run, finish, and 20 more are allocated.

Tuning:

• --slurm-array-parallelism (default 20) caps concurrency. Throughput = parallelism / per-task duration. For short tasks, 20 is often far too low (a 1000-task batch of ~3 s tasks takes ~2.5 h). Raise it (e.g. 50–100) to saturate the cluster, subject to node capacity and MaxJobCount.
• The first chunk is submitted, then the loop blocks until it is terminal before the next chunk (one array in flight at a time) — this keeps the cluster's job-count limits safe.
• main() blocks until all tasks finish; run long jobs under tmux/nohup.
• Chunks of 1000 stay under the cluster MaxArraySize (1001).

Project structure

turboblast/
├── src/
│   └── turboblast/
│       ├── __init__.py       # Package entry point, exposes __version__
│       ├── blaster.py        # Core logic: argument parsing, job submission, task execution
│       └── logo.py           # ASCII art logo used in the CLI help message
├── tests/
│   ├── test_package.py       # Package metadata tests (version check)
│   └── test_blaster.py       # Unit tests for blaster.py
├── pyproject.toml            # Build config, dependencies, tool settings
└── README.md