Libpipe

Common pipeline framework and library

Installation

libpipe can be installed via pip:

$ pip install libpipe

and requires Python 3.10 or higher.

Worker pools on a Slurm cluster

A WorkerPool runs its tasks locally, on other hosts over SSH, or through Slurm with the host slurm: each task is then started with srun, as a job of its own when the pool runs on a login node, as a job step inside an allocation (salloc, sbatch).

pool = worker.WorkerPool(['slurm'], name='Imaging', max_tasks_per_worker=8,
                         slurm_options='-p regular --mem=200G --time=24:00:00')

max_tasks_per_worker is the number of jobs in flight. The srun options (partition, account, resources) come from slurm_options, or from the LIBPIPE_SLURM_OPTIONS environment variable; max_time becomes the job time limit when the options set none. Host constraints (run_on_host) do not apply: Slurm places the tasks, so the data must be on a file system shared with the compute nodes.

With slurm_nodes, a pool on a login node requests one allocation of up to that many nodes when its first task starts, runs each task as a job step in it (srun --jobid --exact), and releases it at the end, or after 10 minutes without step. Slurm packs the steps on the nodes from their CPUs and memory, given by the category of the task; a step starts when the allocation has its CPUs free, so that no srun waits polling the controller:

pool = worker.WorkerPool(['slurm'], name='Cubes', slurm_nodes=4,
                         resources={'heavy': {'cpus': 12, 'mem': '40GB'}})
pool.add('pstool gen_vis_cube ...', category='small', label='gen_vis_cube')

The categories (DEFAULT_RESOURCES) are small (1 CPU, 2 GB), medium (4, 8 GB), high_mem (2, 16 GB), cpu (12, 8 GB) and heavy (12, 40 GB, the default), the others capped at heavy; a step also gets the matching OMP_NUM_THREADS. The options apply to the jobs and the allocation, not to the steps. Inside an allocation started by hand, the tasks are job steps with these resources as well.

Slurm does not necessarily limit the memory of a job step: the memory of a category must stay above the peak memory of its tasks. The local and Slurm pools measure each task (wall time, CPU time, peak RSS, in its log file), and WorkerPoolReport.usage_lines() summarises them per task type (label).