Libpipe
Common pipeline framework and library
Installation
libpipe can be installed via pip:
$ pip install libpipe
and requires Python 3.10 or higher.
Worker pools on a Slurm cluster
A WorkerPool runs its tasks locally, on other hosts over SSH, or through Slurm with the
host slurm: each task is then started with srun, as a job of its own when the pool runs
on a login node, as a job step inside an allocation (salloc, sbatch).
pool = worker.WorkerPool(['slurm'], name='Imaging', max_tasks_per_worker=8,
slurm_options='-p regular --mem=200G --time=24:00:00')
max_tasks_per_worker is the number of jobs in flight. The srun options (partition,
account, resources) come from slurm_options, or from the LIBPIPE_SLURM_OPTIONS
environment variable; max_time becomes the job time limit when the options set none.
Host constraints (run_on_host) do not apply: Slurm places the tasks, so the data must be
on a file system shared with the compute nodes. Give the memory (--mem) in the options:
inside an allocation, a step without it can claim all the memory of its node.
1 maintainer
Metadata
1 maintainer
Release files for libpipe 0.8
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | |
|---|---|---|---|
| libpipe-0.8.tar.gz | 29.0 kB | Details |
Built distribution (wheel)
| File | Interpreter | ABI | Platform | Reset |
|---|---|---|---|---|
| libpipe-0.8-py3-none-any.whl | Python 3 | none | any | Details |
Total release size: 60.4 kB
Release files / libpipe-0.8.tar.gz
| Download URL | libpipe-0.8.tar.gz |
|---|---|
| Size | 29.0 kB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
4d2ef82320974db3ea402d93dfecb0124db7723f537c3f5a1b6c42c50938d9dc
|
|
BLAKE2b-256 checksum How to use checksums |
1a0eccd27211b18994cefddc087ae1263f322f55864e3acc7dd65d169a382f26
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
poetry/2.4.1 CPython/3.14.4 Linux/7.0.0-38-generic
|
Release files / libpipe-0.8-py3-none-any.whl
| Download URL | libpipe-0.8-py3-none-any.whl |
|---|---|
| Size | 31.4 kB |
| Tags | Python 3 |
|
SHA-256 checksum How to use checksums |
9f29b4ad03dc00a4916fce547368ce01f0fb34b8f2750076334e50ee98c3bf36
|
|
BLAKE2b-256 checksum How to use checksums |
bdeab5e23c45eb0d1961ec72e5e50222e4a69bb441e99882cec96c5eb395c770
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
poetry/2.4.1 CPython/3.14.4 Linux/7.0.0-38-generic
|
