A Snakemake executor plugin for submitting jobs to an AWS Parallel Cluster (pcluster) SLURM cluster.
Project description
Snakemake executor plugin: pcluster-slurm v_0.0.6_
Snakemake Executor Plugins (generally)
Snakemake plugin catalog docs.
pcluster-slurm
plugin
AWS Parallel Cluster, pcluster
slurm
AWS Parallel Cluster is a framework to deploy and manage dynamically scalable HPC clusters on AWS, running SLURM as the batch system, and pcluster
manages all of the creating, configuring, and deleting of the cluster compute nodes. Nodes may be spot or dedicated. note, the AWS Parallel Cluster
port of slurm has a few small, but critical differences from the standard slurm distribution. This plugin enables using slurm from pcluster head and compute nodes via snakemake >=v8.*
.
Pre-requisites
Snakemake >=8.*
Conda
conda create -n snakemake -c conda-forge -c bioconda snakemake==8.20.6
conda activate snakemake
Installation (pip)
from an environment with snakemake and pip installed
pip install snakemake-executor-plugin-pcluster-slurm
Example Usage daylily cluster headnode
mkdir -p /fsx/resources/environments/containers/ubuntu/cache/
export SNAKEMAKE_OUTPUT_CACHE=/fsx/resources/environments/containers/ubuntu/cache/
snakemake --use-conda --use-singularity -j 10 --singularity-prefix /fsx/resources/environments/containers/ubuntu/ip-10-0-0-240/ --singularity-args " -B /tmp:/tmp -B /fsx:/fsx -B /home/$USER:/home/$USER -B $PWD/:$PWD" --conda-prefix /fsx/resources/environments/containers/ubuntu/ip-10-0-0-240/ --executor pcluster-slurm --default-resources slurm_partition='i64,i128,i192' --cache --verbose
More Documentation Pending For:
How slurm uses --comment
to tag resources created by the autoscaling cluster to tracke and mnage budgets.
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
Hashes for snakemake_executor_plugin_pcluster_slurm-0.0.6.tar.gz
Algorithm | Hash digest | |
---|---|---|
SHA256 | 346b822d05ea05091f837cc66af608437b0e370835a57e4ee26753e0e0a50c8c |
|
MD5 | 00e48c55257af2d015b707b2b09fdb03 |
|
BLAKE2b-256 | b5dd21e94a48a70faaa30470643a3fc4ef755d1fdc133ba746977062d0621563 |
Hashes for snakemake_executor_plugin_pcluster_slurm-0.0.6-py3-none-any.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 809f1f64d079b6b06a5e38d5166b4bf2ab30783405faffaf80d39167ceb687da |
|
MD5 | 5aa1a07acecb710a07cba979e8414ef1 |
|
BLAKE2b-256 | 1506003e031137b2adaf19fc3a8b8339df56a8e4454031613054b97360d34ebc |