Nexus Snakemake utilities

Shared workflow infrastructure for Python 3.11+ on Linux/macOS, targeting Snakemake 8.14. Newer Snakemake versions are allowed but remain unverified. Install the PyPI release into the Python/Conda environment that runs Snakemake:

python -m pip install "nexus-snakemake-utils==0.2.0"

For an existing conda environment, installation also works without activation:

snakemake_env_dir="/absolute/path/to/snakemake_v8.14.0"
conda run --prefix "$snakemake_env_dir" python -m pip install \
  "nexus-snakemake-utils==0.2.0" "snakemake==8.14.0"
conda run --prefix "$snakemake_env_dir" python -m pip check

Include pip when creating the conda environment; install the conda packages first and the utils afterwards. The explicit Snakemake constraint in this example retains the tested version; the utils package itself allows Snakemake >=8.14.

To install directly from the GitHub release tag instead:

python -m pip install 'nexus-snakemake-utils @ git+https://github.com/ETH-NEXUS/nexus_snakemake_utils.git@v0.2.0'

The package installs controller dependencies. Analysis tools and R dependencies must be supplied separately by your workflow.

Portability

Nexus/Leomed access and Slurm are not required. The supported environment is Linux/macOS with Python 3.11+ and Snakemake 8.14 as the tested target. Packaging requires snakemake>=8.14 without an upper bound; this permits Snakemake 9 but does not establish compatibility. Validation of newer versions is deferred. Core utilities and benchmark reports work with the local executor. Slurm accounting is queried only for Slurm runs; missing accounting tools or records produce warnings without failing the workflow. Container usage registration skips runtimes without the proxy interface.

The optional container proxy uses Unix file locking and requires a real Singularity/Apptainer runtime for container execution. Its singularity launcher is installed with this package and must find the real runtime later on PATH or through SNAKEMAKE_REAL_SINGULARITY. Set an image directory with --image-dir or LEOMED_SNAKEMAKE_IMAGE_DIR when using its image-cache commands. The variable name does not require Leomed storage; mirror mappings are optional. Keep cluster paths, accounts and binds in site profiles, and publish a local profile too.

Reporting libraries are installed as package dependencies even if reporting is unused. Analysis tools and report-rendering software remain workflow dependencies.

Workflow interface

import nexus_snakemake_utils as utils
# Top level of the Snakefile: runs during parsing, including dry runs.
# Preserve an explicit container workflow label, otherwise use the repository
# name; then log time, user, host, Git revision/checkout state and Snakemake version.
utils.initialize(repository_dir)

onstart:
    utils.register_container_usage(workflow)
onsuccess:
    utils.write_resource_report(workflow)
onerror:
    utils.write_resource_report(workflow, partial=True)

initialize preserves an explicit SNAKEMAKE_CONTAINER_WORKFLOW label and otherwise sets it to the repository name, then calls log_run_info. It does not register container usage, execute jobs, or write resource reports. Keep it at the top level of the Snakefile so logging also runs for dry runs. Parsing-time messages are buffered until Snakemake opens its execution log, so provenance appears once in both the terminal and .snakemake/log/. Dry runs retain terminal output without creating an execution log.

Select optional inputs or targets with optional(val, on, dpath=None):

rule all:
    input:
        utils.optional(rules.report.output.html, config, dpath="optional/report"),

This returns val unchanged when the flag is truthy, otherwise []. Omit dpath to pass a flag directly, for example utils.optional(targets, enabled). Config paths use dpath syntax; missing or ambiguous paths raise errors. Define flags explicitly in the base configuration (for example optional: {report: false}) and enable them through overlays. Values are evaluated before calling the helper; use lazy input functions when resolving a value itself requires optional files.

Derive a directory from a resolved output without a lambda:

params:
    outdir=utils.dirname("events", index=0),  # output.events[0]

dirname("quant") selects a single named output; dirname() uses output[0]. It preserves os.path.dirname semantics, including "" for a bare filename.

Plain lookup(...) runs while loading rules unless its query/path needs wildcards. Use lazylookup in input or params so unused rules do not require their settings:

params:
    extra=utils.lazylookup(dpath="rule_specific/tool/extra", within=config),

It accepts Snakemake's lookup arguments, including defaults and wildcard paths. Missing required keys still fail when the job is evaluated, including during dry runs. transform=shlex.split can convert an option string after lookup. Arguments are still evaluated when calling the factory: pass within=config, not an eager lookup of an optional subsection. Use ordinary lookup for load-time setup, output paths and Python conditions; a callable itself is always truthy.

Pass a configuration branch to a script as JSON and extend it with other settings:

params:
    report_config=utils.json_dumps(
        within=config,
        base="rule_specific/de_analysis",
        tx2gene="resources/tx2gene",
        gene_info="resources/gene_info",
    ),

json_dumps returns a callable. It copies the base mapping, applies output-path → input-path mappings in keyword order, then serializes the result. Omit base to start with {}. Additional mappings override existing values; nested output paths use **{"references/tx2gene": "resources/tx2gene"}. Source paths support Snakemake wildcards. All lookups happen at job evaluation; missing paths and a non-mapping base raise errors. The original configuration is never modified.

Register system limits once in workflow/rules/misc.smk:

limits = utils.ResourceLimits(
    memory_mb=lookup(dpath="resources/max_mem_mb", within=config),
    runtime_minutes=lookup(dpath="resources/max_minutes", within=config),
)

Both limits must be positive integers. Each instance holds its own immutable limits. Use its callable factories in each rule's resources block:

resources:
    mem_mb=limits.memory_by_input(
        base_mem_mb=2000,
        mem_per_input_mb=2,
    ),
    runtime=limits.runtime_by_input(
        base_minutes=5,
        minutes_per_gib=1,
    ),

Keep system limits in configuration and use lookup without defaults so missing required settings fail clearly. These limits cap estimates made through the object; fixed resource requests in rules and scheduler limits are separate. The standalone utils.memory_by_input and utils.runtime_by_input factories remain available with an explicit maximum. The factory captures the coefficients and limit during parsing, then returns a function accepting wildcards and input. Snakemake evaluates that function when the job's inputs are available; missing inputs remain <TBD> during dry runs. Memory is ceil(base_mem_mb + mem_per_input_mb * input.size_mb); runtime is ceil(base_minutes + minutes_per_gib * input.size_mb / 1024), each capped at its maximum. Estimates apply to the whole job, not per thread.

Snakemake 8.14 resource functions receive no params or other resource values. Keep resource coefficients here or in workflow configuration, rather than introducing artificial scheduler resources. The lower-level dynamic_mem_mb and dynamic_runtime functions remain available for direct numeric estimates. Metadata loading and scientific validation stay in the workflow.

external_path(value, label, repository_dir) resolves a path against the current run directory and rejects the repository itself and its descendants, including paths that resolve into it through symlinks. It returns an absolute Path without requiring that the file exists. Pass the repository boundary explicitly:

sample_sheet = utils.external_path(
    config["sample_sheet"], "Sample sheet", repository_dir=repository_dir
)

Enable benchmark-extended: true in execution profiles to record job metadata and historical input sizes. The resource report uses those values when available, compares requested and measured resources, and exports measurements.tsv; ordinary benchmarks remain supported. nexus-resource-report --help describes standalone inspection of existing benchmark outputs. See resource reporting. The success and failure hooks retain existing outputs and accounting semantics; missing site services are reported as warnings.

Singularity image proxy

Installation exposes singularity in the environment. Ensure that environment's bin precedes the real runtime on PATH. Discovery skips this launcher; set SNAKEMAKE_REAL_SINGULARITY=/absolute/path/to/real/singularity when needed. Do not point it at the proxy. Native Apptainer can be supplied through this variable too.

export LEOMED_SNAKEMAKE_IMAGE_DIR=/path/to/shared/images
export SNAKEMAKE_CONTAINER_PROXY_CONFIG=/path/to/site/container-proxies.json
singularity list
singularity list '*samtools*'
singularity locate docker://python:3.14
singularity rm 'docker://python:3.14*'
singularity pull docker://python:3.14
singularity exec docker://python:3.14 python --version
singularity --image-dir /another/cache list

To persist the proxy configuration path for a Conda environment:

conda env config vars set --prefix /path/to/env \
  SNAKEMAKE_CONTAINER_PROXY_CONFIG=/path/to/site/container-proxies.json

Reactivate the environment afterward. Image pulls warn on stderr when the proxy configuration variable is unset or its JSON mapping is empty; direct pulls remain allowed. Version checks and other native commands stay quiet.

Use the same image directory variable in a Snakemake profile:

apptainer-prefix: "$LEOMED_SNAKEMAKE_IMAGE_DIR"

An explicit --image-dir before the command or among its leading options wins over the environment. Listing and URL resolution require a location; local image paths and unrelated native commands do not. Quote URL globs to prevent shell expansion. list prints downloaded URLs, paths and usage metadata as TSV, omitting missing files and showing unmapped .simg files with URL unknown. It does not modify the manifest. locate prints only an existing absolute path.

rm URL-GLOB removes cached images matching the same original URL patterns as list, using the same per-image locks as pulls. Quote patterns, for example singularity rm '*python*'. An explicit URL or pattern is required; no matches return an error. Inventory history is retained and deleted images are marked missing. As with list, '*' includes images whose URL is unknown. Remove shared images only when no jobs are using them. This command does not clean the runtime's separate layer cache. Other native commands, including cache, instance and help, continue to pass through unchanged; rm inside container commands is also unaffected.

exec, run and shell resolve the image operand only. URLs in runtime option values or in the container's arguments are preserved. Runtime help supplies option arity, supporting the installed runtime's flags. Local paths and instance:// references pass through unchanged. Missing URL images fail with an explicit pull command; execution never downloads implicitly. Usage denotes an execution attempt, not successful completion. Mutable tags reuse their existing image until an explicit forced pull.

Unnamed pulls write <md5(exact-original-url)>.simg under the selected directory. Explicit filenames and Snakemake's pull --name ... retain their original output location. Mirror rewriting never changes the hash. Keep URLs exactly consistent; registry aliases are not normalized. Use an explicit destination for sandbox pulls. The proxy records file-backed images in container_list.tsv, retaining the existing schema, per-image locks, atomic manifest updates and per-workflow usage timestamps.

Proxy mappings are an optional JSON object of source prefixes to replacements. Longest prefix wins. The generic docker:// mapping applies to Docker Hub while preserving explicit other registries. Without configured mappings, URLs pass through unchanged. Keep site addresses in site configuration, outside this package.

Native commands, including cache list, pass through. This workflow image store is separate from the runtime's layer cache. Existing --proxy-capabilities, --proxy-register, --proxy-real-runtime and --proxy-audit DIRECTORY interfaces remain supported. Audit refreshes presence information and discovers unmapped images; it does not recover their original URLs.

For contributing, validation and releases, see the separate developer guide.