Nexus Snakemake utilities

Shared workflow infrastructure for Python 3.11+ on Linux/macOS, targeting Snakemake 8.14. Newer Snakemake versions are allowed but remain unverified. Install into the Conda environment that runs Snakemake:

python -m pip install 'nexus-snakemake-utils @ git+https://github.com/ETH-NEXUS/nexus_snakemake_utils.git@v0.2.0'
# Local development, from a neighboring workflow checkout:
python -m pip install -e ../nexus_snakemake_utils

Publish the v0.2.0 tag before using the pinned installation command. Earlier tags contain the old package. Until the tag is available remotely, use the local editable installation. Packaging declares the controller dependencies. This does not install analysis or R dependencies.

Portability

Nexus/Leomed access and Slurm are not required. The supported environment is Linux/macOS with Python 3.11+ and Snakemake 8.14 as the tested target. Packaging requires snakemake>=8.14 without an upper bound; this permits Snakemake 9 but does not establish compatibility. Validation of newer versions is deferred. Core utilities and benchmark reports work with the local executor. Slurm accounting is queried only for Slurm runs; missing accounting tools or records produce warnings without failing the workflow. Container usage registration skips runtimes without the proxy interface.

The optional container proxy uses Unix file locking and requires a real Singularity/Apptainer runtime for container execution. Its singularity launcher is installed with this package and must find the real runtime later on PATH or through SNAKEMAKE_REAL_SINGULARITY. Set an image directory with --image-dir or LEOMED_SNAKEMAKE_IMAGE_DIR when using its image-cache commands. The variable name does not require Leomed storage; mirror mappings are optional. Keep cluster paths, accounts and binds in site profiles, and publish a local profile too.

Reporting libraries are installed as package dependencies even if reporting is unused. Analysis tools and report-rendering software remain workflow dependencies.

Workflow interface

import nexus_snakemake_utils as utils
# Top level of the Snakefile: runs during parsing, including dry runs.
# Preserve an explicit container workflow label, otherwise use the repository
# name; then log time, user, host, Git revision/checkout state and Snakemake version.
utils.initialize(repository_dir)

onstart:
    utils.register_container_usage(workflow)
onsuccess:
    utils.write_resource_report(workflow)
onerror:
    utils.write_resource_report(workflow, partial=True)

initialize preserves an explicit SNAKEMAKE_CONTAINER_WORKFLOW label and otherwise sets it to the repository name, then calls log_run_info. It does not register container usage, execute jobs, or write resource reports. Keep it at the top level of the Snakefile so logging also runs for dry runs.

Select optional inputs or targets with optional(val, on, dpath=None):

rule all:
    input:
        utils.optional(rules.report.output.html, config, dpath="optional/report"),

This returns val unchanged when the flag is truthy, otherwise []. Omit dpath to pass a flag directly, for example utils.optional(targets, enabled). Config paths use dpath syntax; missing or ambiguous paths raise errors. Define flags explicitly in the base configuration (for example optional: {report: false}) and enable them through overlays. Values are evaluated before calling the helper; use lazy input functions when resolving a value itself requires optional files.

Use callable factories directly in a rule's resources block:

resources:
    mem_mb=utils.memory_by_input(
        base_mem_mb=2000,
        mem_per_input_mb=2,
        maximum=lookup(dpath="resources/max_mem_mb", within=config),
    ),
    runtime=utils.runtime_by_input(
        base_minutes=5,
        minutes_per_gib=1,
        maximum=lookup(dpath="resources/max_minutes", within=config),
    ),

Both factories require an explicit maximum. Keep limits in workflow configuration and use lookup without defaults so missing required settings fail clearly. The factory captures the coefficients and limit during parsing, then returns a function accepting wildcards and input. Snakemake evaluates that function when the job's inputs are available; missing inputs remain <TBD> during dry runs. Memory is ceil(base_mem_mb + mem_per_input_mb * input.size_mb); runtime is ceil(base_minutes + minutes_per_gib * input.size_mb / 1024), each capped at its maximum. Estimates apply to the whole job, not per thread.

Snakemake 8.14 resource functions receive no params or other resource values. Keep resource coefficients here or in workflow configuration, rather than introducing artificial scheduler resources. The lower-level dynamic_mem_mb and dynamic_runtime functions remain available for direct numeric estimates. Metadata loading and scientific validation stay in the workflow.

external_path(value, label, repository_dir) resolves a path against the current run directory and rejects the repository itself and its descendants, including paths that resolve into it through symlinks. It returns an absolute Path without requiring that the file exists. Pass the repository boundary explicitly:

sample_sheet = utils.external_path(
    config["sample_sheet"], "Sample sheet", repository_dir=repository_dir
)

Enable benchmark-extended: true in execution profiles to record job metadata and historical input sizes. The resource report uses those values when available, compares requested and measured resources, and exports measurements.tsv; ordinary benchmarks remain supported. nexus-resource-report --help describes standalone inspection of existing benchmark outputs. See resource reporting. The success and failure hooks retain existing outputs and accounting semantics; missing site services are reported as warnings. All Snakemake private API access belongs in hooks/resource_report.py.

Singularity image proxy

Installation exposes singularity in the environment. Ensure that environment's bin precedes the real runtime on PATH. Discovery skips this launcher; set SNAKEMAKE_REAL_SINGULARITY=/absolute/path/to/real/singularity when needed. Do not point it at the proxy. Native Apptainer can be supplied through this variable too.

export LEOMED_SNAKEMAKE_IMAGE_DIR=/path/to/shared/images
export SNAKEMAKE_CONTAINER_PROXY_CONFIG=/path/to/site/container-proxies.json
singularity list
singularity list '*samtools*'
singularity locate docker://python:3.14
singularity pull docker://python:3.14
singularity exec docker://python:3.14 python --version
singularity --image-dir /another/cache list

Use the same variable in a Snakemake profile:

apptainer-prefix: "$LEOMED_SNAKEMAKE_IMAGE_DIR"

An explicit --image-dir before the command or among its leading options wins over the environment. Listing and URL resolution require a location; local image paths and unrelated native commands do not. Quote URL globs to prevent shell expansion. list prints downloaded URLs, paths and usage metadata as TSV, omitting missing files and showing unmapped .simg files with URL unknown. It does not modify the manifest. locate prints only an existing absolute path.

exec, run and shell resolve the image operand only. URLs in runtime option values or in the container's arguments are preserved. Runtime help supplies option arity, supporting the installed runtime's flags. Local paths and instance:// references pass through unchanged. Missing URL images fail with an explicit pull command; execution never downloads implicitly. Usage denotes an execution attempt, not successful completion. Mutable tags reuse their existing image until an explicit forced pull.

Unnamed pulls write <md5(exact-original-url)>.simg under the selected directory. Explicit filenames and Snakemake's pull --name ... retain their original output location. Mirror rewriting never changes the hash. Keep URLs exactly consistent; registry aliases are not normalized. Use an explicit destination for sandbox pulls. The proxy records file-backed images in container_list.tsv, retaining the existing schema, per-image locks, atomic manifest updates and per-workflow usage timestamps.

Proxy mappings are an optional JSON object of source prefixes to replacements. Longest prefix wins. The generic docker:// mapping applies to Docker Hub while preserving explicit other registries. Without configured mappings, URLs pass through unchanged. Keep site addresses in site configuration, outside this package.

Native commands, including cache list, pass through. This workflow image store is separate from the runtime's layer cache. Existing --proxy-capabilities, --proxy-register, --proxy-real-runtime and --proxy-audit DIRECTORY interfaces remain supported. Audit refreshes presence information and discovers unmapped images; it does not recover their original URLs.

Maintenance and validation

Use Black and Flake8 (88 columns, ignoring E203 for Black compatibility). Keep Snakemake 8.14 compatibility and private APIs confined to the resource-report module. No automatic test suite is introduced. Verify changed branches with focused local execution and temporary caches; never use a production image cache for development checks.

On Leomed, manually verify dry runs, prefetching, cache listing and a known URL execution against the configured cache. Check successful and failed resource reports and repeat delayed accounting inspection using existing outputs. Suggest these checks without executing cluster jobs automatically. Publish packages, Git tags and site updates separately; no push is part of local maintenance.