Skip to main content

"Staging" for Snakemake

This package provides a mechanism for Snakemake workflows to explicitly "stage out" the output files from certain rules to a public repository like Zenodo to allow faster re-execution of the workflow, using these previously generated artifacts. This can be especially useful for workflows with computationally expensive rules that don't need to be frequently re-run.

snakemake-staging is a spin-off of the showyourwork project, which provides a "caching" framework for Snakemake workflows, to transparently avoid re-execution of rules that have been cached to Zenodo. The implementation of this logic in showyourwork is, however, somewhat fragile and unpredictable. In snakemake-staging, we take a more explicit approach, where "staged" rules are always either explicitly executed or restored.

Installation

To use snakemake-staging in your workflow, you can install it using pip (it's probably best to set up your Snakemake installation following the Snakemake docs first):

python -m pip install snakemake-staging

Quickstart

The Snakefile

While testing, it's probably best to use the Zenodo Sandbox, rather than the main site, since any archive published to Zenodo is permanent. To use the sandbox, you'll need a personal access token stored in the SANDBOX_TOKEN environment variable. You can generate a new token here.

Once you've added this token to your environment, you can edit the Snakefile for your workflow to use snakemake-staging as follows. First, towards the top of your Snakefile, add:

import snakemake_staging as staging

stage = staging.ZenodoStage(
    "zenodo-stage",
    config.get("restore", False)
)

to create a new stage called zenodo-stage. Note that here we're extracting a restore flag from the Snakemake config, which will be used to determine whether to restore files for the stage. This means that you can control the behavior of this stage from the command line. By passing --config restore=True to the snakemake command line interface, all files staged out by the zenodo-stage stage will be restored from the archive rather than generated.

Then, to stage out a rule, you can apply the stage as follows:

rule expensive:
    input:
        ...
    output:
        stage(
            "path/to/output1.txt",
            "path/to/output2.txt",
        )
    shell:
        ...

Finally, after defining all the rules that you want to stage out, you must add the following include which defines all the staging rules:

include: staging.snakefile()

At this point, here's the full Snakefile:

Full Snakefile
import snakemake_staging as staging

stage = staging.ZenodoStage(
    "zenodo-stage",
    config.get("restore", False)
)

rule expensive:
    input:
        ...
    output:
        stage(
            "path/to/output1.txt",
            "path/to/output2.txt",
        )
    shell:
        ...

include: staging.snakefile()

Usage

With the Snakefile defined in the previous section, you can now run your workflow in 3 ways:

  1. Normal execution: If you run something like snakemake path/to/output1.txt (where I have omitted the usual --cores and --conda arguments) will execute the workflow as normal, without staging out any files.

  2. Stage upload: If you instead have Snakemake target the staging__upload rule, the expensive rule will be executed, and the outputs will be uploaded to Zenodo, saving the record information to zenodo-stage.zenodo.json (this filename can be changed by passing the info_file argument to the ZenodoStage constructor).

  3. Stage restore: Finally, after these outputs have been uploaded to Zenodo, you can call Snakemake --config restore=True to disable the expensive rule, and force the outputs to be restored from Zenodo.

Metadata

Release files for snakemake-staging 0.0.2

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for snakemake-staging 0.0.2
File Size Uploaded Provenance Actions
snakemake_staging-0.0.2.tar.gz 14.9 kB Without provenance Details for snakemake_staging-0.0.2.tar.gz

Built distribution (wheel)

Table of built distributions (wheels) for snakemake-staging 0.0.2
File Interpreter ABI Platform Actions
snakemake_staging-0.0.2-py3-none-any.whl Python 3 none any Details for snakemake_staging-0.0.2-py3-none-any.whl

Total release size: 32.8 kB

Release files / snakemake_staging-0.0.2.tar.gz

Download URL snakemake_staging-0.0.2.tar.gz
Size 14.9 kB
Tags Source
SHA-256 checksum
How to use checksums
e3e783db2998662e25a6c9009a9ed57abef606c27bee8050d724eb15da164ed9
BLAKE2b-256 checksum
How to use checksums
61eeaacf9faab85333a05dd29bac95a1c8d7c1003e15faba8fbb1139f6f39d0b
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
Yes
Uploaded via twine/4.0.1 CPython/3.11.4

Release files / snakemake_staging-0.0.2-py3-none-any.whl

Download URL snakemake_staging-0.0.2-py3-none-any.whl
Size 17.9 kB
Tags Python 3
SHA-256 checksum
How to use checksums
8d96323d390c349a75639318a93c137578a4ddfea4627f31d76582071fe53751
BLAKE2b-256 checksum
How to use checksums
85951868b907ee8db840697d3a0b2bd1d618ed938e4b1e220c2e8d627a41ec9c
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
Yes
Uploaded via twine/4.0.1 CPython/3.11.4

Security for snakemake-staging 0.0.2

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

No provenance found

None of the files in this release were uploaded with provenance metadata.

A note on package security

PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.

To learn more about protecting yourself from malicious actors, visit our package security documentation.

Release history Release notifications | RSS feed

This release

0.0.2 This release

2 release files No provenance

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page