Snakemake workflow: dvp-imaging-pipeline

A Snakemake workflow for Scalable processing of DVP imaging data with snakemake.

Usage

Detailed information about input data and workflow configuration can also be found in the workflow/config/README.md.

Deployment options

1. Clone this repository or download it via (Code > Download ZIP)

git clone https://github.com/lucas-diedrich/dvp-imaging-pipeline.git

2. Change your working directory into the repository

cd path/to/dvp-imaging-pipeline

3. Create and activate the environment that provides snakemake. The environment is named snakemake by environment.yaml.

conda env create -n snakemake --file environment.yaml && conda activate snakemake

4. Test your setup on a small artificial dataset before running the workflow on your own data

snakemake --cores 2 --sdm conda --directory workflow --configfile tests/integration/config/config.yaml

Results (spatialdata objects, logs, and plots) are written to tests/integration/results.

5. Adapt the configuration files

Adjust options in the config file workflow/config/config.yaml. Use the test file tests/integration/config/config.yaml as template.

6. Run the workflow

Run the workflow locally (not recommended for large datasets).

snakemake --cores all --sdm conda --directory workflow --configfile workflow/config/config.yaml

For large datasets, submit the workflow to a SLURM cluster instead, see HPC below.

HPC

You can run this workflow on a high-performance computing cluster.

On the cluster, create the snakemake base environment:

conda env create --file environment.yaml

Then submit the provided workflow script on a cluster. Please check the script and the official snakemake slurm plugin documentation to learn about relevant flags and settings.

cd workflow
sbatch snakemake.sbatch

Logging

Monitor the progress of the workflow in the workflow directory with snkmt, see also the documentation.

The snkmt logger is not part of environment.yaml and has to be installed into the snakemake environment separately:

pip install snakemake-logger-plugin-snkmt

Add --logger snkmt to the snakemake calls above, then follow the run from the workflow directory:

snkmt console

Install utility functions

The utility functions used by the workflow (source in src/) are published as the python package dvpimg.

conda create -n dvpimg python=3.12 -y && conda activate dvpimg
# only pre-releases are available so far
pip install --pre dvpimg

To install the version in this repository instead, run pip install . from the repository root.

Components

See the documentation for more information on the individual components and detailed instructions on their configuration.

References

(alphabetical order)

dvp-io: MannLabs/dvp-io. Lucas Diedrich (2025). https://github.com/MannLabs/dvp-io.git

Harpy: saeyslab/harpy. Saeys Lab (2025). https://github.com/saeyslab/harpy.git

Lazyslide: Zheng, Y., Abila, E., Chrenková, E., Winkler, J. & Rendeiro, A. F. LazySlide: accessible and interoperable whole slide image analysis. 2025.05.28.656548 Preprint at https://doi.org/10.1101/2025.05.28.656548 (2025).

Snakemake Köster, J., Mölder, F., Jablonski, K. P., Letcher, B., Hall, M. B., Tomkins-Tinch, C. H., Sochat, V., Forster, J., Lee, S., Twardziok, S. O., Kanitz, A., Wilm, A., Holtgrewe, M., Rahmann, S., & Nahnsen, S. Sustainable data analysis with Snakemake. F1000Research, 10:33, 10, 33, 2021. https://doi.org/10.12688/f1000research.29032.2.

Spatialdata: Marconato, L. et al. SpatialData: an open and universal data framework for spatial omics. Nat Methods 1–5 (2024) doi:10.1038/s41592-024-02212-x.

Citation

If you use this workflow in a paper, don't forget to give credits to the authors by citing the URL of this repository or its DOI.

DVP-Imaging-Pipeline [Computer software]. https://github.com/lucas-diedrich/dvp-imaging-pipeline.git