miniwdl (Fovus fork)

This is an unofficial Fovus fork of miniwdl, published to PyPI as the distribution miniwdl-fovus-runtime (the import package is still WDL). It adds non-POSIX shared-storage support required by the miniwdl-fovus backend. It is not the official miniwdl and is not endorsed by CZI. See NOTICE and CHANGES.md. Upstream: https://github.com/chanzuckerberg/miniwdl

Fork base: upstream main @ 8443ffd (post v1.14.2).

Maintaining this fork

The Fovus deltas live as commits directly on fovus-main (this repo's default branch) on top of a pinned upstream base commit. Upstream is a git remote, not a local mirror branch; rebase fovus-main onto newer upstream tags as needed.

Bump to a newer upstream release:

git remote add upstream https://github.com/chanzuckerberg/miniwdl.git   # once
git fetch upstream --tags
# rebase the Fovus commits on fovus-main onto the new base (e.g. v1.15.0)
git rebase --onto v1.15.0 <old-base> fovus-main
# resolve conflicts, then:
pip install -e ".[dev]" && make qtest                # re-run the fork tests
# bump packaging + provenance:
#   - pyproject.toml  version = "<nearest-tag>.postN"
#   - NOTICE / CHANGES.md  fork base + change summary
#   - miniwdl-fovus pyproject.toml  pin miniwdl-fovus-runtime == <new version>
git push --force-with-lease origin fovus-main        # fovus-main history was rewritten
git tag v<new version> && git push origin v<new version>   # triggers the release workflow

Shrink the fork: as upstream merges the generic patches (see CHANGES.md), drop them on the next rebase; the goal is eventually depending on stock miniwdl.

Workflow Description Language local runner & developer toolkit for Python 3.8+

Install miniwdl

Installation requires Python 3.8+, pip (or conda) and Docker (or Podman/Singularity/udocker). Linux preferred; macOS compatible with extra steps. More detail in full documentation.

• Install with pip : run pip3 install miniwdl
• Install with conda : setup conda-forge and run conda install miniwdl
• Verify your miniwdl installation: miniwdl run_self_test
• Install from source code for development: see CONTRIBUTING.md

Use miniwdl

Run an example bioinformatics WDL pipeline using miniwdl, or learn more abut miniwdl via a short course (screencast examples). If you are new to the WDL language, see the open source learn-wdl course.

• Run an example using a viral genome assembly workflow
• Learn miniwdl course w/screencasts - shown below
• Learn WDL course w/screencasts

Documentation

The online documentation includes a user tutorial, reference manual, and Python development codelabs:

See the Releases for change logs. The Project board shows the current prioritization of issues.

Scaling up

The miniwdl runner schedules WDL tasks in parallel up to the CPUs & memory available on the local host; so a more-powerful host enables larger workloads. Separately-maintained projects can distribute tasks to cloud & HPC backends with a shared filesystem:

• AWS:
  • miniwdl-omics-run tool for the Amazon Omics workflow service
  • AWS Batch plugin (DIY)
• SLURM

Getting Help

• Open an issue
• OpenWDL Slack (#miniwdl channel)
• Bioinformatics Stack Exchange

Contributing

Feedback and contributions to miniwdl are welcome, via issues and pull requests on this repository. See CONTRIBUTING.md for guidelines, instructions to set up your development environment, and a codebase overview.

Security

Please disclose security issues responsibly by contacting security@chanzuckerberg.com.

Code of Conduct

This project adheres to the Contributor Covenant code of conduct. By participating, you are expected to uphold this code. Please report unacceptable behavior to opensource@chanzuckerberg.com.