unionbio 0.1.0

No project description provided

🧬 Bioinformatics on Flyte

---

This repo contains tasks, workflows, image definitions, and datatypes used to standardize the orchestration of common bioinformatics tasks using Flyte.

🐳 Container Images

✨Adding custom dependencies alongside Flytekit

ImageSpecs contained in the images module build a standard set of OCI-compliant container images for use throughout the different workflows. They can be built with entrypoints present in pyproject.toml.

📈 Datatypes

✨Leverage dataclasses to keep things organized

Using dataclasses to define your samples provides a clean and extensible data structure to keep your workflows tidy. Instead of writing to directories and keeping track of things manually on the commandline, these dataclasses will capture relevant metadata about your samples and let you know where to find them in object storage.

🔍 Quality Control and Pre-processing

FastQC

✨Run arbitrary shell commands

FastQC is a very common tool written in Java for gathering QC metrics about raw reads. It doesn't have any python bindings, but luckily Flyte lets us run arbitrary ShellTasks with a clean way of passing in inputs and receiving outputs. Just define a script for what you need to do and ShellTask will handle the rest.

Automatic QC checkpointing

✨Decide wether to continue workflow execution based on QC metrics via conditionals

FastQC generates a summary file with a simple PASS / WARN / FAIL call across a number of different metrics. We can use conditionals in our workflow to check for any FAIL lines in the summary and automatically halt execution. This can surface an early failure without wasting valuable compute or anyone's time doing manual review.

FastP

✨Specify resources and parallelize via map task

FastP is another common pre-processing tool for filtering out bad reads, trimming, and adapter removal. It can be a bit more memory hungry than Flyte's defaults are set to; luckily we can use Resources to bump that up and allow it to run efficiently. Additionally, we can make use of a map task in our workflow to parallelize the processing of fastp across all our samples.

👩‍🔬 Human-in-the-Loop Approval

✨Pause processing while waiting for human input

As a final check before moving onto the alignment, we can define an explicit approval right in the workflow. Aggregating reports of all processing done up to this point, and visualizing it via Decks (more on that later), a researcher is able to quickly get a high level view of the work done so far and approve the analysis for further processing.

📏 Alignment

Generate indices

✨Leverage caching to save time on successive runs

Index generation can be a very compute intensive step. Luckily, we can take advantage of Flyte's native caching when building that index for bowtie and hisat. We've also defined a cache_version in the config that relies on a hash of the reference location in the object store. This means that changing the reference will invalidate the cache and trigger a rebuild, while allowing you to go back to your old reference with impunity.

Bowtie2 vs Hisat2

✨Compare aligners across an arbitrary number of inputs via dynamic workflows

When prototyping a new pipeline, it's usually a good idea to evaluate a few different tools to see how they perform with respect to runtime and resource requirements. This is easy with a dynamic workflow, which allows us to pass in an arbitrary number of inputs to be used with whatever tasks we want. In the main workflow you'll pass a list of filtered samples to each tool and be able to capture run statistics in the Alignment dataclass as well as visualize their runtimes in the Flyte console.

📋 Reporting

✨Visualize performance via Decks

We use MultiQC, an excellent multi-modal visualization tool for reporting. After gathering all relative metrics from a workflow, we're able to render that report via Decks, giving us rich run statistics without ever leaving the Flyte console!