sophios 0.4.0

DSL for inferring the edges of a CWL workflow DAG

Sophios

Scientific computing can be difficult in practice due to various complex software issues. In particular, chaining together software packages into a computational pipeline can be very error prone. Using the Common Workflow Language (CWL) greatly helps, but like many other workflow languages users still need to explicitly specify how to connect inputs & outputs. Sophios allows users to specify computational protocols at a very high level of abstraction, it automatically infers almost all connections between inputs & outputs, and it compiles to CWL for execution.

Documentation

The documentation is available on readthedocs.

Quick Start

See the installation guide for more details, but:

For pip users:

pip install sophios

In order to execute the CWL workflows that are generated by sophios, cwltool and all of its underlying dependencies need to be present in the system. Unfortunately pip has no capability to resolve and install these dependencies. PLease refer to the cwltool installation guide to prepare the system to run CWL workflows.

For conda users / developers:

See the installation guide for developers

sophios --yaml ../workflow-inference-compiler/docs/tutorials/helloworld.wic --graphviz --run_local --quiet

Sophios is a Domain Specific Language (DSL) based on the Common Workflow Language. CWL is fantastic, but explicitly constructing the Directed Acyclic Graph (DAG) associated with a non-trivial workflow is not so simple. Instead of writing raw CWL, users can write workflows in a much simpler yml DSL. For technical reasons edge inference is far from unique, so users should always check that edge inference actually produces the intended DAG.

Edge Inference

The key feature is that in most cases, users do not need to specify any of the edges! They will be automatically inferred for users based on types, file formats, and naming conventions. For more information, see the user guide If for some reason edge inference fails, there is a syntax for creating explicit edges.

Subworkflows

Subworkflows are very useful for creating reusable, composable building blocks. As shown above, recursive subworkflows are fully supported, and the edge inference algorithm has been very carefully constructed to work across subworkflow boundaries.

Explicit CWL

Since the yml DSL files are automatically compiled to CWL, users should not have to know any CWL. However, the yml DSL is secretly CWL that is simply missing almost all of the tags! In other words, the compiler merely adds missing information to the files, and so if the users know CWL, they are free to explicitly add the information themselves. Thus, the yml DSL is intentionally a leaky abstraction.

Python API

In addition to the underlying declarative yaml syntax, there is an API for writing WIC workflows in python. The python API is philosophically the exact opposite: users should not have to know any CWL, and in fact all CWL features are hidden unless explicitly exposed.