Skip to main content

StreamFlow framework

Project description


CWL Conformance

The StreamFlow framework is a container-native Workflow Management System (WMS) written in Python 3. It has been designed around two main principles:

  • Allow the execution of tasks in multi-container environments, in order to support concurrent execution of multiple communicating tasks in a multi-agent ecosystem.
  • Relax the requirement of a single shared data space, in order to allow for hybrid workflow executions on top of multi-cloud or hybrid cloud/HPC infrastructures.

Use StreamFlow


The StreamFlow module is available on PyPI, so you can install it using pip.

pip install streamflow

Please note that StreamFlow requires python >= 3.8. Then you can execute it directly from the CLI

streamflow run /path/to/streamflow.yml


StreamFlow Docker images are available on Docker Hub. In order to run a workflow inside the StreaFlow image

  • A StreamFlow project, containing a streamflow.yml file and all the other relevant dependencies (e.g. a CWL description of the workflow steps and a Helm description of the execution environment) need to be mounted as a volume inside the container, for example in the /streamflow/project folder
  • Workflow outputs, if any, will be stored in the /streamflow/results folder. Therefore, it is necessary to mount such location as a volume in order to persist the results
  • StreamFlow will save all its temporary files inside the /tmp/streamflow location. For debugging purposes, or in order to improve I/O performances in case of huge files, it could be useful to mount also such location as a volume
  • The path of the streamflow.yml file inside the container (e.g. /streamflow/project/streamflow.yml) must be passed as an argument to the Docker container

The script below gives an example of StreamFlow execution in a Docker container

docker run -d \
    --mount type=bind,source="$(pwd)"/my-project,target=/streamflow/project \
    --mount type=bind,source="$(pwd)"/results,target=/streamflow/results \
    --mount type=bind,source="$(pwd)"/tmp,target=/tmp/streamflow \
    alphaunito/streamflow run /streamflow/project/streamflow.yml


It is also possible to execute the StreamFlow container as a Job in Kubernetes. In this case, StreamFlow is able to deploy Helm models directly on the parent cluster through the ServiceAccount credentials. In order to do that, the inCluster option must be set to true for each involved module on the streamflow.yml file

    type: helm
      inCluster: true

A Helm template of a StreamFlow Job can be found in the helm/chart folder.

Please note that, in case RBAC is active on the Kubernetes cluster, a proper RoleBinding must be attached to the ServiceAccount object, in order to give StreamFlow the permissions to manage deployments of pods and executions of tasks.

CWL Compatibility

StreamFlow relies on the Common Workflow Language (CWL) standard to design workflow models. CWL conformance badges for StreamFlow are reported below.

CWL v1.0


Required features

Optional features

CWL v1.1


Required features

Optional features

CWL v1.2


Required features

Optional features

Contribute to StreamFlow

As a first step, get StreamFlow from GitHub

git clone

Then you can install all the requred packages using the pip install command

cd streamflow
pip install .

StreamFlow relies on GitHub Actions for PyPI and Docker Hub distributions. Therefore, in order to publish a new version of the software, you only have to augment the version number in file.

StreamFlow Team

Iacopo Colonnelli (creator and maintainer)
Barbara Cantalupo (maintainer)
Marco Aldinucci (maintainer)

Gaetano Saitta (contributor)
Alberto Mulone (contributor)

Project details

Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

streamflow-0.1.6.tar.gz (150.7 kB view hashes)

Uploaded source

Built Distribution

streamflow-0.1.6-py2.py3-none-any.whl (175.1 kB view hashes)

Uploaded py2 py3

Supported by

AWS AWS Cloud computing and Security Sponsor Datadog Datadog Monitoring Fastly Fastly CDN Google Google Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Sentry Sentry Error logging StatusPage StatusPage Status page