wfcommons 1.4

A Framework for Enabling Scientific Workflow Research and Education

           

An open-source ecosystem of workflow execution instances, synthetic workflow generators, and benchmark specifications. It helps the community study scheduling, performance, resilience, and emerging AI-driven workflow automation on modern distributed and HPC platforms.

• Real instances: Workflow executions curated in a common JSON format (WfFormat).
• Synthetic realism: Generate realistic workflows from real traces.
• Benchmarks: Produce executable specs for repeatable experiments and fair comparisons.

Quick links: Documentation · Website · GitHub Issues

Quickstart

WfCommons requires Python 3.11+ and has been tested on Linux and macOS.

python3 -m venv .venv
source .venv/bin/activate
python3 -m pip install wfcommons

Generate a synthetic workflow in a few lines:

import pathlib
from wfcommons.wfchef.recipes import SeismologyRecipe
from wfcommons import WorkflowGenerator

generator = WorkflowGenerator(SeismologyRecipe.from_num_tasks(250))
workflow = generator.build_workflow()
workflow.write_json(pathlib.Path("seismology-workflow.json"))

Next steps:

• Learn how to build recipes in the WfChef guide.
• Generate larger workflow families in the WfGen guide.
• Produce benchmark specs in the WfBench guide.

Installation

WfCommons is available on PyPI.

python3 -m pip install wfcommons

Installing from source (latest)

git clone https://github.com/wfcommons/wfcommons
cd wfcommons
python3 -m pip install .

Optional Requirements

Graphviz

WfCommons uses pygraphviz for generating visualizations for the workflow task graph. If you want to enable this feature, you will have to install the graphviz package (version 2.16 or later). You can install graphviz easily on Linux with your favorite package manager, for example for Debian-based distributions:

sudo apt-get install graphviz libgraphviz-dev

and for RedHat-based distributions:

sudo yum install python-devel graphviz-devel

On macOS you can use brew package manager:

brew install graphviz

Then you can install pygraphviz by running:

python3 -m pip install pygraphviz

pydot

WfCommons uses pydot for reading and writing DOT files. If you want to enable this feature, you will have to install the pydot package:

python3 -m pip install pydot

Get in Touch

The main channel to reach the WfCommons team is via the support email: [e-mail].

Bug Report / Feature Request: our preferred channel to report a bug or request a feature is via WfCommons's Github Issues Track.

Citing WfCommons

When citing WfCommons, please use the following paper. You should also actually read that paper, as it provides a recent and general overview on the framework.

@article{wfcommons,
    title = {{WfCommons: A Framework for Enabling Scientific Workflow Research and Development}},
    author = {Coleman, Taina and Casanova, Henri and Pottier, Loic and Kaushik, Manav and Deelman, Ewa and Ferreira da Silva, Rafael},
    journal = {Future Generation Computer Systems},
    volume = {128},
    number = {},
    pages = {16--27},
    doi = {10.1016/j.future.2021.09.043},
    year = {2022},
}