nf-meta: metapipeline representation, editor, and runner

This project features a reproducible representation for meta-pipelines, featuring an interactive editor running locally in a browser, as well as a cli based validator and runner.

This is a beta version: Things can be unstable and change

Find a growing collection of examples in our dedicated showcase repository.

Representation

This project proposes a representation for how metapipeline can be generalized in a in .yml file format. The most important keys for a metapipeline in this format are:

• config_version - which allows the representation to change over time without breaking
• workflows - lists the nextflow / nf-core pipeline runs. The flow from workflow to workflow is described by transitions defined with the from: key
• globals - defines settings that apply to all workflows

config_version: 0.3.0
globals:
  nextflow_version: 25
workflows:
  n01:
    name: nf-core/fetchngs
    version: dev
    params:
      input: fetchngs-ids.csv
      outdir: out_fetchngs
      nf_core_pipeline: rnaseq
      nf_core_rnaseq_strandedness: auto
  n02:
    from: n01
    name: nf-core/rnaseq
    version: dev
    params:
      input: ${n01:params:outdir}/samplesheet/samplesheet.csv
      genome: EB1
      outdir: out_rnaseq   

This project features at it core validation logic for these sort of .yml representations.

To enable referencing fields form other workflows, constructs like could be made possible:

    # write this:
    input: ${n01:params:outdir}/samplesheet/samplesheet.csv
    # instead of this:
    input: fetchngs_out/samplesheet/samplesheet.csv

Editor

This project features a small editor which intends to ease the creation and updating of these config files, by visualizing the config as a graph and offering user-firendly form for entering and validating values.

Metapipeline Runners

Currently:

• Python Runner: Wraps nextflow commands, exectutes in dag order

Planned:

1. nf-cascade runner: Convert config into one nextflow daisy-chaining nextflow script
2. Seqera Platform runner: Start and monitor locally, call and poll a Platform instance with run parameters via API to handle running Nextflow
3. Meta-Pipeline runner: compile config into new monolithic nextflow project, that imports required workflows to achieve most efficient orchestration + graceful errors handling

Install from PyPI

The released version of this package can be installed from the Python Package Index

# with pip
pip install nf-meta

# with uv 
uv add nf-meta

Install and run the Development Setup

Install and run the frontend

$ cd /src/nf_meta/editor/frontend
$ npm install
$ npm run dev

Run the cli in dev mode

NF_META_DEVMODE=1 uv run nf-meta editor [</path/to/metapipeline.yml>]

Project Structure

• Engine: Config Validation, Editor Session with History and exposing API
• Runners
• Editor frontend for creating or editing a config