miniwdl-fovus

Run WDL workflows on the Fovus Serverless HPC Platform using miniwdl.

miniwdl-fovus registers a miniwdl container backend (fovus_job) that submits each WDL task as a containerized Fovus job, grouping every task of a run under a single Fovus pipeline so the whole workflow appears as one pipeline on app.fovus.co with unified status / stop / resume.

It talks to Fovus by shelling out to the official fovus CLI — the same approach as Fovus's Nextflow plugin nf-fovus.

How it works

miniwdl run (cfg: container_backend = fovus_job)
        │
        ├─ one Fovus pipeline per run (fovus pipeline create)
        │
        └─ per WDL task:  FovusJob(TaskContainer)
                            ├─ build job-config JSON from runtime{} Fovus attrs
                            ├─ fovus job create <cfg> <task_dir> --pipeline-id ...
                            ├─ poll: fovus job status --job-id ...
                            └─ read rc / stdout / stderr from the shared mount

Shared-storage I/O model

Fovus storage is mounted (fovus storage mount) at one path that exists on the miniwdl host and on every Fovus compute node. The miniwdl run directory lives on that mount, so — like miniwdl-aws's EFS backend — container_dir == host_dir. There is no per-task upload, no fovus job download: a task's outputs are visible to the host the moment its Fovus job finishes. Inputs produced by upstream tasks are staged into each task's directory on the same mount (a copy within Fovus storage).

Installation

pip install miniwdl-fovus    # pulls in miniwdl-fovus-runtime and the fovus CLI
fovus auth login             # authenticate once

miniwdl-fovus depends on miniwdl-fovus-runtime, a Fovus-patched build of miniwdl (import package still WDL) that adds non-POSIX shared-storage support. Do not also install stock miniwdl in the same environment. Compatibility: miniwdl-fovus 1.0.x <-> miniwdl-fovus-runtime 1.14.2.post1.

Usage

The simplest path is the bundled wrapper, which verifies auth, mounts the shared storage, creates the pipeline, and runs miniwdl with the right config and run dir (default --dir: /fovus-storage/pipelines/<pipeline_id>/fovus-work):

miniwdl-fovus run workflow.wdl -i inputs.json --pipeline-name my-run \
    --storage-mount-point /fovus-storage

Or drive miniwdl run directly once storage is mounted and auth is set up:

fovus storage mount /fovus-storage
miniwdl run workflow.wdl -i inputs.json \
    --cfg $(python -c "import miniwdl_fovus,os;print(os.path.join(os.path.dirname(miniwdl_fovus.__file__),'fovus.cfg'))") \
    --dir /fovus-storage/miniwdl_run

The run directory (--dir) must be under the storage mount point.

Declaring Fovus resources in WDL

Tasks declare Fovus-native resources directly in runtime{}, using the Fovus job-config field names verbatim (no prefix). Anything omitted falls back to the [fovus] defaults in the config.

task align {
    command <<< ... >>>
    runtime {
        docker: "biocontainers/bwa:latest"   # task image (required)
        benchmarkingProfileName: "Default CPU"
        minvCpu: 4
        maxvCpu: 16
        minvCpuMemGiB: 8
        minGpu: 0
        storageGiB: 100
        walltimeHours: 3
        supportedCpuArchitectures: ["x86-64", "arm-64"]
        timeToCostPriorityRatio: "0.5/0.5"
        allowPreemptible: false
    }
}

Power users can bypass the builder entirely with configJson: "/path/to/job_config.json" — the file is used verbatim, with the generated runCommand injected.

Configuration

All [fovus] keys (see miniwdl_fovus/fovus.cfg) can be overridden by environment variable, e.g. MINIWDL__FOVUS__BENCHMARKING_PROFILE_NAME="Default CPU".

Key settings: storage_mount_point, poll_interval_seconds, auto_stop_pipeline_on_exit (stop the pipeline on Ctrl-C / crash), and the resource defaults applied when a task omits the corresponding runtime attribute.

Development

# The patched runtime lives in a separate repo (Fovus/miniwdl). For local dev,
# clone it under vendor/ (git-ignored) and install it before the plugin:
git clone https://github.com/Fovus/miniwdl.git vendor/miniwdl   # once
pip install ./vendor/miniwdl   # miniwdl-fovus-runtime (satisfies the runtime pin)
pip install -e ".[test]"       # the plugin
pytest                         # unit + simulated-backend integration tests

Alternatively, once miniwdl-fovus-runtime is published, pip install -e ".[test]" alone resolves the runtime from PyPI. CI and released builds always install it from the index (see the Release GitHub Actions workflow).

tests/e2e/ contains fake fovus and docker executables used to exercise the full miniwdl run integration locally without a Fovus account or real Docker.

Status / scope

v1.0.0 supports containerized Fovus jobs grouped under one pipeline per run, with the shared-storage I/O model. Out of scope for now: monolithic (license-bound) jobs, auto-provisioning the storage mount, per-task live log streaming, and miniwdl call-cache integration.

License

MIT