miniwdl-fovus 1.0.0

miniwdl container backend that runs WDL tasks as jobs on the Fovus Serverless HPC Platform, grouped under one Fovus pipeline per workflow.

miniwdl-fovus

Run WDL workflows on the Fovus Serverless HPC Platform using miniwdl.

miniwdl-fovus registers a miniwdl container backend (fovus_job) that submits each WDL task as a containerized Fovus job, grouping every task of a run under a single Fovus pipeline so the whole workflow appears as one pipeline on app.fovus.co with unified status / stop / resume.

It talks to Fovus by shelling out to the official fovus CLI — the same approach as Fovus's Nextflow plugin nf-fovus.

How it works

miniwdl run (cfg: container_backend = fovus_job)
        │
        ├─ one Fovus pipeline per run (fovus pipeline create)
        │
        └─ per WDL task:  FovusJob(TaskContainer)
                            ├─ build job-config JSON from runtime{} fovus_* attrs
                            ├─ fovus job create <cfg> <task_dir> --pipeline-id ...
                            ├─ poll: fovus job status --job-id ...
                            └─ read rc / stdout / stderr from the shared mount

Shared-storage I/O model

Fovus storage is mounted (fovus storage mount) at one path that exists on the miniwdl host and on every Fovus compute node. The miniwdl run directory lives on that mount, so — like miniwdl-aws's EFS backend — container_dir == host_dir. There is no per-task upload, no fovus job download: a task's outputs are visible to the host the moment its Fovus job finishes. Inputs produced by upstream tasks are staged into each task's directory on the same mount (a copy within Fovus storage).

Installation

pip install miniwdl-fovus    # pulls in miniwdl-fovus-runtime and the fovus CLI
fovus auth login             # authenticate once

miniwdl-fovus depends on miniwdl-fovus-runtime, a Fovus-patched build of miniwdl (import package still WDL) that adds non-POSIX shared-storage support. Do not also install stock miniwdl in the same environment. Compatibility: miniwdl-fovus 1.0.x <-> miniwdl-fovus-runtime 1.14.2.post1.

Usage

The simplest path is the bundled wrapper, which verifies auth, mounts the shared storage, creates the pipeline, and runs miniwdl with the right config and run dir:

miniwdl-fovus-submit workflow.wdl -i inputs.json --pipeline-name my-run \
    --storage-mount-point /fovus-storage

Or drive miniwdl run directly once storage is mounted and auth is set up:

fovus storage mount /fovus-storage
miniwdl run workflow.wdl -i inputs.json \
    --cfg $(python -c "import miniwdl_fovus,os;print(os.path.join(os.path.dirname(miniwdl_fovus.__file__),'fovus.cfg'))") \
    --dir /fovus-storage/miniwdl_run

The run directory (--dir) must be under the storage mount point.

Declaring Fovus resources in WDL

Tasks declare Fovus-native resources directly via fovus_* keys in runtime{}. Anything omitted falls back to the [fovus] defaults in the config.

task align {
    command <<< ... >>>
    runtime {
        docker: "biocontainers/bwa:latest"   # task image (required)
        fovus_benchmarking_profile_name: "Default CPU"
        fovus_computing_device: "cpu"          # or "gpu"
        fovus_min_vcpu: 4
        fovus_max_vcpu: 16
        fovus_min_vcpu_mem_gib: 8
        fovus_min_gpu: 0
        fovus_storage_gib: 100
        fovus_walltime_hours: 3
        fovus_supported_cpu_architectures: ["x86-64", "arm-64"]
        fovus_time_to_cost_priority_ratio: "0.5/0.5"
        fovus_allow_preemptible: false
    }
}

runtime key	Fovus job-config field
fovus_benchmarking_profile_name	constraints.jobConstraints.benchmarkingProfileName
fovus_computing_device	constraints.jobConstraints.computingDevice
fovus_supported_cpu_architectures	constraints.jobConstraints.supportedCpuArchitectures
fovus_allow_preemptible	constraints.jobConstraints.allowPreemptible
fovus_min_vcpu / fovus_max_vcpu	constraints.taskConstraints.minvCpu / maxvCpu
fovus_min_vcpu_mem_gib	constraints.taskConstraints.minvCpuMemGiB
fovus_min_gpu / fovus_max_gpu	constraints.taskConstraints.minGpu / maxGpu
fovus_storage_gib	constraints.taskConstraints.storageGiB
fovus_walltime_hours	constraints.taskConstraints.walltimeHours
fovus_time_to_cost_priority_ratio	objective.timeToCostPriorityRatio

Power users can bypass the builder entirely with fovus_config_json: "/path/to/job_config.json" — the file is used verbatim, with the generated runCommand injected.

Configuration

All [fovus] keys (see miniwdl_fovus/fovus.cfg) can be overridden by environment variable, e.g. MINIWDL__FOVUS__BENCHMARKING_PROFILE_NAME="Default CPU".

Key settings: storage_mount_point, poll_interval_seconds, auto_stop_pipeline_on_exit (stop the pipeline on Ctrl-C / crash), and the resource defaults applied when a task omits the corresponding fovus_* attribute.

Development

# The patched runtime lives in a separate repo (Fovus/miniwdl). For local dev,
# clone it under vendor/ (git-ignored) and install it before the plugin:
git clone https://github.com/Fovus/miniwdl.git vendor/miniwdl   # once
pip install ./vendor/miniwdl   # miniwdl-fovus-runtime (satisfies the runtime pin)
pip install -e ".[test]"       # the plugin
pytest                         # unit + simulated-backend integration tests

Alternatively, once miniwdl-fovus-runtime is published, pip install -e ".[test]" alone resolves the runtime from PyPI. CI and released builds always install it from the index (see the Release GitHub Actions workflow).

tests/e2e/ contains fake fovus and docker executables used to exercise the full miniwdl run integration locally without a Fovus account or real Docker.

Status / scope

v1.0.0 supports containerized Fovus jobs grouped under one pipeline per run, with the shared-storage I/O model. Out of scope for now: monolithic (license-bound) jobs, auto-provisioning the storage mount, per-task live log streaming, and miniwdl call-cache integration.

License

MIT