bio-pype-runtime-models

Learn per-workflow resource, timeline and power-curve models from Bio-Pype runtime data, then use them to predict a new run's cost from a single input size, to grade a finished run against its own baseline, and to suggest better default requirements.

Built to be imported into other projects (compute.bio trains and serves these models from its database). Pure numpy; models serialise to plain JSON — no pickle.

Status: alpha. The modelling choices and the JSON model format will change between releases, and estimates from few runs are rough. Retrain after upgrading rather than loading models written by an older version.

The distribution is bio-pype-runtime-models; the import name and the command are pype_runtime_models / pype-runtime-models.

Install

pip install bio-pype-runtime-models

Requires Python >= 3.9 and numpy. Bio-Pype itself is not needed: the package only reads the runtime data Bio-Pype writes.

Input data

One run is the content of a Bio-Pype pipeline_runtime.yaml: a mapping of job id to job record (command, status, timings, requirements, resource_consumption, data_sizes), plus run metadata and environment. It can come from:

• a directory of *.json files, one per run ("dumps");
• parsed Run objects, built with Run.from_dict(mapping), when the runs live somewhere else, such as a database.

What it does (mapped to the original goals)

Goal	Where
Detect which workflow a runtime object is	workflow.detect_workflow — pipeline_name, else a snippet-set signature match
One model per workflow, improving as more runs land	train.train_models rebuilds the whole registry each time; safe to schedule
Per-step resource consumption, spot deviations	StepModel per snippet; evaluate_run flags per-step anomalies
Predict timeline + resources of each step from input size	predict.predict_run
Flag steps whose memory/CPU is driven by input size	StepModel.input_sensitive / sensitivity_report
Recommend raising/lowering the default requirement	StepModel.recommend → increase / decrease / ok
Post-hoc performance eval, overall and per-step	evaluate.evaluate_run
Predict the power curve for a workflow	predict.predict_power_curve (experimental, see below)

Workflow identity & provenance

Snippets live in $PYPE_MODULES/snippets and are shared by every pipeline in a module, so the natural owner of a snippet's resource model is the module, not the individual pipeline. Identity is therefore pluggable (--identity on train, identity_mode= in the API):

• module (default) — key models by the PYPE_MODULES basename, pooling every pipeline in the module. Shared snippets like mosdepth train on all their samples at once (e.g. n=116 across the 5 ensemble_sv pipelines instead of split five ways). Resource models are pooled; each pipeline keeps its own timeline view (snippet set + start offsets), so predict <module> --pipeline X gives X's schedule against the pooled models.
• pipeline — the older one-model-per-pipeline_name behaviour.
• registry — key by a versioned namespace/name:version parsed from a registry-style module path (e.g. .../registry/sci-bioinformatics/somatic:20260701). This is the collision-proof, version-aware identity the module registry will provide; it falls back to the module basename when no registry info is present.

Every model records its provenance (module_paths, namespaces, versions, pipeline_names, profiles), so the day the registry lands, switching to version-aware keying — or sharing/federating models at the app level — is a config change, not a rewrite. predict/evaluate accept either an identity key or a pipeline_name (auto-resolved to its module).

Provenance includes module paths, which often contain user or group names. Strip it before showing a model to anyone other than the people whose runs trained it.

Caveat for module mode: two unrelated modules that happen to share a basename (your somatic vs a third party's) would be merged — the registry namespace is what ultimately disambiguates them.

Key modelling choices

• Canonical step identity is the snippet name parsed from the job command (ingest.snippet_name_from_command), not the name field — the latter carries per-sample suffixes (mosdepth_EGAD2585) that would fragment the models.
• Driver size. Each step is regressed against its own driver size: input_size_bytes, or output_size_bytes when the input is 0 (generators / aggregators). Because a caller supplies only one number at prediction time, we also learn how each step's driver scales with the run-level driver (driver_scale), so the whole chain is predictable from a single input size. The run driver is the largest input among the earliest-submitted jobs — the recorded dependencies are often partial in dumps, so we don't rely on a strict topological root.
• Per-step fit picks the best of constant / linear / power (power fit and scored in log space for numerical stability) by leave-one-out R², and only beats the constant baseline if it generalises — noisy/sparse steps stay constant instead of overfitting.
• Few samples means no size dependence. A step needs at least 5 samples with differing driver sizes before a size-dependent fit is even tried. Below that the prediction is the median of what was seen, whatever the input size: three runs of a workflow give the same expected duration for a small and a large input.
• Robust throughout: medians, MAD, and Spearman rank correlation, because runtime telemetry is heavy-tailed. Predictions are capped at 50× the largest observed value so queries far outside the training range can't blow up.

How a run is graded

evaluate_run compares each completed step with the prediction for that step's own driver size:

• A step is flagged when duration or peak memory is at least 3 robust standard deviations above the prediction, or at least 2x the prediction. Average CPU is reported for context and never flagged.
• The spread used for the z-score has a floor of 5% of the predicted value, so a step whose training runs happened to be almost identical does not flag every normal variation.
• Steps with no model are reported as unmodelled, not dropped.
• The overall performance_score (0-100) and verdict come from the duration z-scores. The verdict is unknown when no step's duration could be graded.

Requirement recommendations

StepModel.recommend compares the default requirement recorded with the jobs to what the step used: predicted peak memory with 1.3x headroom, predicted duration with 1.5x headroom. It suggests lowering a requirement that is more than 3x above the 95th percentile of use. Check a suggestion against the underlying measurements before applying it; a bad measurement in the training runs produces a bad suggestion.

GPU steps

Steps are also grouped by a vendor-neutral GPU class. Once a class has at least 3 samples the model learns a duration factor for it relative to the size-based prediction; --gpu-class / gpu_class= applies it, and omitting it predicts for the mix of cards seen in training.

Data hygiene (split runs & mis-measured inputs)

These matter because of how runs are actually dumped, and are handled generally — no per-snippet logic is hardcoded anywhere:

• Split / partial runs are kept, not discarded. A logical run is often spread across several dump files: a failed run is dropped, a resumed run completes the same file, and a relaunched run starts a fresh file missing the steps that already succeeded. Every completed step in every file is a valid per-step sample, so all of them train the per-step models. The quality report shows how many runs per workflow are partial.
• Run-level input size is only trusted when it's actually present. Each workflow's input-tier ("entry") snippets are learned from the data (the steps whose driver size is largest by a wide margin). A partial file that lacks an entry step contributes its per-step samples but no run-level driver, so it can't feed the chain-scaling model a misleading input size.
• Implausible driver sizes are dropped per step. Some inputs are folders historically mis-measured as a few bytes, or are overwritten in place with a dummy value by convert/cleanup steps. A driver is dropped only when it is both < 1 MB and > 10,000× smaller than that step's own largest driver — a general statistical test that removes the bogus cluster while leaving genuinely small-input steps (all a few KB) untouched. Steps left with no size signal fall back to a constant (median) model, which is correct for cleanup-style steps whose cost isn't driven by input size. Counts surface in the quality report and per-step n_driver_outliers.

python -m pype_runtime_models quality models/     # partial runs, dropped outliers, entry snippets

CLI

# train / retrain from a dumps directory
pype-runtime-models train dumps/ --out models/

# list what got trained
pype-runtime-models workflows models/

# predict per-step resources + timeline for a 50 GB input
pype-runtime-models predict models/ ensemble_sv_TO --input-size 50gb --full

# grade a finished run
pype-runtime-models evaluate models/ dumps/<hash>.json

# requirement changes worth making
pype-runtime-models recommend models/ ensemble_sv_TO --changes-only

# learned per-GPU-class duration factors
pype-runtime-models gpu models/ ensemble_sv_TO

# predict the power curve (30-min bins by default; experimental)
pype-runtime-models power models/ ensemble_sv_TO --input-size 50gb --bin-minutes 60

train options: --identity module|pipeline|registry (default module), --min-workflow-runs N (skip identities seen in fewer runs, default 1) and --min-step-samples N (skip steps with fewer completed jobs, default 3). python -m pype_runtime_models ... is equivalent to the pype-runtime-models command.

Library

from pype_runtime_models import (
    Run, train_models, load_models, predict_run, evaluate_run,
)

reg = train_models("dumps/", "models/")          # or load_models("models/")
pred = predict_run(reg, "ensemble_sv_TO", input_size_bytes=50 * 1024**3)
report = evaluate_run(reg, "dumps/somehash.json")

print(report["verdict"], report["performance_score"])
for step in report["flagged_steps"]:
    print(step["name"], step["flags"])

Without files, for runs held in a database or loaded from pipeline_runtime.yaml:

runs = [Run.from_dict(mapping) for mapping in runtime_mappings]
reg = train_models(runs, min_workflow_runs=3)    # nothing written to disk
report = evaluate_run(reg, runs[0])

for snippet, recs in reg.get("ensemble_sv").all_recommendations().items():
    ...

train_models only writes to disk when given a models directory. A workflow model round-trips through WorkflowModel.to_json().

Power curve (experimental)

predict_power_curve turns a predicted run into a binned power curve with a copy of Bio-Pype's power estimate that is bundled with this package.

Known issue: when Bio-Pype is importable in the same environment, the package tries to use Bio-Pype's own pipeline_power_curve instead, and with current Bio-Pype that call fails with a TypeError. Training, prediction of resources and timeline, evaluation and recommendations are not affected.

Recurrent retraining

train_models is idempotent — point it at the growing dumps directory and it rebuilds every workflow model from the full corpus, so new runs simply sharpen the estimates. Schedule it every few hours, e.g. cron:

0 */4 * * * pype-runtime-models train /path/to/dumps --out /path/to/models \
    >> /var/log/pype_models.log 2>&1

Package layout

pype_runtime_models/
  ingest.py      Run/Job parsing, snippet name parsing, driver sizes
  units.py       memory / time requirement parsing
  provenance.py  module/registry provenance + pluggable identity key
  gpu.py         vendor-neutral GPU-class key + display name guess
  workflow.py    identity detection (provenance + signature fallback)
  features.py    per-step training samples, metric list
  regression.py  numpy constant/linear/power fitting + driver cleaning
  model.py       StepModel / WorkflowModel / PipelineView / ModelRegistry
  train.py       train_models / load_models (module-level pooling)
  predict.py     predict_run / predict_power_curve (GPU band, pipeline view)
  evaluate.py    evaluate_run (per-step + overall, GPU-adjusted grading)
  power.py       power estimate (bundled copy of Bio-Pype's, see Power curve)
  cli.py         argparse entrypoint
  tests/         self-contained smoke test (no pytest needed)

The smoke test is not part of the published package. From a source checkout, with a directory of runtime dumps: python -m pype_runtime_models.tests.test_smoke dumps/

Source and issues: https://codeberg.org/bio-pype/bio-pype-runtime-models