Clew

Clew answers questions about workflow runs that the engine cannot: what a bad input reached, what can safely be deleted, and what one run took from another. It reads the record the engine already writes, for Nextflow, Snakemake, Cromwell, Horus, DNAnexus and Latch, and changes nothing in your pipeline.

The examples are from genomics because that is where it was first used. The graph underneath is neutral: tasks that read files and write files, whatever the field.

A clew is the ball of thread Ariadne gave Theseus. You follow it back out.

Install

pip install clew-lineage
clew demo

Python 3.9 or later, no dependencies. The demo runs over a real nf-core/sarek run that ships with the package.

What it answers

Something upstream went bad. A reference update, a broken container, an input that turned out wrong.

clew impact --graph graph.json --container gatk4

Every affected task gets a verdict, re-run, quarantine, delete or disclose, with the derivation chain as evidence. A verdict that depends on whether the artifact still exists needs --work-root and --results; without them it is withheld, not guessed. Nothing unknown is reported as clean.

The disk is full and nothing is wrong.

clew reclaim --graph graph.json --work-root work/ --results results/

Proposes only the directories the graph proves redundant, and deletes nothing without --apply and a receipt. s3://bucket/prefix works as either root.

Did the new version produce what the old one did?

clew drift --before a.json --after b.json

Names the first task on each chain whose outputs differ and why: an input changed, a container changed, or nothing changed and the tool is not deterministic. Everything else is confirmed reproduced, digest for digest.

One run consumed another's output.

clew stitch --graph a=a.json --graph b=b.json --out chain.json

Joins runs by content digest, so a question follows a change across launches, machines and engines.

Add --html to impact, reclaim or drift for a one-page report.

Your runs

One command per engine turns a run into a graph.

clew extract lists every engine it knows, and clew providers shows every adapter and extractor installed with the package each came from. An engine or pipeline that is not there is one subclass away: see providers.

Or skip the file: reclaim, drift and digest take --runs pointing at the engine's own record, a .lineage store or a horus-lineage root, and read the run they need. impact takes a graph file.

Content digests make every answer exact. Horus records them, and Nextflow does with cache 'deep'. For any other run, clew digest reads each file once and fills them in. Sources has what each engine records and what that limits.

Every answer can be checked

Clew is built for the day someone else has to verify what it said. Storage is checked, never assumed. Facts go in an append-only log. The rules are versioned data, so an old plan replays under the rules that produced it. An evidence bundle re-derives every verdict offline, with no database and no credentials. A gate can block a run before it starts.

Storage, event log, policy, evidence, gate, auditor surfaces, architecture, providers for adding your own pipeline or engine.

Status

Verified on real Nextflow, Snakemake, Cromwell and Horus runs. DNAnexus and Latch are built from the platform APIs and await their first live runs. CHANGELOG.md lists what changed in each release.

Contributing

Issues and pull requests are welcome, especially from people who run workflows for a living and can say where the model is wrong. See CONTRIBUTING.md. No agreement to sign.

License

AGPL-3.0.