Clew
Clew answers questions about workflow runs that the engine cannot: what a bad input reached, what can safely be deleted, and what one run took from another. It reads the record the engine already writes, for Nextflow, Snakemake, Cromwell, Horus, DNAnexus and Latch, and changes nothing in your pipeline.
The examples are from genomics because that is where it was first used. The graph underneath is neutral: tasks that read files and write files, whatever the field.
A clew is the ball of thread Ariadne gave Theseus. You follow it back out.
Install
pip install clew-lineage
clew demo
Python 3.9 or later, no dependencies. The demo runs over a real nf-core/sarek run that ships with the package.
Three questions
Something upstream went bad. A reference update, a broken container, a withdrawn sample.
clew impact --graph graph.json --container gatk4
Every affected task gets a verdict, re-run, quarantine, delete or disclose, with the derivation chain as evidence. Nothing unknown is reported as clean.
The disk is full and nothing is wrong.
clew reclaim --graph graph.json --work-root work/ --results results/
Proposes only the directories the graph proves redundant, and deletes
nothing without --apply and a receipt.
One run consumed another's output.
clew stitch --graph a=a.json --graph b=b.json --out chain.json
Joins runs by content digest, so a question follows a change across launches, machines and engines.
Add --html to impact or reclaim for a one-page report.
Your runs
One command per engine turns a run into a graph.
| Engine | Command |
|---|---|
| Nextflow, including Seqera Platform | clew extract-store --store .lineage --run <run> --json-out graph.json |
| Snakemake | clew extract-snakemake --workdir . --json-out graph.json |
| Cromwell and WDL, including Terra | clew extract-cromwell --metadata metadata.json --json-out graph.json |
| Horus, through horus-lineage | clew extract-horus --run-dir <run> --json-out graph.json |
| DNAnexus | clew extract-dnanexus --analysis <id> --json-out graph.json |
| Latch | clew extract-latch --execution <id> --json-out graph.json |
Content digests make every answer exact. Horus records them, and Nextflow
does with cache 'deep'. For any other run, clew digest reads each file
once and fills them in. Sources has what each engine
records and what that limits.
Every answer can be checked
Clew is built for the day someone else has to verify what it said. Storage is checked, never assumed. Facts go in an append-only log. The rules are versioned data, so an old plan replays under the rules that produced it. An evidence bundle re-derives every verdict offline, with no database and no credentials. A gate can block a run before it starts.
Storage, event log, policy, evidence, gate, auditor surfaces, architecture.
Status
Verified on real Nextflow, Snakemake, Cromwell and Horus runs. DNAnexus and Latch are built from the platform APIs and await their first live runs. CHANGELOG.md lists what changed in each release.
Contributing
Issues and pull requests are welcome, especially from people who run workflows for a living and can say where the model is wrong. See CONTRIBUTING.md. No agreement to sign.
License
1 maintainer
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
Filter files by name, interpreter, ABI, and platform.
If you're not sure about the file name format, learn more about wheel file names.
Copy a direct link to the current filters
File details
Details for the file clew_lineage-0.3.0.tar.gz.
File metadata
- Download URL: clew_lineage-0.3.0.tar.gz
- Upload date:
- Size: 549.6 kB
- Tags: Source
- Uploaded using Trusted Publishing? Yes
- Uploaded via:
twine/7.0.0 CPython/3.13.14
File hashes
| Algorithm | Hash digest | |
|---|---|---|
| SHA256 |
7717266fb0effe5cbbac6b882cf21a0ded6cf5f821b404f06d6a83d9238f1821
|
|
| MD5 |
29497648ca508f2a08261d1702f25316
|
|
| BLAKE2b-256 |
fa839d342dae6d9456d8e443528b8ff2476420b26e7b8768f19e4cc85e782967
|
Provenance
The following attestation bundles were made for clew_lineage-0.3.0.tar.gz:
Publisher:
release.yml on QuietFlare/clew
-
Statement:
-
Statement type:
https://in-toto.io/Statement/v1 -
Predicate type:
https://docs.pypi.org/attestations/publish/v1 -
Subject name:
clew_lineage-0.3.0.tar.gz -
Subject digest:
7717266fb0effe5cbbac6b882cf21a0ded6cf5f821b404f06d6a83d9238f1821 - Sigstore transparency entry: 2742214635
- Sigstore integration time:
-
Permalink:
QuietFlare/clew@35a90a716aa98aa187bd58a9ad7ffaf02e954118 -
Branch / Tag:
refs/tags/v0.3.0 - Owner: https://github.com/QuietFlare
-
Access:
public
-
Token Issuer:
https://token.actions.githubusercontent.com -
Runner Environment:
github-hosted -
Publication workflow:
release.yml@35a90a716aa98aa187bd58a9ad7ffaf02e954118 -
Trigger Event:
push
-
Statement type:
File details
Details for the file clew_lineage-0.3.0-py3-none-any.whl.
File metadata
- Download URL: clew_lineage-0.3.0-py3-none-any.whl
- Upload date:
- Size: 517.0 kB
- Tags: Python 3
- Uploaded using Trusted Publishing? Yes
- Uploaded via:
twine/7.0.0 CPython/3.13.14
File hashes
| Algorithm | Hash digest | |
|---|---|---|
| SHA256 |
2e70f8e4298697e3c59d111d66b7858f343574205b1c790fe26a29c2826365ad
|
|
| MD5 |
bfdbd0953524fd055d26a2623eedaee7
|
|
| BLAKE2b-256 |
4c413edcda649b1df87f63410ecf9d95dddbda041cfb310cbb4a934fdc4c809d
|
Provenance
The following attestation bundles were made for clew_lineage-0.3.0-py3-none-any.whl:
Publisher:
release.yml on QuietFlare/clew
-
Statement:
-
Statement type:
https://in-toto.io/Statement/v1 -
Predicate type:
https://docs.pypi.org/attestations/publish/v1 -
Subject name:
clew_lineage-0.3.0-py3-none-any.whl -
Subject digest:
2e70f8e4298697e3c59d111d66b7858f343574205b1c790fe26a29c2826365ad - Sigstore transparency entry: 2742214664
- Sigstore integration time:
-
Permalink:
QuietFlare/clew@35a90a716aa98aa187bd58a9ad7ffaf02e954118 -
Branch / Tag:
refs/tags/v0.3.0 - Owner: https://github.com/QuietFlare
-
Access:
public
-
Token Issuer:
https://token.actions.githubusercontent.com -
Runner Environment:
github-hosted -
Publication workflow:
release.yml@35a90a716aa98aa187bd58a9ad7ffaf02e954118 -
Trigger Event:
push
-
Statement type:
