Clew

Clew turns the lineage your workflow engine already writes into decisions you can defend. When an input or container goes bad, it says what to do about each output it reached. When the disk fills up, it names the work directories that are safe to delete and shows why. When an incident report arrives as a sentence, it turns the sentence into a question the run can answer, or holds it for a person. You can seal any of these decisions as evidence that an auditor checks offline, with no access to your systems.

It reads the record from Nextflow, Snakemake, Cromwell, Horus, DNAnexus and Latch, and changes nothing in your pipeline. No model touches a verdict: where a model is used, it proposes, versioned rules decide, and the record says which model was asked.

The examples are from genomics because that is where it was first used. The graph underneath is neutral: tasks that read files and write files, whatever the field.

In the field's terms: Clew is data lineage and provenance turned into impact analysis and remediation planning, with policy as versioned data, an append-only hash-chained audit log, and offline-verifiable evidence bundles. Incident intake is a typed classification step, a model choosing from a fixed option set with a confidence, under deterministic decision rules and a human-in-the-loop decision on anything held. The agentic layer exposes the same steps as MCP tools, runs them under a governed agent runtime, and lets an agent generate providers that pass a code-based conformance check before a person approves them. A read-only MCP server serves the evidence to auditors with citations.

A clew is the ball of thread Ariadne gave Theseus. You follow it back out.

Install

pip install clew-lineage
clew --version
clew demo

Python 3.9 or later. The core has no dependencies. The demo runs three questions over a real nf-core/sarek run of 81 tasks that ships with the package, so you can see every kind of answer before touching your own runs. Steps 1 to 5 below need nothing more. Step 6, the agent, needs Mainsheet, which pip install "clew-lineage[agent]" brings in. clew serve works with any MCP client without it.

The flow

Every use of Clew follows the same six steps. Each step is one command, reads what the step before wrote, and leaves a file the next step reads. Nothing is held in memory between them, so any step can be rerun or checked on its own.

engine record --> graph.json --> plan.json --> bundle/ --> auditor
   extract         ask            seal         share
                   triage, decide

1. Extract: the engine's record becomes one graph

Point Clew at where your workflow was launched. The engine's own record is read, and graph.json is written: every task, every file it read and wrote, and the content digests the engine recorded.

clew extract alone lists every engine installed. Two helpers belong to this step. clew digest reads each file of a run once and fills in digests an engine did not record, which is what makes drift and reclaim exact. clew stitch --graph a=a.json --graph b=b.json --out chain.json joins runs that consumed each other's outputs, by digest, so a question follows a change across launches and machines.

reclaim, drift, digest and impact also take --runs pointing at the engine's record itself, and read the run they need without a file.

2. Ask: what did a change reach, and what should happen

Something upstream went bad. A reference update, a broken container, an input that turned out wrong.

clew impact --graph graph.json --container gatk4 --json plan.json

Every affected task gets a verdict, re-run, quarantine, delete or disclose, with the derivation chain as evidence and the policy version that produced it. A verdict that depends on whether an artifact still exists needs --work-root and --results. Without them it is withheld as UNDETERMINED, never guessed. Nothing unknown is reported as clean.

The disk is full and nothing is wrong.

clew reclaim --graph graph.json --work-root work/ --results results/

Proposes only the directories the graph proves redundant, and deletes nothing without --apply and a receipt. s3://bucket/prefix works as either root.

Did the new version produce what the old one did?

clew drift --before a.json --after b.json

Names the first task on each chain whose outputs differ and why: an input changed, a container changed, or nothing changed and the tool is not deterministic. Everything else is confirmed reproduced, digest for digest.

Add --html report.html to any of the three for a one-page report.

3. Triage: an incident report becomes a question

Most changes arrive as a sentence, not a trigger: a tool advisory, a release note, a withdrawal. Triage turns the sentence into one of the triggers this run can answer, or holds it.

clew triage --graph graph.json --json triage.json "the duplicate marking step flags optical duplicates wrongly"

The options offered are the run's own: every tool its containers name, every outside input, every label, and the kinds a pipeline adapter declares. A classifier picks one and says how sure it is. Versioned settings then decide: ask the trigger, hold the incident for a person, or dismiss it. Dismissing needs more confidence than asking, and is refused outright when the sentence names an id the run's record contains. With TYPESAFE_API_KEY set the classifier is TypeSafe's Jev; without it, names are matched. The record carries the backend, the model and the hash of the request, so the same question can be asked again.

A held incident waits for a person, who decides under their own name:

clew decide --record triage.json --ask container:gatk4 --actor "qa lead" --reason "the advisory names our version"

Asking a removal, patient:donor_003 say, asserts the withdrawal as a fact. With a log attached it is written there, under the person's name, and the gate reads it. Nobody types a withdrawal in by hand.

Exit codes say what happened: 0 asks, 1 holds, 3 dismisses. A pipeline plugin can do the asking itself with --print-request and --answer.

4. Seal: the answer becomes evidence

clew evidence seal --plan plan.json --out bundle/
clew evidence verify bundle/

A bundle holds the plan, the policy it was computed under, the triage record and decision where there was one, and the hashes of every input. verify recomputes every verdict from the bundle alone, with no database, no credentials and no access to the run. witness checks a live log against what the bundle saw, and sign countersigns it with an SSH key.

Optionally, an event log keeps what people decided and adopted, as an append-only, hash-chained table on Postgres. Nothing is typed into it by hand: triage and decide write to it when --dsn names it, clew rulebook register records the adoption of a policy version, and a bundle built with --dsn witnesses the log head it saw, so a later reader can tell whether anything was removed afterwards.

clew log verify

clew rulebook shows and diffs the versioned policy table, so an old plan replays under the rules that produced it. clew gate reads the log before a run starts and blocks a run whose inputs it says are not usable, failing closed on anything unknown.

5. Share: an auditor checks it without you

clew dashboard --bundles bundles/ --out clew.html
clew mcp --bundles bundles/

The dashboard is one self-contained HTML page over every bundle. The MCP server answers an auditor's questions in their own words, from the bundles alone, read-only, with a citation on every answer and an instruction to the model never to conclude compliance.

6. Automate: an agent runs the steps, a person keeps the decisions

The agent that runs steps 3 and 4 for you is a Mainsheet agent. Mainsheet is our agent runtime: it reads one definition file, runs the model with exactly the tools that file allows, checks every tool call against a policy before it runs, keeps it inside a sandbox with no network, and signs a record of the run. clew ui and clew build start it for you, so both need Mainsheet installed in the same environment:

pip install "clew-lineage[agent]"
clew ui

clew ui is a page on this machine: pick a run folder, write the incident, and watch triage, impact and evidence appear as their files do. A held incident shows a decision card, and the agent has no tool to decide one. The definition it runs is clew/agent/agent.yaml: the model, the prompt, the seven Clew tools, and the limits on each.

clew serve --dir /path/to/dir

clew serve needs no Mainsheet. It offers the same seven steps as tools over MCP to any agent that speaks it, Claude Code, Cursor or your own: inbox, triage, impact, seal, options, the incident text, and a recommendation on a held incident. No tool takes a trigger, which travels from triage to impact in the record on disk, and no tool decides a held incident. skills/clew-incident gives the flow to a person's own coding agent. What such an agent lacks is Mainsheet's gate and record, which is why the tools are built to be safe by what they take.

Command reference

Every command answers --help with its flags.

Providers: your pipeline, your engine

Clew knows nothing about any field. What a site knows about one pipeline, that an id in a launch sheet is a specimen and which tasks it entered, lives in an adapter. An engine Clew cannot read yet gets an extractor. Both are one Python class, found by name, with no change inside Clew. How to build your own has the three ways, by hand, with your own coding agent, or with Clew's, and the guide below it.

An agent can write one. It works in a sandbox, a conformance check it never sees runs on what it wrote, and nothing loads until a person has read the code and approved it under their name:

clew build adapter --graph graph.json --name site-ligands --kind ligand --sheet ligands.smi
clew providers --approve ~/.clew/ui/jobs/<build>/verdict.json --actor "your name"

Approved files live in ~/.clew/providers, or the folder CLEW_PROVIDER_DIR names, and every command loads them from there. The approval record pins the file's hash. A changed file stops loading until someone approves it again. clew build judge <folder> runs the check again on a finished build without the agent. skills/clew-provider gives the same brief to a person's own coding agent.

What is guaranteed

• Storage is checked, never assumed. A verdict that depends on a file existing is withheld until the file is looked at.
• Rules are versioned data. Every plan names its policy version and hash, and an old plan replays under the rules that produced it.
• A model never writes a verdict. It picks a trigger from a fixed list, or recommends on a held incident. Settings and people decide, and the record names the model asked.
• Facts are append-only. The event log is hash-chained with two clocks, and a bundle witnesses the log head it saw.
• Evidence verifies offline. A bundle re-derives every verdict with no database and no credentials.
• Agents are governed. Under Mainsheet every tool call passes a policy gate and is signed. Over MCP the tools stay safe by what they take: an id, never a trigger or code.
• The engine stays neutral. A test fails the build if the graph, the ledger or the contracts mention a sample, a donor, a consent or an engine.

Storage, event log, policy, evidence, gate, auditor surfaces, architecture, sources for what each engine records and what that limits.

Status

Verified on real Nextflow, Snakemake, Cromwell and Horus runs. DNAnexus and Latch are built from the platform APIs and await their first live runs. Triage has one eval baseline, on the release notes of sixteen projects. The adapter builder has run live once. 894 tests run on Python 3.9, 3.11 and 3.13 on every push. CHANGELOG.md lists what changed in each release.

Contributing

Issues and pull requests are welcome, especially from people who run workflows for a living and can say where the model is wrong. See CONTRIBUTING.md. No agreement to sign.

License

AGPL-3.0.