dbtips-curate 0.4.0

Three-way merge curation for dbtips pipeline-generated dossier caches. Ships agentskills.io-compatible skills for Claude Code, OpenAI Codex, and Google Antigravity.

dbtips-curate

Three-way merge curation for dbtips pipeline-generated dossier JSON caches.

Scientists hand-curate cache files after pipeline generation (deleting noise, adding records from external sources, patching status fields). When the pipeline refreshes, those edits get wiped. dbtips-curate preserves scientist intent as an append-only git-tracked operation log and replays it onto every refresh — so the merged served file is always "fresh pipeline data + scientist intent" without manual re-curation.

Published on PyPI as dbtips-curate. Latest version: 0.3.1.

Install (end user)

# 1. Install the CLI (persistent global tool)
uv tool install --force --refresh dbtips-curate
# or:  pipx install dbtips-curate
# or:  pip install dbtips-curate  (into a venv)

# 2. Bootstrap SKILL.md files into your AI coding harness(es)
dbtips-curate dev bootstrap

dev bootstrap shows an interactive menu (Claude / Codex / Antigravity / all combinations). Non-interactive form:

dbtips-curate dev bootstrap --targets all
dbtips-curate dev bootstrap --targets claude,codex
dbtips-curate dev bootstrap --targets claude --project .    # also install to <repo>/.claude/skills/

Update path: uv tool install --force --refresh dbtips-curate picks up the latest PyPI release. --refresh is important — uv caches its resolver decisions and will happily reinstall an old version without it.

Multi-harness skill install matrix

Every supported harness reads the same agentskills.io SKILL.md format we ship — zero conversion.

Harness	Global path	Project path
Claude Code	~/.claude/skills/<name>/	<project>/.claude/skills/<name>/
OpenAI Codex	~/.agents/skills/<name>/	<project>/.agents/skills/<name>/
Google Antigravity	~/.gemini/config/skills/<name>/	<project>/.agents/skills/<name>/ (shared with Codex)

Sections registered (11 out of the box)

All sections have entity_scope, identity (single or compound), and where appropriate an editable_fields allowlist for op patch. Adding a new one is usually a config edit only.

Endpoint	Scope	Identity	Notes
/evidence/target-literature/	target_disease	[PMID]	Literature refs; noise filters + abstract modal
/evidence/target-pathway/	target	[image_url]	Pathway figure diagrams
/evidence/literature-supplementary-materials-analysis/	target_disease	[pmcid]	LLM-analyzed PMC supp materials
/evidence/target-literature-images/	target_disease	[image_url]	Disease-specific pathway figures; nested array_path = "results.pathways.{disease}"
/target-assessment/antibodies/	target	[Antibody, Provider]	Compound key (same catalog # from multiple vendors)
/target-assessment/assayability/	target	[ChEMBL ID]	ChEMBL assays
/target-assessment/orthologs/	target	[speciesId, homologue]	Cross-species homology; compound key handles paralogs
/target-assessment/geneEssentialityMap/	target	[tissueName]	DepMap tissue-essentiality map
/target-profile/ontology/	target	[GO ID]	GO terms attributed to target
/market-intelligence/target-pipeline/	target	[nct_id, Drug, Disease]	Compound triple; handles basket trials (same NCT+Drug, different indications) and preclinical rows (empty nct_id)

Adding a new section? See docs/adding-sections.md. It has the data constraints to check, a copy-paste identity-uniqueness validator, and the config-vs-code decision tree.

Two entity-scope folders

• target_disease/<target>-<disease>.json — per-dossier data for a target × disease pair (e.g. wars2-obesity.json). Entity id includes the disease suffix.
• target/<target>.json — per-target data (e.g. acvr2a.json). Entity id is just the target symbol.

The CLI auto-picks --type from the section's entity_scope at runtime, so you rarely need to pass --type explicitly.

How it works (the model in one paragraph)

Every tracked entity has a curation/<entity-type>/<id>/ folder containing: immutable raw snapshots of the section (Base for 3-way merge), an append-only ops.jsonl (scientist intent), state.json (mode + pointers), and per-refresh changelogs. Each scientist action is one git commit, so git blame works on every clinical decision. On every pipeline refresh, entity ingest takes a fresh snapshot (= Theirs), runs a 3-way merge (Base + Theirs + ops → Merged), and splices the merged section back into the served file. Scientist edits survive across refreshes; pipeline updates land on top.

One-time setup (per platform repo)

From the platform repo root:

dbtips-curate dev init \
    --path backend/res-immunology-automation/res_immunology_automation/src/scripts/curation

Drops .dbtips-curate.toml next to cached_data_json/, pre-registered with all 11 curated sections. Commit the toml to git so every scientist gets identical config.

Commands at a glance

dbtips-curate
  entity      list | status | track | snapshot | diff | ingest | records | compare | finalize
  op          delete | add | patch | undo
  history     ops | report
  dev         init | bootstrap | install-skill

Every command supports --json for machine-readable output, exits 0 / 1 / 2 for success / runtime error / usage error, and follows the contract:

• Success: {"ok": true, "data": {...}}
• Error: {"ok": false, "error": {"code": "...", "message": "...", "hint": "..."}}

entity — lifecycle

Command	Purpose
entity list [--type T]	All tracked entities of one type with mode + last refresh.
entity track <id> [--section S] [--skip-history] [--max-history-commits N]	Bootstrap. Walks git history of the served file with git log --follow --full-history --reverse → synthesizes delete ops for removed records; flags historical additions for review. Uses --full-history to survive non-linear merge topologies (fixed in 0.3.1). --skip-history bypasses the walk.
entity snapshot <id> [--section S] [--reason ...]	Capture the current served file's section as an immutable raw snapshot. First snapshot becomes Base for 3-way merge.
entity status <id> [--section S]	Prioritized worklist: conflicts → historical_additions → bootstrap_candidates → resurrected → new from pipeline. Includes suggested_filters for greenfield noise triage where the section has them defined.
entity records <id> [--section S] [--page N --page-size 20] [--full] [--fields F1,F2]	List current records, paginated. Skills render as widget artifacts.
entity diff <id> [--section S] [--emit-widget FILE]	Read-only preview of the 3-way merge vs the current served file. Writes nothing. With --emit-widget, generates a self-contained HTML artifact with checkbox selection, per-row View + Edit modal, and Apply deletions batch button.
entity compare <id> [--section S] [--page N --page-size N] [--fields ...]	Side-by-side rows: previous (Base snapshot) vs current (served file).
entity ingest <id> [--section S]	Take fresh snapshot → 3-way merge → splice merged section back into served file. Writes changelog + conflicts file. This is what the pipeline hook calls.
entity finalize <id> [--section S] [--no-push]	Mark review complete: advance last_reviewed_raw, clear bootstrap_candidates, prune old snapshots, git commit + optional push.

op — scientist actions

Every op is a per-op git commit with full audit trail (SHA, author, reason). All support --dry-run.

Command	Purpose
op delete <id> --section S --key K=V [--key K2=V2 ...] --reason "..."	Tombstone a record. Compound-key sections pass one --key flag per identity field (e.g. --key "Antibody=ab001" --key "Provider=Abcam").
op add <id> --section S --record-file F --source "..."	Add a record from an external source. Validates against the section's required_fields. Use - to read the record body from stdin.
op patch <id> --section S --key K=V [--key K2=V2 ...] --set field=value [--set ...] --reason "..."	Change one or more fields on an existing record. Values are JSON-decoded when possible (Phase=3 → int, Approved=true → bool, gene_symbols=["A","B"] → list). Identity fields are rejected (PATCH_TOUCHES_IDENTITY) — use delete + re-add if you truly need to re-key.
op undo <id> --undoes <op_id> --reason "..."	Cancel a prior op. Endpoint is inferred from the target op — no --section flag. Rejects unknown op_ids, undo-of-undo, and double-undo with structured errors. The original op stays in the log for audit.

history — read-only views

Command	Purpose
history ops <id> [--since DATE] [--actor EMAIL]	The full operation log with optional filters.
history report <id> [--output FILE] [--open]	Standalone HTML report (all queues + ops table). --open launches it in your default browser.

dev — setup + maintenance

Command	Purpose
dev init --path DIR	Create the curation folder + write a default .dbtips-curate.toml.
dev bootstrap [--targets claude,codex,antigravity|all] [--project DIR]	Install every skill into the chosen AI coding harnesses. Interactive menu if --targets omitted. --project . also drops skills into the current repo's workspace-scope skill folder. Idempotent.
dev install-skill [--target claude|codex|antigravity] [--home DIR] [--only NAME]	Single-target install primitive used by bootstrap. Prefer bootstrap for end-user setup.

Engine features

Compound identity

Sections with multiple identity fields pass one --key per field. Examples:

• Antibodies — --key "Antibody=ab001" --key "Provider=Abcam" (same catalog number can be resold by multiple vendors).
• Orthologs — --key "speciesId=9606" --key "homologue=ACVR2B" (paralogs share species id).
• Target-pipeline — --key "nct_id=NCT00752557" --key "Drug=DIBOTERMIN ALFA" --key "Disease=bone disease" (basket trials share NCT+Drug across indications; preclinical rows have empty nct_id).

Dotted + templated array_path

The array_path field on each section is walked as a dotted path and supports a {disease} template that's substituted from the entity_id at runtime. Example (target-literature-images, a nested-container section):

[sections."/evidence/target-literature-images/"]
entity_scope = "target_disease"
array_path = "results.pathways.{disease}"     # walks results → pathways → obesity for wars2-obesity
identity = ["image_url"]

Single-key array_path values still work unchanged (backward compatible).

editable_fields allowlist

Each section can declare which fields op patch and the widget's Edit modal expose. Identity fields are always excluded. Example (target-pipeline):

editable_fields = [
    "Phase", "Status", "Sponsor", "ApprovalStatus",
    "Mechanism of Action", "Disease", "Modality",
    "WhyStopped", "OutcomeStatus", "intervention_types",
]

Brownfield detection via git history

entity track walks the served file's git history to detect prior scientist edits (from Strapi, hand edits, other tools) and synthesizes matching ops so those changes survive the next refresh. As of 0.3.1 the walker uses git log --follow --full-history --reverse, which disables git's default merge-based path-history simplification — the simplification silently drops commits at merge points and previously caused record changes to be attributed to the wrong commit in non-linear repo histories.

Mode state machine

Mode	Set by	Meaning
greenfield	track with no git-history record changes	Fresh pipeline output, no prior scientist edits detected.
brownfield-from-git	track when git history shows record-level adds/removes	Prior scientist edits detected via the git-history walk; synthesized delete ops + historical_additions queue surfaced.
brownfield-detected	ingest auto-promotion when pipeline drops Base records that aren't tombstoned	Records preserved as bootstrap_candidates pending scientist confirmation.
brownfield-bootstrap	Legacy label; behaves identically to brownfield-detected	Backward-compatible.
active	finalize after first review session	Steady-state operation.

Filesystem layout

backend/.../scripts/
  cached_data_json/                             (pipeline writes here, served to clients)
    target/<target>.json                        (target-scope dossiers)
    target_disease/<target>-<disease>.json      (target × disease dossiers)
  curation/                                     (this tool writes here, tracked in git)
    .dbtips-curate.toml                         (per-team config + section schema registry)
    target/<entity-id>/                         (or target_disease/<entity-id>/)
      raw/<ts>.json                             (immutable section snapshots)
      ops.jsonl                                 (append-only scientist operations)
      state.json                                (mode + last_reviewed_raw + git-history summary)
      conflicts/<ts>.json                       (open conflicts after last merge)
      changelog/<ts>.md                         (human-readable per-refresh summary)
      merged.json                               (post-merge section, spliced back into served file)
      reports/<ts>.html                         (HTML reports — created on demand)

Three-way merge semantics

Each entity ingest runs:

Base    = snapshot at state.last_reviewed_raw     (what scientist last accepted)
Theirs  = fresh snapshot of current served file   (what pipeline just produced)
Ours    = apply_ops(Base, ops.jsonl)              (scientist intent layered on Base)

Merged  = three_way_merge(Base, Theirs, Ours)

Per identity-keyed record:

In Base	In Theirs	Has op	Result
✓	✗	tombstone	silent (both agree to remove)
✓	✗	none	removed_by_pipeline queue — in brownfield modes, preserved as bootstrap candidate
✗	✓	none	new_from_pipeline queue — added to merged
✗	✓	tombstone	dropped from merged, logged as resurrected_tombstone
✓	✓	patch	per-field merge; pipeline-changed + scientist-patched same field → conflict (scientist wins by default)
✗	✗	add	scientist-added record, included in merged

Ops are cancelled by op undo; the two-pass state computation in the merge engine treats cancelled ops as if they never applied, without dropping later ops on the same record/field.

Skills catalog (11 SKILL.md files)

The dev bootstrap command installs all of these into your chosen harness(es). Each is agentskills.io standard — a folder with a SKILL.md, YAML frontmatter, markdown body.

Skill	Section it drives
dbtips-curate	Umbrella router — resolves entity id + section, hands off. Owns CLI mechanics + universal ops (undo, finalize, widget batches).
dbtips-curate-target-literature	/evidence/target-literature/
dbtips-curate-antibodies	/target-assessment/antibodies/
dbtips-curate-target-pathway	/evidence/target-pathway/
dbtips-curate-assayability	/target-assessment/assayability/
dbtips-curate-target-pipeline	/market-intelligence/target-pipeline/
dbtips-curate-ontology	/target-profile/ontology/
dbtips-curate-gene-essentiality	/target-assessment/geneEssentialityMap/
dbtips-curate-orthologs	/target-assessment/orthologs/
dbtips-curate-supplementary-materials	/evidence/literature-supplementary-materials-analysis/
dbtips-curate-literature-images	/evidence/target-literature-images/ (results.pathways.<disease> sub-array only)

Pipeline integration (build_dossier.py)

Already wired via a CURATION HOOK block at the top of build_dossier.py. Two knobs:

IS_REFRESH: bool = False        # True = clear section + redis before pipeline regen
CURATE_ENABLED: bool = True     # False = bypass the hook entirely (rollback)

With IS_REFRESH = True, the hook (in order):

1. Snapshots the current served file's section (pre-refresh Base)
2. Clears the section key from served JSON + redis
3. Lets the pipeline regenerate from scratch
4. Calls entity ingest → 3-way merge → splices merged back into served file

With IS_REFRESH = False (the safe default), only step 4 runs — keeps curation state in sync without forcing regeneration.

Driving from an AI coding harness

After dev bootstrap, in a session (Claude Code Desktop, Codex, or Antigravity):

QC ACVR2A target for antibodies
curate WARS2 for obesity literature images
review NPY1R pipeline

The umbrella skill:

1. Parses the target + section from what you said.
2. Asks explicitly if the section is missing — never guesses.
3. Resolves entity_id (<target> for target-scoped sections, <target>-<disease> for target × disease) and hands off to the matching section skill.
4. The section skill runs entity status --json, presents the worklist as an inline widget artifact, walks decisions with per-row View/Edit modals, and executes ops via the widget's Apply batch button.
5. On session end, calls entity finalize and summarizes.

Harness compatibility (tested)

Harness	Widget rendering	Notes
Claude Code Desktop	Inline, native artifact — first try	Recommended. Full support.
OpenAI Codex (tested with local Qwen on DGX)	Widget HTML redirected to browser (not inline). Occasionally requires up to 3 iterations before Codex opens the artifact correctly.	Works with friction.
Google Antigravity	Falls back to the Markdown renderer — no widget component. Data integrity is inconsistent.	Usable in limited cases.

Requires a Claude Pro subscription for Claude Code Desktop. This release ships SKILL.md files only — no MCP server — so users can't converse with it in the browser at claude.ai.

Testing

poetry install --with dev && poetry run pytest      # 76 tests

Test coverage:

• Three-way merge engine (base, brownfield, resurrected, conflicts)
• CLI end-to-end (track → snapshot → ingest → op delete → refresh → ingest → finalize)
• op patch compound-identity + JSON-typed values + basket-trial isolation
• op undo (delete/add/patch, undo-of-undo rejection, already-undone rejection)
• Git-history walk (linear + merge-with-side-branch topologies)
• Dotted + templated array_path for nested containers
• --help smoke tests on every subgroup (guards against typer/click ABI breaks)

Version pinning notes

• typer >=0.15.0,<1.0.0 — 0.12.x misparses options against click 8.4+.
• click >=8.1,<8.2 — 8.2 changed Parameter.make_metavar() to require ctx, which breaks help rendering in every typer version we've tested.
• Both pins are baked into the published wheel; uv tool install dbtips-curate gets the working combo automatically.

Status

• Production-ready for 11 sections across target and target × disease dossiers.
• 76/76 tests passing.
• Pipeline hook in build_dossier.py gated behind IS_REFRESH / CURATE_ENABLED flags.
• Skills published, Claude Code Desktop verified end-to-end.

Roadmap

• Field-level brownfield detection — extend the git-history walker to also detect same-key field-level changes (currently identity-set only).
• MCP server — a mcp serve subcommand so the same skills can drive browser-based Claude conversations at claude.ai.
• Singleton-dict sections — /target-assessment/targetability/, /target-assessment/tractability/, and similar shape-mismatched endpoints. Design note lives at src/dbtips_curate/skill/_deferred/targetability-design-note.md.
• /genomics/evidence-heatmap/ — wildcard array_path support for sections keyed by target symbol.
• op accept — CLI form for resolving conflicts by choice. Merge engine already handles the semantic; only the CLI wrapper is missing.