dbtips-curate 0.5.0

Three-way merge curation for dbtips pipeline-generated dossier caches. Ships agentskills.io-compatible skills for Claude Code, OpenAI Codex, and Google Antigravity.

dbtips-curate

Three-way merge curation for dbtips pipeline-generated dossier JSON caches.

Scientists hand-curate cache files after pipeline generation (deleting noise, adding records from external sources, patching status fields). When the pipeline refreshes, those edits get wiped. dbtips-curate preserves scientist intent as an append-only git-tracked operation log and replays it onto every refresh — so the merged served file is always "fresh pipeline data + scientist intent" without manual re-curation.

Published on PyPI as dbtips-curate. Latest version: 0.4.0.

What's new in 0.4.0

• Multi-section per entity (breaking change to on-disk layout). One target or target × disease can now be tracked across many sections simultaneously. state.json v2 shape holds a {sections: {endpoint: {...}}} dict rather than a single section. Snapshots and conflicts are per-section.
• New commit strategy: op delete / op add / op patch / op undo no longer create per-op git commits. entity ingest commits everything from the cycle as one atomic commit with a summary message. Keeps git history proportional to review cycles.
• New command entity progress <target>: aggregates per-section state across scopes and returns reviewed / in_progress / pending buckets. Drives guided multi-section QC flows from AI coding harnesses.
• New command dev migrate: one-shot v1→v2 on-disk migration for existing curation folders. --dry-run previews. Backups originals to .backup-v1/.
• merged.json deleted — was write-only, redundant with the served file after splice.
• changelog/ folder collapsed to single changelog.md at entity root, appended per ingest with ## <endpoint> · <ts> headers.
• New section registered: /evidence/search-patent/ (target × disease scope, identity=[publication_number], editable fields for assignee / grant/expiry/priority dates). Includes two noise filters (expired_patent, no_grant_date).
• New skill: dbtips-curate-patents for the search-patent section, and a "Multi-section session flow" paragraph added to the umbrella skill.

Migration for existing users:

uv tool install --force --refresh dbtips-curate
dbtips-curate dev migrate --dry-run              # preview
dbtips-curate dev migrate                        # apply

Install (end user)

# 1. Install the CLI (persistent global tool)
uv tool install --force --refresh dbtips-curate
# or:  pipx install dbtips-curate
# or:  pip install dbtips-curate  (into a venv)

# 2. Bootstrap SKILL.md files into your AI coding harness(es)
dbtips-curate dev bootstrap

dev bootstrap shows an interactive menu (Claude / Codex / Antigravity / all combinations). Non-interactive form:

dbtips-curate dev bootstrap --targets all
dbtips-curate dev bootstrap --targets claude,codex
dbtips-curate dev bootstrap --targets claude --project .    # also install to <repo>/.claude/skills/

Update path: uv tool install --force --refresh dbtips-curate picks up the latest PyPI release. --refresh is important — uv caches its resolver decisions and will happily reinstall an old version without it.

Multi-harness skill install matrix

Every supported harness reads the same agentskills.io SKILL.md format we ship — zero conversion.

Harness	Global path	Project path
Claude Code	~/.claude/skills/<name>/	<project>/.claude/skills/<name>/
OpenAI Codex	~/.agents/skills/<name>/	<project>/.agents/skills/<name>/
Google Antigravity	~/.gemini/config/skills/<name>/	<project>/.agents/skills/<name>/ (shared with Codex)

Sections registered (12 out of the box)

All sections have entity_scope, identity (single or compound), and where appropriate an editable_fields allowlist for op patch.

Endpoint	Scope	Identity	Notes
/evidence/target-literature/	target_disease	[PMID]	Literature refs; noise filters + abstract modal
/evidence/target-pathway/	target	[image_url]	Pathway figure diagrams
/evidence/literature-supplementary-materials-analysis/	target_disease	[pmcid]	LLM-analyzed PMC supp materials
/evidence/target-literature-images/	target_disease	[image_url]	Disease-specific pathway figures; nested array_path = "results.pathways.{disease}"
/evidence/search-patent/	target_disease	[publication_number]	Patent landscape; noise filters for expired and no-grant patents
/target-assessment/antibodies/	target	[Antibody, Provider]	Compound key (same catalog # from multiple vendors)
/target-assessment/assayability/	target	[ChEMBL ID]	ChEMBL assays
/target-assessment/orthologs/	target	[speciesId, homologue]	Cross-species homology; compound key handles paralogs
/target-assessment/geneEssentialityMap/	target	[tissueName]	DepMap tissue-essentiality map
/target-profile/ontology/	target	[GO ID]	GO terms attributed to target
/market-intelligence/target-pipeline/	target	[nct_id, Drug, Disease]	Compound triple; handles basket trials and preclinical rows

Two entity-scope folders on disk:

• target_disease/<target>-<disease>.json — target × disease dossiers
• target/<target>.json — target-only dossiers

The CLI auto-picks --type from the section's entity_scope at runtime, so you rarely need to pass --type yourself.

Adding a new section? See docs/adding-sections.md. It has the data constraints to check, a copy-paste identity-uniqueness validator, and the config-vs-code decision tree.

How it works (the model in one paragraph)

Every tracked entity has a curation/<entity-type>/<id>/ folder. Inside it, state.json holds a per-section cursor (mode + Base pointer + queues awaiting decision). ops.jsonl is an append-only log of scientist decisions across all sections. raw/ holds section-slugged snapshots (Base + latest per section). conflicts/<section-slug>.json holds the latest conflicts per section. changelog.md is a single human-readable log appended per ingest with ## <endpoint> · <ts> headers. Each entity ingest cycle takes a fresh snapshot (= Theirs), runs a 3-way merge (Base + Theirs + ops → Merged), splices the merged section back into the served file, then commits everything in one atomic git commit summarizing the ops applied.

One-time setup (per platform repo)

From the platform repo root:

dbtips-curate dev init \
    --path backend/res-immunology-automation/res_immunology_automation/src/scripts/curation

Drops .dbtips-curate.toml next to cached_data_json/, pre-registered with all 12 curated sections. Commit the toml to git so every scientist gets identical config.

Commands at a glance

dbtips-curate
  entity      list | progress | status | track | snapshot | diff | ingest | records | compare | finalize
  op          delete | add | patch | undo
  history     ops | report
  dev         init | migrate | bootstrap | install-skill

Every command supports --json for machine-readable output, exits 0 / 1 / 2 for success / runtime error / usage error, and follows the contract:

• Success: {"ok": true, "data": {...}}
• Error: {"ok": false, "error": {"code": "...", "message": "...", "hint": "..."}}

entity — lifecycle

Command	Purpose
entity list [--type T]	All tracked entities of one type. Now shows every section tracked per entity (multi-section-aware).
entity progress <target>	NEW. Per-target QC progress across scopes: reviewed (mode=active), in_progress (has ops but not finalized), pending (registered but never tracked). Drives guided multi-section flows from AI harnesses.
entity track <id> [--section S] [--skip-history] [--max-history-commits N]	Bootstrap. Walks git history of the served file with git log --follow --full-history --reverse. Synthesizes delete ops for removed records; flags historical additions for review. --skip-history bypasses the walk. Multi-section-aware: tracking a new section on an already-tracked entity is now supported.
entity snapshot <id> [--section S] [--reason ...]	Capture the current served file's section as an immutable raw snapshot.
entity status <id> [--section S]	Prioritized worklist: conflicts → historical_additions → bootstrap_candidates → resurrected → new from pipeline.
entity records <id> [--section S] [--page N --page-size 20] [--full] [--fields F1,F2]	List current records, paginated. Widget-friendly.
entity diff <id> [--section S] [--emit-widget FILE]	Read-only preview of the 3-way merge. Writes nothing. With --emit-widget, generates a self-contained HTML artifact.
entity compare <id> [--section S] [--page N --page-size N] [--fields ...]	Side-by-side rows: previous (Base snapshot) vs current (served file).
entity ingest <id> [--section S]	Take fresh snapshot → 3-way merge → splice merged section back into served file. In 0.4.0, this is the ONLY command that creates a git commit — one atomic commit per ingest cycle summarizing all ops applied.
entity finalize <id> [--section S] [--no-push]	Mark review complete: advance last_reviewed_raw, clear bootstrap_candidates, prune old snapshots, git commit + optional push.

op — scientist actions

Every op is an append-only line in ops.jsonl. In 0.4.0, ops no longer commit per op — they accumulate as staged filesystem changes and are committed together by the next entity ingest. All support --dry-run.

Command	Purpose
op delete <id> --section S --key K=V [--key K2=V2 ...] --reason "..."	Tombstone a record. Compound-key sections pass one --key flag per identity field.
op add <id> --section S --record-file F --source "..."	Add a record from an external source. Validates against the section's required_fields.
op patch <id> --section S --key K=V [--key K2=V2 ...] --set field=value [--set ...] --reason "..."	Change one or more fields on an existing record. Values are JSON-decoded when possible. Identity fields are rejected (PATCH_TOUCHES_IDENTITY).
op undo <id> --undoes <op_id> --reason "..."	Cancel a prior op. Endpoint is inferred from the target op — no --section flag.

history — read-only views

Command	Purpose
history ops <id> [--since DATE] [--actor EMAIL]	The full operation log with optional filters.
history report <id> [--output FILE] [--open]	Standalone HTML report (all queues + ops table). --open launches it in your browser.

dev — setup + maintenance

Command	Purpose
dev init --path DIR	Create the curation folder + write a default .dbtips-curate.toml.
dev migrate [--path DIR] [--dry-run]	NEW. One-shot v1→v2 on-disk migration. Rewrites state.json to nested shape, renames snapshots/conflicts with section slugs, concatenates changelogs into changelog.md, deletes merged.json. Backs up originals to .backup-v1/. Idempotent.
dev bootstrap [--targets claude,codex,antigravity|all] [--project DIR]	Install every skill into the chosen AI coding harnesses. Interactive menu if --targets omitted.
dev install-skill [--target claude|codex|antigravity] [--home DIR] [--only NAME]	Single-target install primitive used by bootstrap.

Engine features

Compound identity

Sections with multiple identity fields pass one --key per field. Examples:

• Antibodies — --key "Antibody=ab001" --key "Provider=Abcam" (same catalog number can be resold by multiple vendors).
• Orthologs — --key "speciesId=9606" --key "homologue=ACVR2B" (paralogs share species id).
• Target-pipeline — --key "nct_id=NCT00752557" --key "Drug=DIBOTERMIN ALFA" --key "Disease=bone disease" (basket trials share NCT+Drug across indications; preclinical rows have empty nct_id).

Dotted + templated array_path

The array_path field on each section is walked as a dotted path and supports a {disease} template that's substituted from the entity_id at runtime. Example (target-literature-images, a nested-container section):

[sections."/evidence/target-literature-images/"]
entity_scope = "target_disease"
array_path = "results.pathways.{disease}"     # walks results → pathways → obesity for wars2-obesity
identity = ["image_url"]

Single-key array_path values still work unchanged (backward compatible).

editable_fields allowlist

Each section can declare which fields op patch and the widget's Edit modal expose. Identity fields are always excluded. Example (target-pipeline):

editable_fields = [
    "Phase", "Status", "Sponsor", "ApprovalStatus",
    "Mechanism of Action", "Disease", "Modality",
    "WhyStopped", "OutcomeStatus", "intervention_types",
]

Brownfield detection via git history

entity track walks the served file's git history to detect prior scientist edits (from Strapi, hand edits, other tools) and synthesizes matching ops so those changes survive the next refresh. As of 0.3.1 the walker uses git log --follow --full-history --reverse, which disables git's default merge-based path-history simplification — the simplification silently drops commits at merge points and previously caused record changes to be attributed to the wrong commit in non-linear repo histories.

Per-ingest commit strategy (v0.4.0)

Prior to 0.4.0, every scientist op (delete / add / patch / undo) created its own git commit. In a real review session with dozens of decisions, that polluted history quickly. As of 0.4.0:

• Ops append to ops.jsonl but do NOT commit. The line is staged as an uncommitted change.
• entity ingest commits everything as one atomic commit — ops.jsonl + section-slugged snapshots + changelog.md + conflicts/<slug>.json + updated state.json — with a summary message: curate(<entity>): ingest <endpoint> @ <ts> · delete=X, patch=Y, add=Z, undo=W.
• Git history stays proportional to review cycles, not to individual clicks.
• Each op still has its own line in ops.jsonl for granular audit; only the git commit is consolidated.
• Undo semantics are unchanged — you undo by op_id from ops.jsonl, regardless of when the underlying op was committed.

Mode state machine

Mode	Set by	Meaning
greenfield	track with no git-history record changes	Fresh pipeline output, no prior scientist edits detected.
brownfield-from-git	track when git history shows record-level adds/removes	Prior scientist edits detected via the git-history walk.
brownfield-detected	ingest auto-promotion when pipeline drops Base records that aren't tombstoned	Records preserved as bootstrap_candidates pending scientist confirmation.
brownfield-bootstrap	Legacy label; behaves identically to brownfield-detected	Backward-compatible.
active	finalize after first review session	Steady-state operation.

Filesystem layout (v0.4.0)

backend/.../scripts/
  cached_data_json/                             (pipeline writes here)
    target/<target>.json                        (target-only dossiers)
    target_disease/<target>-<disease>.json      (target × disease dossiers)

  curation/                                     (this tool writes here, tracked in git)
    .dbtips-curate.toml                         (per-team config + section schema registry)
    target/<entity-id>/                         (or target_disease/<entity-id>/)
      state.json                                (nested per-section cursor state — v2 shape)
      ops.jsonl                                 (mixed sections, endpoint field per line)
      changelog.md                              (mixed sections, appended per ingest)
      raw/
        <section-slug>__<ts>.json               (Base + latest per section)
      conflicts/
        <section-slug>.json                     (latest only per section, overwritten)

state.json v2 shape

{
  "schema_version": 2,
  "entity_id": "acvr2a",
  "entity_type": "target",
  "sections": {
    "/target-profile/ontology/": {
      "mode": "active",
      "tracked_at": "...",
      "last_reviewed_raw": "...",
      "bootstrap_candidates": [],
      "historical_additions": [],
      "git_history_summary": {...},
      "last_committed_op_id": "..."
    },
    "/market-intelligence/target-pipeline/": {...}
  }
}

Three-way merge semantics

Each entity ingest runs:

Base    = snapshot at state.sections[endpoint].last_reviewed_raw   (what scientist last accepted)
Theirs  = fresh snapshot of current served file                    (what pipeline just produced)
Ours    = apply_ops(Base, ops.jsonl)                               (scientist intent layered on Base)

Merged  = three_way_merge(Base, Theirs, Ours)

Per identity-keyed record:

In Base	In Theirs	Has op	Result
✓	✗	tombstone	silent (both agree to remove)
✓	✗	none	removed_by_pipeline queue — in brownfield modes, preserved as bootstrap candidate
✗	✓	none	new_from_pipeline queue — added to merged
✗	✓	tombstone	dropped from merged, logged as resurrected_tombstone
✓	✓	patch	per-field merge; pipeline-changed + scientist-patched same field → conflict (scientist wins by default)
✗	✗	add	scientist-added record, included in merged

Ops are cancelled by op undo; the two-pass state computation in the merge engine treats cancelled ops as if they never applied, without dropping later ops on the same record/field.

Skills catalog (12 SKILL.md files)

The dev bootstrap command installs all of these into your chosen harness(es). Each is agentskills.io standard — a folder with a SKILL.md, YAML frontmatter, markdown body.

Skill	Section it drives
dbtips-curate	Umbrella router — resolves entity id + section, hands off. Owns CLI mechanics + universal ops (undo, finalize, widget batches). Contains the multi-section session flow paragraph that calls entity progress.
dbtips-curate-target-literature	/evidence/target-literature/
dbtips-curate-antibodies	/target-assessment/antibodies/
dbtips-curate-target-pathway	/evidence/target-pathway/
dbtips-curate-assayability	/target-assessment/assayability/
dbtips-curate-target-pipeline	/market-intelligence/target-pipeline/
dbtips-curate-ontology	/target-profile/ontology/
dbtips-curate-gene-essentiality	/target-assessment/geneEssentialityMap/
dbtips-curate-orthologs	/target-assessment/orthologs/
dbtips-curate-supplementary-materials	/evidence/literature-supplementary-materials-analysis/
dbtips-curate-literature-images	/evidence/target-literature-images/ (results.pathways.<disease> sub-array only)
dbtips-curate-patents	/evidence/search-patent/

Pipeline integration (build_dossier.py)

Already wired via a CURATION HOOK block at the top of build_dossier.py. Two knobs:

IS_REFRESH: bool = False        # True = clear section + redis before pipeline regen
CURATE_ENABLED: bool = True     # False = bypass the hook entirely (rollback)

With IS_REFRESH = True, the hook (in order):

1. Snapshots the current served file's section (pre-refresh Base)
2. Clears the section key from served JSON + redis
3. Lets the pipeline regenerate from scratch
4. Calls entity ingest → 3-way merge → splices merged back into served file

With IS_REFRESH = False (the safe default), only step 4 runs.

Driving from an AI coding harness

After dev bootstrap, in a session (Claude Code Desktop, Codex, or Antigravity):

QC ACVR2A target for antibodies
curate WARS2 for obesity literature images
review NPY1R pipeline
QC ACVR2A   # (multi-section session — asks entity progress, walks pending sections)

The umbrella skill:

1. Parses the target + section from what you said.
2. Asks explicitly if the section is missing — never guesses.
3. For a full-dossier ask, runs entity progress <target> and presents three grouped tables (reviewed / in progress / pending), then asks which section to do next.
4. Resolves entity_id and hands off to the matching section skill.
5. The section skill runs entity status --json, presents an inline widget, walks decisions with per-row View/Edit modals, and executes ops via the widget's Apply batch button.
6. When the section skill returns control, the umbrella re-runs entity progress and repeats step 3.

Harness compatibility (tested)

Harness	Widget rendering	Notes
Claude Code Desktop	Inline, native artifact — first try	Recommended. Full support.
OpenAI Codex (tested with local Qwen on DGX)	Widget HTML redirected to browser (not inline). Occasionally requires up to 3 iterations before Codex opens the artifact correctly.	Works with friction.
Google Antigravity	Falls back to the Markdown renderer — no widget component. Data integrity is inconsistent.	Usable in limited cases.

Requires a Claude Pro subscription for Claude Code Desktop. This release ships SKILL.md files only — no MCP server — so users can't converse with it in the browser at claude.ai.

Testing

poetry install --with dev && poetry run pytest      # 87 tests

Test coverage:

• Three-way merge engine (base, brownfield, resurrected, conflicts)
• CLI end-to-end (track → snapshot → ingest → op delete → refresh → ingest → finalize)
• op patch compound-identity + JSON-typed values + basket-trial isolation
• op undo (delete/add/patch, undo-of-undo rejection, already-undone rejection)
• Git-history walk (linear + merge-with-side-branch topologies)
• Dotted + templated array_path for nested containers
• v0.4.0 features: multi-section per entity, per-ingest commit strategy, v1→v2 migration + startup guard, entity progress bucketing
• --help smoke tests on every subgroup (guards against typer/click ABI breaks)

Version pinning notes

• typer >=0.15.0,<1.0.0 — 0.12.x misparses options against click 8.4+.
• click >=8.1,<8.2 — 8.2 changed Parameter.make_metavar() to require ctx, which breaks help rendering in every typer version we've tested.
• Both pins are baked into the published wheel; uv tool install dbtips-curate gets the working combo automatically.

Status

• Production-ready for 12 sections across target and target × disease dossiers.
• 87/87 tests passing.
• Multi-section tracking + per-ingest commit strategy validated end-to-end.
• Skills published, Claude Code Desktop verified end-to-end.

Roadmap

• Field-level brownfield detection — extend the git-history walker to also detect same-key field-level changes (currently identity-set only).
• MCP server — a mcp serve subcommand so the same skills can drive browser-based Claude conversations at claude.ai.
• Singleton-dict sections — /target-assessment/targetability/, /target-assessment/tractability/, and similar shape-mismatched endpoints. Design note lives at src/dbtips_curate/skill/_deferred/targetability-design-note.md.
• /genomics/evidence-heatmap/ — wildcard array_path support for sections keyed by target symbol.
• op accept — CLI form for resolving conflicts by choice. Merge engine already handles the semantic; only the CLI wrapper is missing.