dbtips-curate 0.3.0

Three-way merge curation for dbtips pipeline-generated dossier caches. Ships agentskills.io-compatible skills for Claude Code, OpenAI Codex, and Google Antigravity.

dbtips-curate

Three-way merge curation for dbtips pipeline-generated dossier JSON caches.

Scientists hand-curate cache files after pipeline generation (deleting noise, adding records from external sources, patching status fields). When the pipeline refreshes, those edits get wiped. dbtips-curate preserves scientist intent as an append-only git-tracked operation log and replays it onto every refresh — so the merged served file is always "fresh pipeline data + scientist intent" without manual re-curation.

Scope

• Sections (section-agnostic engine; add more via config — see docs/adding-sections.md):
  • /evidence/target-literature/ — target_disease, key PMID
  • /target-assessment/antibodies/ — target, key Antibody + Provider
  • /evidence/target-pathway/ — target, key image_url
  • /target-assessment/assayability/ — target, key ChEMBL ID
• Entity-type folders: cached_data_json/target_disease/<target>-<disease>.json and cached_data_json/target/<target>.json
• Implemented ops: delete, add, undo
• Deferred to v2: patch, accept CLI commands (merge engine handles both)

Adding a new section? Read docs/adding-sections.md — it has the data constraints to check, a copy-paste validator, and the config-vs-code decision flow.

How it actually works (the model in one paragraph)

Every tracked entity has a curation/<entity-type>/<id>/ folder containing: immutable raw snapshots of the section (Base for 3-way merge), an append-only ops.jsonl (scientist intent), state.json (mode + pointers), and per-refresh changelogs. Each scientist action is one git commit, so blame works on every clinical decision. On every pipeline refresh, entity ingest takes a fresh snapshot (= Theirs), runs a 3-way merge (Base + Theirs + ops → Merged), and splices the merged section back into the served file. Scientist edits survive across refreshes; pipeline updates land on top.

Install

The package is published to public PyPI. Any Python tool runner works.

End-user install (recommended)

# One-shot, no persistent install (great for trying it out):
uvx dbtips-curate --version

# Persistent global install (recommended for daily use):
uv tool install dbtips-curate
# or:  pipx install dbtips-curate
# or:  pip install dbtips-curate

dbtips-curate --version              # dbtips-curate/0.2.0

Then bootstrap the skills into your AI coding harnesses:

dbtips-curate dev bootstrap

Interactive menu picks Claude / Codex / Antigravity / all. Non-interactive form:

dbtips-curate dev bootstrap --targets all
dbtips-curate dev bootstrap --targets claude,codex
dbtips-curate dev bootstrap --targets claude --project .

Skill install matrix

Every supported harness reads the same agentskills.io SKILL.md format we ship — no conversion.

Harness	Global path	Project path
Claude Code	~/.claude/skills/<name>/	<project>/.claude/skills/<name>/
OpenAI Codex	~/.agents/skills/<name>/	<project>/.agents/skills/<name>/
Google Antigravity	~/.gemini/config/skills/<name>/	<project>/.agents/skills/<name>/ (shared with Codex)

Dev install (contributors only)

git clone https://github.com/aganitha/dbtips-curate.git
cd dbtips-curate
poetry install
poetry run dbtips-curate --version

For the test suite: poetry install --with dev && poetry run pytest.

Version note: typer 0.12.x is incompatible with click 8.4+ (released April 2026) — options like --path get misparsed as positional args and --version stops exiting eagerly. The typer = "^0.15.0" pin in pyproject.toml sidesteps this. If you see "Got unexpected extra argument" or "Missing command" on a valid flag, upgrade typer.

One-time setup (per platform repo)

From the platform repo root, create the curation folder + per-team config:

dbtips-curate dev init \
    --path backend/res-immunology-automation/res_immunology_automation/src/scripts/curation

This drops .dbtips-curate.toml next to cached_data_json/ and registers all ten curated section endpoints (literature, antibodies, pathway, assayability, target-pipeline, ontology, gene-essentiality, orthologs, supplementary materials). The file is committed to git so every scientist gets identical config.

Commands at a glance

dbtips-curate
  entity      list | status | track | snapshot | diff | ingest | records | compare | finalize
  op          delete | add | patch | undo
  history     ops | report
  dev         init | bootstrap | install-skill

Every command supports --json for machine-readable output, exits 0/1/2 for success / runtime error / usage error, and follows the contract: {"ok": true, "data": {...}} on success, {"ok": false, "error": {"code", "message", "hint"}} on error.

entity — lifecycle

Command	Purpose
entity list [--type T]	All tracked entities of one type with their mode + last refresh
entity track <id>	Bootstrap. Walks git history of the served file → synthesizes delete ops for historical removals; flags historical additions for scientist review; sets mode based on what was found (see modes below). --skip-history opts out.
entity snapshot <id> [--reason ...]	Capture the current served file's section as an immutable raw snapshot. First snapshot becomes Base for 3-way merge.
entity status <id>	Prioritized worklist: conflicts → historical_additions → bootstrap_candidates → resurrected → new from pipeline. Includes suggested_filters for greenfield noise triage.
entity records <id> [--page N --page-size 20] [--full] [--fields F1,F2]	List current records, paginated. For tabular review in chat — Claude renders as markdown table with delete action per row.
entity diff <id>	Read-only preview of the 3-way merge vs the current served file. Writes nothing. Use before ingest if scientist wants to see the table first.
entity ingest <id>	Take fresh snapshot → 3-way merge → splice merged section back into served file. Writes changelog + conflicts file. This is what the pipeline calls.
entity finalize <id> [--no-push]	Mark current review complete: advance last_reviewed_raw, clear bootstrap_candidates, prune old snapshots, git commit. Pushes to remote if auto_push_on = "finalize".

op — scientist actions

Command	Purpose
op delete <id> --section <s> --key K=V [--key K2=V2 ...] --reason "..."	Tombstone a record. Whole-record delete. Compound-key sections pass one --key flag per identity field. Per-op git commit.
op add <id> --section <s> --record-file F --source "..."	Add a record from an external source. Validates against the section's required_fields. Use - to read from stdin.
op patch <id> --section <s> --key K=V [--key K2=V2 ...] --set field=value [--set ...] --reason "..."	Change one or more fields on an existing record. Values are JSON-decoded when possible (Phase=3 → int, Approved=true → bool). Identity fields cannot be --set (rejects with PATCH_TOUCHES_IDENTITY).
op undo <id> --undoes <op_id> --reason "..."	Cancel a prior op. Endpoint inferred from the target op. Rejects unknown op_ids, undo-of-undo, and double-undo. Original op stays in the log for audit.

All support --dry-run to preview what would be logged without writing.

history — read-only views

Command	Purpose
history ops <id> [--since DATE] [--actor EMAIL]	The full operation log with optional filters.
history report <id> [--output FILE] [--open]	Generate standalone HTML report (all queues + ops table). --open launches it in your browser. Good for review meetings or async sharing.

dev — setup + maintenance

Command	Purpose
dev init --path DIR	Create the curation folder + write a default .dbtips-curate.toml.
dev bootstrap [--targets claude,codex,antigravity|all] [--project DIR]	Install every skill into the chosen AI coding harnesses. Interactive menu if --targets omitted. --project . also drops skills into the current repo's workspace-scope skill folder. Idempotent.
dev install-skill [--target claude|codex|antigravity] [--home DIR] [--only NAME]	Single-target install primitive used by bootstrap. Prefer bootstrap for end-user setup.

Mode state machine

Mode	Set by	Meaning
greenfield	track with no git history changes	Fresh pipeline output, no prior scientist edits detected
brownfield-from-git	track when git history shows record-level adds/removes in past commits	Prior scientist edits detected via git history; synthesized delete ops + historical_additions queue surfaced
brownfield-detected	ingest (auto-promotion) when pipeline output drops Base records that aren't tombstoned	Records preserved as bootstrap_candidates pending scientist confirmation
brownfield-bootstrap	Legacy label; behaves identically	(Backward compatible)
active	finalize after first review session	Steady-state operation

Filesystem layout

backend/.../scripts/
  cached_data_json/                                  (pipeline writes here, served to clients)
  curation/                                          (this tool writes here, tracked in git)
    .dbtips-curate.toml                              (per-team config + section schema registry)
    target_disease/<entity-id>/
      raw/<ts>.json                                  (immutable section snapshots)
      ops.jsonl                                      (append-only scientist operations)
      state.json                                     (mode + last_reviewed_raw + git history summary)
      conflicts/<ts>.json                            (open conflicts after last merge)
      changelog/<ts>.md                              (human-readable per-refresh summary)
      merged.json                                    (post-merge section, spliced back into served file)
      reports/<ts>.html                              (HTML reports — only created on demand)

Three-way merge semantics

Each entity ingest runs:

Base    = snapshot at state.last_reviewed_raw         (what scientist last accepted)
Theirs  = fresh snapshot of current served file       (what pipeline just produced)
Ours    = apply_ops(Base, ops.jsonl)                  (scientist intent layered on Base)

Merged  = three_way_merge(Base, Theirs, Ours)

Per identity-keyed record (PMID for literature):

In Base	In Theirs	Has op	Result
✓	✗	tombstone	silent (both agree to remove)
✓	✗	none	removed_by_pipeline queue — in brownfield modes, preserved as bootstrap candidate
✗	✓	none	new_from_pipeline queue — added to merged
✗	✓	tombstone	dropped from merged, logged as resurrected_tombstone
✓	✓	patch	per-field merge; pipeline-changed + scientist-patched same field → conflict (scientist wins by default)
✗	✗	add	scientist-added record, included in merged

Pipeline integration (build_dossier.py)

Already wired in via a CURATION HOOK block at the top of build_dossier.py. Two knobs:

IS_REFRESH: bool = False        # True = clear section + redis before pipeline regen
CURATE_ENABLED: bool = True     # False = bypass the hook entirely (rollback)

With IS_REFRESH = True, the hook (in order):

1. Snapshots the current served file's section (pre-refresh Base)
2. Clears the section key from served JSON + redis
3. Lets the pipeline regenerate from scratch
4. Calls entity ingest → 3-way merge → splices merged back into served file

With IS_REFRESH = False (the safe default), only step 4 runs — keeps curation state in sync without forcing regeneration.

Driving from Claude (Claude Code only)

After dev install-skill, in any Claude Code session:

curate acvr2a-no-disease

Claude reads the SKILL.md, then:

1. Runs entity status and presents the worklist as a markdown table
2. Surfaces brownfield-from-git evidence (commits walked, historical adds/removes) prominently before anything else
3. Walks decisions one-by-one, calling op delete / op add with --json after explicit confirmation
4. On show me records, renders entity records --page N --page-size 20 as a paginated markdown table with delete actions per row
5. On diff or preview, calls entity diff (read-only) and renders the four delta tables
6. On session end, calls entity finalize and summarizes

Claude Code is required. Skills work in Claude Code (terminal, desktop app, IDE extensions). The Claude.ai consumer Desktop app would need an MCP server wrapper (the mcp_server.py stub exists; mcp serve subcommand not built).

Testing

make test   # 26 tests covering merge engine + CLI end-to-end + git history walk + records pagination

The fixture in tests/fixtures/acvr2a-no-disease.json is a real 5-record slice of the production literature section. End-to-end tests run the full lifecycle (track → snapshot → ingest → op delete → simulate refresh → ingest → finalize) against a temp git repo mirroring the production layout.

Status

• Production-ready for literature curation on target_disease entities
• 26/26 tests passing
• Pipeline hook in build_dossier.py ready behind the IS_REFRESH / CURATE_ENABLED flags
• Scientist-facing skill installed via dev install-skill

Roadmap (deferred to v2)

• op patch (field-level updates) — merge engine handles them, no CLI yet
• op accept (resolve conflicts via choice)
• MCP server runtime (mcp serve subcommand) for Claude.ai consumer Desktop
• Field-level brownfield detection (needed when expanding to clinical trial endpoints)
• Other section schemas: /evidence/search-patent/, /market-intelligence/*, etc.