mempalace-refresh

Incremental, crash-resilient re-mining wrapper for mempalace.

Mempalace's built-in mine command is skip-if-filed by design — once a file has any drawer in the palace, mine never re-visits it. That's correct for static files, but wrong for live-appending Claude Code session logs that grow every message, across compactions, for days.

mempalace-refresh makes mempalace properly incremental:

• Only mines what's new since the last run (per-file mtime tracking)
• Subprocess per file — a ChromaDB segfault on file N doesn't nuke the batch
• Additive — upsert semantics + stable drawer IDs mean re-mining never deletes or duplicates; old chunks no-op, new chunks are appended
• source_file metadata stays correct — points at the real .jsonl, not a tmp path
• Targeted — mine just one session with ONLY <uuid>
• Fail-loud on API drift — if a mempalace update renames what we monkey-patch, the script exits 99 with a clear message

Install

pip install mempalace-refresh

Requires mempalace 3.2.x installed.

Usage

mempalace-refresh              # catch up: mine everything new since last run
mempalace-refresh STATUS       # per-file change status
mempalace-refresh ONLY <uuid>  # mine a single session (substring match)
mempalace-refresh RESET        # wipe state (next run re-mines everything)
mempalace-refresh REPAIR       # delegates to `mempalace repair --yes`

State lives at ~/.cache/mempalace-refresh/state.json. Nothing else is stored — the palace itself is mempalace's.

Environment

• PROJECTS_DIR — override ~/.claude/projects/ (Claude Code session logs)
• MEMPALACE_PALACE — override ~/.mempalace

How it works (short)

1. Each .jsonl under PROJECTS_DIR is tracked by mtime in state.
2. Changed files are mined one at a time via a fresh Python subprocess.
3. The subprocess monkey-patches mempalace.palace.file_already_mined to bypass mempalace's skip-check, and mempalace.convo_miner.scan_convos to feed it exactly one file.
4. Mempalace then scans the real .jsonl, runs its regex-based general_extractor, and upserts drawers. Because drawer IDs are hash(source_file + chunk_index):
   • Chunks that already existed → upsert updates metadata in place, no-op
   • Chunks for newly appended content → new IDs → genuinely new drawers
5. State is committed per file so any later crash loses zero progress.

Result: your palace reflects every session exactly as it would if you'd mined each one from the start, plus incremental additions for all subsequent growth.

License

MIT. See LICENSE.