harness-memory 0.2.0

Pluggable memory system with hierarchical recall, FTS search, and multiple backend support.

Harness Memory

Pluggable long-term memory system for LLM agents — structured recall, full-text search, and automatic knowledge extraction from conversations.

English · 中文

---

Highlights

• Zero Dependencies — Core package uses only Python stdlib + sqlite3; no heavy ML frameworks required
• L0→L4 Memory Pipeline — Automatic extraction of facts from conversations, promotion to structured atoms, entity pages, and audit journal
• Multi-Stage Recall — Query parsing, routing, multi-source gather, rerank, diversify, suppress, token budget, cache, and co-reference resolution
• Hierarchical Memory Tree — Manual root → branch → leaf structure; maintained independently and included in recall
• Pluggable Backends — SQLite (default), PostgreSQL
• Host Integrations — Native plugins for OpenClaw and Hermes agent platforms
• Full CLI — 20+ command groups for every layer of the system

---

How It Works

Harness Memory runs a five-layer pipeline that converts raw conversation events into structured, searchable knowledge:

L0  RawEvent       Host messages/events captured according to adapter configuration (immutable)
      │  LLM extraction (agent_end or scheduled)
      ▼
L1  Candidate      LLM-extracted fact awaiting promotion (Fact/Decision/Task/Preference)
      │  5-check promotion worker (value → evidence → entity → duplicate → conflict)
      ▼
L2  AtomCard       Structured fact linked to an entity, FTS-indexed, append-only
      │  grouped by
      ▼
L3  Entity         Anchor node (User/Person/Project/…) + alias table
    EntityPage     LLM-generated markdown summary, regenerated when marked dirty
      │  decisions and lifecycle changes logged to
      ▼
L4  Journal        Append-only audit log (promote/merge/conflict/deprecate/gc)

At recall time, a query runs through the recall pipeline (parse → route → gather → rerank → diversify → suppress → budget) and returns snippets from atoms and raw events. The hierarchical tree remains available as an organizational view: root/branch nodes store labels, while each leaf references one canonical AtomCard. EntityPage records are generated and readable through the page APIs, but page headlines are not yet a source in the main recall gather path.

---

Quick Start

Installation

# Core (zero dependencies — SQLite + FTS5 only)
pip install harness-memory

Optional extras:

# ── Storage backends ─────────────────────────────────────────────────────
pip install "harness-memory[postgres]"          # PostgreSQL backend (psycopg3)

# ── Vector search layer ──────────────────────────────────────────────────
pip install "harness-memory[chroma]"            # ChromaDB local vector index
pip install "harness-memory[qdrant]"            # Qdrant vector index
pip install "harness-memory[embeddings]"        # Local Sentence-Transformers model

# Typical vector search combo:
pip install "harness-memory[chroma,embeddings]"

# ── LangGraph checkpointer ───────────────────────────────────────────────
pip install "harness-memory[langgraph]"         # SQLite checkpointer
pip install "harness-memory[langgraph-postgres]" # PostgreSQL checkpointer

# ── CLI ──────────────────────────────────────────────────────────────────
pip install "harness-memory[cli]"

# ── Full stack (dev / self-hosted) ───────────────────────────────────────
pip install "harness-memory[postgres,chroma,embeddings,langgraph,cli]"

Python API

from harness_memory import Memory
from harness_memory.recall import recall_for_prompt_v2

memory = Memory(namespace="my-agent")

# Store a fact; this creates RawEvent → Candidate → AtomCard plus a leaf reference
memory.store("User prefers Python over Java", topic="preferences")

# Recall via the full M4 pipeline (atom + raw, reranked)
result = recall_for_prompt_v2(memory, "programming language preference")
print(result.rendered)   # markdown block ready for prompt injection

# Or recall canonical atoms projected as MemoryNode leaves
for node in memory.recall("programming language"):
    print(f"[{node.topic}] {node.content}")

Auto-extraction from conversations

from harness_memory import Memory
from harness_memory.extractor import CandidateExtractor
from harness_memory.llm import LLMClient  # plug in your host LLM

memory = Memory(namespace="my-agent")
extractor = CandidateExtractor(llm=my_llm_client)

# At session end: extract candidates from L0 raw events
from harness_memory.extractor import extract_session
result = extract_session(memory=memory, extractor=extractor, session_id="sess-123")

# Promote pending candidates to structured atoms
promotion = memory.promote_candidates()
print(f"promoted={promotion.promoted}, merged={promotion.merged}, conflicts={promotion.conflicts}")

CLI

# Ingest conversation files
harness-memory --namespace my-agent ingest --source ~/.claude/projects/myapp/

# Recall
harness-memory --namespace my-agent recall "user authentication preferences"

# Inspect the pipeline
harness-memory --namespace my-agent candidate list --status pending
harness-memory --namespace my-agent atom list --entity-id <id>
harness-memory --namespace my-agent entity list
harness-memory --namespace my-agent memory tree

# Operations
harness-memory --namespace my-agent export --out backup.jsonl
harness-memory --namespace my-agent gc run

---

Architecture

Storage Layers

Layer	Type	Table(s)	Description
L0	RawEvent	raw_events	Immutable evidence log
L1	Candidate	candidates	LLM-extracted facts awaiting promotion
L2	AtomCard	atoms	Promoted structured facts; primary recall source
L3	Entity, EntityPage, Alias	entities, entity_pages, aliases	Entity anchors + auto-generated summaries
L4	JournalEntry	journal	Append-only audit log
—	MemoryNode	memory_nodes	Hierarchical organization; leaves reference canonical atoms

Promotion Pipeline (L1 → L2)

The promotion worker runs 5 checks in order. First terminal result wins:

1. Value — drop if importance=low AND confidence=low (chit-chat noise)
2. Evidence — verify cited raw event IDs exist in L0
3. Entity — resolve or create the entity this fact belongs to (alias → name → LLM → new)
4. Duplicate — merge if an identical assertion already exists on the entity
5. Conflict — flag if polarity flips vs an existing atom (negation detection)

Recall Pipeline (M4)

recall_for_prompt_v2 runs the following stages per query:

cache lookup → parse query → route (entity/time/coref hints)
→ gather (atom+raw)
→ rerank (BM25 + importance + confidence + recency + layer_prior)
→ diversify (≤N per entity) → suppress (Jaccard dedup)
→ token budget enforcement → active-entity update
→ render markdown → cache result

Memory Tree ↔ Canonical Atoms

The MemoryNode tree keeps its original root → branch → leaf model without creating a second fact store:

• Root / branch: store organizational labels in MemoryNode.content.
• Leaf: stores a unique atom_id; backends project AtomCard.assertion into MemoryNode.content when reading.
• Manual write: Memory.store(..., level="leaf") creates RawEvent → Candidate → AtomCard, then links the leaf.
• Promotion / direct atom write: Memory.add_atom() creates the same leaf reference automatically.
• Recall: Memory.recall() searches atoms and returns their linked leaves; recall_for_prompt_v2() gathers atom + raw only.

Leaf facts are append-only through AtomCard: Memory.update(..., content=...) rejects leaf text changes, while deleting a leaf deprecates its atom. This preserves the tree API and removes the previous two-source-of-truth ambiguity.

---

CLI Reference

harness-memory [GLOBAL OPTIONS] COMMAND [ARGS]

Global options: --config PATH, --backend sqlite|postgres|qdrant, --db PATH, --dsn DSN, --namespace NS, --json

The CLI still accepts qdrant as a reserved backend choice for compatibility, but this repository does not currently ship a QdrantMemoryBackend.

Command group	Description
config show/set	Show or set configuration values
ingest	Archive conversation files (Claude Code, OpenAI, generic JSONL)
conversations list/get/export	Manage stored conversations
summary pending/set	Manage conversation summaries
raw add/list/search/show	L0 raw event management
candidate extract/list/show/promote/review/fallback	L1 extraction, promotion, inspection, and manual review
atom list/show/search	L2 atom inspection
entity list/show	L3 entity management
page show/list-dirty/regen/edit	L3 entity page management
journal list	L4 audit log
memory store/get/update/delete/tree/recall	MemoryNode tree management
recall	Run the full M4 recall pipeline
thread show	Inspect LangGraph checkpoint thread state
export / import / migrate / backfill	Migration and data operations
gc run, consolidate run	Lifecycle maintenance
openclaw setup/doctor/uninstall/print-config	OpenClaw host integration
hermes install/doctor	Hermes host integration

---

Host Integrations

Harness Memory can run as a native memory provider inside agent hosting platforms.

OpenClaw

pip install "harness-memory[cli]"
openclaw plugins install npm:@harnessmemory/openclaw-plugin
harness-memory openclaw setup    # register the plugin and bridge Python
harness-memory openclaw doctor   # verify the integration

The OpenClaw plugin (plugins/openclaw/harnessmemory/) is a TypeScript shell that spawns the Python bridge (harnessmemory-bridge) and registers registerMemoryCapability with the host runtime.

Hermes

pip install harness-memory-hermes
harness-memory-hermes install    # install the Hermes adapter
harness-memory-hermes doctor     # verify the integration

See docs/release-packaging.md for release artifact boundaries and docs/harnessmemory-openclaw-hermes-usage.md for source-checkout setup guides.

---

Python API Reference

Memory

from harness_memory import Memory

memory = Memory(namespace="my-agent", backend="sqlite",
                backend_config={"db_path": "~/my-memory.db"})

Method	Layer	Description
store(content, topic, level, parent_id, metadata)	Tree/L0-L2	Create a directory node, or create an atom-backed leaf
get(node_id)	Tree	Fetch a MemoryNode by ID
update(node_id, content, topic, metadata)	Tree	Update directory content or leaf organization; leaf fact text is immutable
delete(node_id, cascade)	Tree/L2	Delete organization and deprecate linked atoms
recall(query, limit)	L2→Tree	Search atoms and return linked MemoryNode leaves
get_tree()	Tree	Return the full MemoryNode tree
add_raw(content, event_type, ...)	L0	Append a raw event
get_raw(event_id) / list_raw(...) / search_raw(query, limit)	L0	Read or search raw events
add_candidate(candidate)	L1	Persist a candidate
get_candidate(id) / list_candidates(...) / search_candidates(...)	L1	Read or search candidates
promote_candidates(candidates, limit, llm_hook)	L1→L2	Run promotion worker
add_atom(atom)	L2	Persist an atom directly
get_atom(id) / list_atoms(...) / search_atoms(query, limit)	L2	Read or search atoms
add_entity(entity)	L3	Persist an entity
get_entity(id) / list_entities(...)	L3	Read entities
upsert_entity_page(page)	L3	Upsert an entity page
get_entity_page(entity_id) / list_dirty_entity_pages(...)	L3	Read entity pages
append_journal(entry)	L4	Append an audit entry
list_journal(...)	L4	Query the audit log
add_conversation(record)	Conv	Persist a conversation
search(query, limit)	Conv	FTS over conversation messages
list_conversations(...)	Conv	List conversation summaries
get_conversation(conversation_id)	Conv	Fetch a complete conversation

recall() and search() are intentionally different: recall() searches canonical atoms and returns their tree leaves, while search() searches archived conversation messages. Use recall_for_prompt_v2() below for the atom/raw prompt-recall pipeline with routing, reranking, deduplication, and budgeting.

Recall

from harness_memory.recall import recall_for_prompt_v2

result = recall_for_prompt_v2(
    memory,
    query="authentication flow",
    thread_id="thread-abc",   # enables co-reference resolution
    limit=5,
    total_budget_tokens=1500,
)
print(result.rendered)        # inject directly into prompt
for snippet in result.snippets:
    print(snippet.layer, snippet.text)   # "atom" | "raw"

---

Supported Conversation Formats

Format	Extension	Detection
Claude Code	.jsonl	type field: human_message / assistant_message
Generic JSONL	.jsonl	role + content fields per line
OpenAI JSON	.json	messages array with role/content objects

---

Development

make dev          # Install dev dependencies
make lint         # Ruff check + format check
make typecheck    # mypy strict
make test         # pytest
make all          # lint + typecheck + test

---

Related Projects

Project	Description
harness-agent	Production-grade AI agent platform built on LangChain Deep Agents
harness-browser	AI-friendly browser automation via CDP
harness-im-bridge	Multi-platform IM channel bridge for AI agents

---

License

MIT