eidetic-cli

Agent/CLI providing eidetic perfect-recall memory

What you get

• An agent-first CLI cited from teken (afi-cli) — the runtime declares data-refinery-cli[store] as its storage dependency; neo4j and pymongo arrive transitively via that extra. Consumers stay dependency-free because they call eidetic over a subprocess boundary.
• A mesh identity — culture.yaml (suffix + backend) and the matching prompt file (CLAUDE.md for backend: claude).
• The canonical guildmaster skill kit under .claude/skills/, vendored cite-don't-import, plus eidetic's own first-party remember / recall skills (a shared ~/.eidetic/memory store both Claude and the colleague backend can drive). See docs/skill-sources.md.
• A build + deploy baseline — pytest, lint, the agent-first rubric gate, and PyPI Trusted Publishing wired into GitHub Actions.

Quickstart

uv sync
uv run pytest -n auto               # run the test suite
uv run eidetic whoami              # identity from culture.yaml (console script is `eidetic`, not `eidetic-cli`)
uv run eidetic learn               # self-teaching prompt (add --json)
uv run teken cli doctor . --strict  # the agent-first rubric gate CI runs

CLI

Every command supports --json. Results go to stdout, errors/diagnostics to stderr (never mixed). Exit codes: 0 success, 1 user error, 2 environment error, 3+ reserved.

Storage

The files backend (default) is self-contained — JSONL on disk, no services needed. For the mongo / neo4j backends, the storage stack is owned and published by data-refinery-cli as the GHCR image ghcr.io/agentculture/data-refinery-stack. Bring it up with:

data-refinery stack up              # mongo on host :27018, neo4j bolt :7687 / UI :7474
eidetic remember --backend mongo …  # eidetic's default connection settings already match

The --backend token is uniform across every verb: files, mongo, neo4j, or graph (graph is an alias for neo4j, kept as the operator-preferred display label) — so a driving agent can reuse one token everywhere.

eidetic depends on data-refinery-cli[store] (0.6.x) and imports data_refinery.store / data_refinery.quality for storage operations. It keeps all memory semantics — the record schema, recall ranking modes, scoring, freshness signal, and no-hard-delete lifecycle — while the storage substrate is owned by data-refinery-cli. Storage mechanics never cross the boundary: even the on-disk format upgrade (migrate store) is delegated to data-refinery's store.migrate endpoint, so eidetic constructs no filesystem write path of its own. (Migration tracked in eidetic#13 / data-refinery-cli#1; the migrate endpoint in #8.)

Embeddings + rerank come from a separate HTTP endpoint — the local lobes fleet gateway, which fronts every role (embedder, reranker, cortex, …) on one OpenAI-compatible port. The per-role vLLM containers are not published to the host, so a per-gear port is always wrong; lobes endpoint embedder reports the live value.

The gateway routes on the request's model field, which is why embed and rerank name different models. It also enforces a bearer token: with no key set the request 401s and eidetic degrades to a deterministic local lexical fallback — recall still answers, but not semantically. Without --rerank, only approximate/hybrid recall use the endpoint; exact/keyword are pure lexical and work fully offline. --rerank changes that for every mode — see below.

Reranking

recall --rerank is an opt-in second pass over the primary tier with the cross-encoder reranker. It is off by default and accepted with every --mode. The stage runs after lifecycle filtering (a shadowed or archived record is never shipped to the reranker) and before the --top-k slice, so the wider --rerank-pool (default 50) can rescue a record the search mode ranked below k — that rescue is the reason the pool exists. The same width cuts the other way: under --rerank only pooled records reach the primary tier, so a pool narrower than --top-k shrinks the result set.

Three things to know before turning it on:

• Ordering diverges from score. Items are ordered by the new rerank_score field; the existing score keeps its hybrid/BM25 value and is never overwritten. A consumer that re-sorts items by score will disagree with the order that was emitted — keep the emitted order, or sort by rerank_score. When the stage ran, the bundle also carries "rerank": {"lane": "remote"|"local"|null, "dropped": N}.
• rerank_score is on every item, always. A default recall that never ran the stage emits "rerank_score": null, exactly as score and signal are null before they are computed — so the item schema stays constant instead of varying with the flags. This is the one key --rerank adds to default output; the payload's own keys are unchanged unless the stage ran.
• exact and keyword are offline-safe only without --rerank. The stage needs the remote lane, and fails closed (exit 2, naming the API-key variables above) when that lane does not answer, rather than passing local lexical-overlap numbers off as cross-encoder scores. --rerank-allow-fallback is the explicit opt-in to the local lane; taking it names the lane in the payload, warns once on stderr, and skips the remote-calibrated --rerank-threshold.
• --rerank-threshold removes supporting records, not just junk. The cross-encoder is near-binary. Measured against the live lane for "how does eidetic recall traverse the links graph", the two records that directly answer it scored 0.9998 and 0.9963, while two records that genuinely are about traversal (the pure engine; the max_nodes/truncated bounds) scored only 0.0048 and 0.0035 — against an irrelevant-document floor of about 1.7e-04. Relevant supporting material sits one order of magnitude above noise and three below a direct answer, so any cutoff that clears the noise also discards it. There is no default cutoff: --rerank reorders but never filters until you ask. A literal --rerank-threshold 0.0 drops nothing, because the scores are strictly positive.

--rerank POSTs the pooled records' text to the reranker endpoint, --rerank-pool documents per batch. Without it, only the query string leaves the process — a private-scope caller should weigh that before opting in.

Borrowed credentials are scoped. EIDETIC_EMBED_API_KEY is eidetic's own variable — set it and it is sent wherever you point EIDETIC_EMBED_URL. The other two belong to sibling tools and are borrowed so a configured box needs no extra setup; because you never paired them with eidetic's endpoint, they are sent only to a loopback or https:// URL and withheld from a cleartext remote host (with a one-time warning on stderr). lobes tunnel publishes over HTTPS, so remote deployments are unaffected. Passing api_key="" explicitly disables auth regardless of the environment.

Reference deployment. The default above is the reference rig's real embedder, and it is a single agreed value across every surface — the eidetic/memory/embed.py code default, the vendored recall.sh/remember.sh wrapper exports, and this README (eidetic-cli#28 / colleague#293, drift-tested by tests/test_embed_default_drift.py; see docs/contract.md for the machine-readable record). Agreement alone is not correctness: #28 aligned all three surfaces on :8002, a per-gear container port never published to the host, so every surface agreed on an endpoint that always refused — and the lexical fallback hid it. Check changes against the live fleet (lobes endpoint embedder), not just against each other. On the AgentCulture mesh, the same endpoint is discoverable rather than hardcoded: the lobes gateway serves an embedder role in its GET /capabilities response (lobes-cli >= 0.38, where each role's endpoint field is client-reachable — see colleague's colleague/lobes.py), and a consumer such as colleague resolves that role and injects EIDETIC_EMBED_URL/EIDETIC_EMBED_MODEL into the environment before shelling out to eidetic. That injection never overrides an operator's own explicit export: EmbedClient.__init__ (eidetic/memory/embed.py) reads os.environ.get("EIDETIC_EMBED_URL"/"EIDETIC_EMBED_MODEL") ahead of the built-in default and no CLI flag intervenes between the two, so an explicitly-set env var always wins — eidetic's own env-first resolution order already guarantees this; lobes discovery just supplies a smarter default for the same environment slot, never a competing one.

Freshness signal

Every record carries temporal state used to compute a freshness signal — a float in [0, 1] that blends into recall ranking so recently-created and frequently-recalled records surface ahead of stale ones:

• created — ISO-8601 date stamped at remember time; drives the age-decay factor (1/(1 + days_old * DECAY_RATE)).
• last_recall + recall_count — updated passively on every recall hit; drive an access bonus (capped at +0.5) and a staleness penalty (days_since_recall * DECAY_RATE).
• links — related-memory references; reserved for a future corroboration term (weight is currently 0.0, the hook is wired).

The signal is computed at recall time and exposed as signal in every hit alongside score. Records with no temporal data (undated legacy records) pass through unmodified — the blend is an exact identity for them.

The blend is multiplicative around the neutral midpoint (SIGNAL_BLEND_BETA = 0.25), so a fully neutral signal is a no-op and only records carrying real temporal data move in rank. The formula:

access_bonus = min(0.5, recall_count * 0.05)
age_factor   = 1 / (1 + days_old * 0.01)
staleness    = days_since_last_recall * 0.01
signal       = clamp((0.5 - staleness + access_bonus) * age_factor, 0, 1)
blended_score = score * (1 + 0.25 * (signal - 0.5))

Attribution (added_by)

Every record carries an added_by field that identifies the agent or caller that ingested it. eidetic remember auto-stamps the field when it is absent from the record JSON, using this resolution order:

1. --added-by <value> — explicit override on the CLI flag.
2. culture.yaml mesh nick — the suffix declared in the repo's culture.yaml; this is the normal case when running inside a mesh agent.
3. None — when no culture.yaml is present (e.g. a wheel install or a bare subprocess call without a repo context).

An explicit added_by value already present in the record JSON is always preserved verbatim — remember never overwrites a caller-supplied attribution. The field is None for legacy records that pre-date this feature.

eidetic overview (and overview --store --scope <name>) reports distinct contributors per scope: the union of each record's added_by and any metadata.author value, deduplicated and sorted.

# Ingest as a named caller:
eidetic remember --added-by my-agent '{"id":"r1","text":"hello","type":"note"}'

# Override the mesh nick for a bulk import:
echo '{"id":"r2","text":"world","type":"note"}' | eidetic remember --added-by importer

# View contributors per scope:
eidetic overview --scope default

Lifecycle (no hard-delete)

eidetic never deletes a record. Records move through a lifecycle state machine:

Transitions are applied by eidetic sweep (the only command that writes lifecycle changes). --dry-run reports without writing. Records with metadata.protected set to a truthy value are exempt from all transitions.

Within-scope shadowing only. A supersedes link only shadows its target when both records share the same scope (name and visibility). Cross-scope links are ignored, preserving the public/private no-leak invariant.

Ingest with supersedes and links:

# New version of a record shadows the old one (same scope required):
eidetic remember '{"id":"r2","text":"...","type":"note","supersedes":"r1","links":["r3","r4"]}'

Then run eidetic sweep to apply the transition: r1 gets lifecycle=shadowed, r2 stays active.

Migrate QQ memory

eidetic migrate qq performs a one-shot idempotent import of the legacy QQ (Claude's personal) memory stack into a private eidetic scope:

• Sources read: ~/.claude/skills/memory/references/core.md and notes.md (one record per ## section), MongoDB claude_notes collection, Neo4j entities tagged knowledge_context="claude".
• Destination: --scope qq --visibility private by default — personal data never leaks into a public recall.
• Idempotent: stable per-source ids (qq-file:<path>#<section-slug>, qq-mongo:<id>, qq-neo4j:<id>) make re-runs safe.
• Resilient: a down Mongo or Neo4j is skipped with a warning, not fatal.

eidetic migrate qq --json           # migrate from all sources, JSON report
eidetic migrate qq --backend mongo  # store into eidetic's mongo backend
eidetic migrate qq --file ~/my.md   # restrict to a specific markdown file

Known limitations (tracked follow-ups): --filter is exact string-equality on metadata (time-range filtering is future work); the files backend re-embeds candidates per search (no embedding cache yet); the Neo4j backend fetches nodes and ranks in Python (vector-index pushdown is future work).

Make it your own

1. Rename the package eidetic/ and the eidetic-cli CLI/dist name throughout pyproject.toml, the package, tests/, sonar-project.properties, and this README.md. The name is hard-coded in ~30 files, so list every occurrence first — see the git grep discovery command in CLAUDE.md, the authoritative rename procedure.
2. Edit culture.yaml with your suffix and backend.
3. Rewrite CLAUDE.md for your agent and run /init.
4. Re-vendor only the skills you need from guildmaster (see docs/skill-sources.md).

See CLAUDE.md for the full conventions (version-bump-every-PR, the cicd PR lane, deploy setup).

License

MIT — see LICENSE.