athenaeum 0.13.5

Open source knowledge management pipeline — append-only intake, tiered compilation, configurable schemas

Athenaeum

Production-tested agentic memory for teams deploying multiple AI agents. Append-only intake, a tiered librarian that compiles raw observations into a trustworthy wiki, and a sidecar that makes recall happen passively on every turn.

Is this for me? If you're running more than one agent on shared knowledge — or if you want agents and humans reading and writing the same institutional memory — yes. If you're building a single-user chatbot, mem0 or Letta may be a better fit.

Why Athenaeum

Four design choices separate a production memory system from a single-user markdown file. Each one fixes something that quietly breaks when a team scales past one agent:

1. Sources as first-class objects — every claim carries provenance, the way Wikipedia does. An unfootnoted fact is an assertion.
2. The librarian — a tiered compilation pipeline — agents can only append to raw intake. A separate compiler is the only writer to the wiki. Safety from structure, not trust.
3. Passive recall — a hybrid FTS5+vector search fires on every turn and injects breadcrumbs into context. The agent doesn't have to remember to look.
4. An editable observation filter — what the agent saves is governed by a prompt you can read, edit, and audit. Not a black box.

Full rationale, comparison with alternatives (Claude memory, Anthropic's memory tool, RAG, Karpathy's gist, mem0/Letta/Zep/Cognee), and the lessons from running it on our own operations live in docs/why-athenaeum.md. For the companion blog post: What We Learned Running Our Own Operations on Agentic Memory.

Installation

Requires Python 3.11+.

pip install athenaeum

Quick start

# Initialize a knowledge directory
athenaeum init                  # default: ~/knowledge
athenaeum init --path ~/my-knowledge

# Run the librarian (compile raw intake → wiki entities).
# `athenaeum run` needs ANTHROPIC_API_KEY — use --dry-run to explore keyless.
# `run` and `status` operate on ~/knowledge by default (point elsewhere with
# `--path`; `run` also accepts `--knowledge-root` as the original spelling).
athenaeum run
athenaeum run --dry-run         # inspect without writing

# Check status
athenaeum status

Full run with custom paths and budgets (--max-api-calls 200 here deliberately lowers the per-run API budget below the default of 800 — omit the flag to accept the default):

athenaeum run \
  --raw-root ~/knowledge/raw \
  --wiki-root ~/knowledge/wiki \
  --path ~/knowledge \
  --max-files 50 \
  --max-api-calls 200 \
  --verbose

Data lifecycle & upgrade impact

Upgrade impact (0.10.0) — athenaeum run now MOVES and DELETES raw auto-memory by default. raw/auto-memory/ is an expiring intake queue, not a permanent store. As of 0.10.0, once the librarian has compiled a cluster into its canonical wiki/auto-<topic>.md entry and the contradiction detector has run clean, the move-then-retire pass (issue #261) moves each non-contradictory raw fact into the wiki entry (as an origin-traced footnote) and git rms the raw file so it no longer re-enters the nightly loop. This is on by default. If you upgrade and run without reading this, your raw auto-memory files will start disappearing from the working tree — recoverable, but only from git history.

What is moved vs. held. Only non-contradictory clusters are retired. A cluster is held in the queue (never deleted) when the detector flags a contradiction, when the detection degraded (offline / API error / unparseable response), or when a member is referenced by an open entry in _pending_questions.md / _pending_merges.md. When in doubt, the pass keeps the raw.

Recovery is git-only. The pass refuses to run when knowledge_root is not a git repo, and it never hard-unlinks. Each retirement lands as two commits in your knowledge repo:

• Commit A — provenance snapshot. The raw intake about to be retired is committed first (scoped git add of exactly those files) so every deleted file is recoverable from history.
• Commit B — move + delete together. The wiki updates (new footnotes, retired: true marker) and the raw git rms land in a single commit, so the fact is never simultaneously absent from both the raw file and the wiki.

To recover a retired raw file, find commit B (or A) in your knowledge repo and git show/git checkout the path. Warning: because recovery depends entirely on git history, anything that rewrites or discards that history can lose retired raw permanently — git gc pruning unreachable objects, a squash/rebase that collapses the snapshot commits, or simply never committing (running on a dirty repo) / never pushing to a backup remote. If you rely on retired-raw recovery, keep the knowledge repo's history intact and pushed.

Pushing after every run (opt-in, issue #284). A scheduled nightly run commits locally, but does not push by default — so origin silently drifts and the git-only recovery story only holds on the machine that ran the librarian. Two ways to turn on a post-run push so origin stays current:

athenaeum run --push               # one run: push after this run

# athenaeum.yaml — persistent opt-in
librarian:
  push_after_run: true
  # Optional; defaults are origin + the current branch's upstream.
  # push_remote: origin
  # push_branch: develop

The --push CLI flag overrides the yaml toggle. When enabled, athenaeum invokes git push (using the operator's ambient git auth — credential helper / SSH; no tokens or secrets are handled by athenaeum) after a successful run that produced at least one commit. --dry-run never pushes; a run with no new commits never pushes. A push failure is reported as a non-fatal warning (distinct log line athenaeum-push-failed:) — commits remain local and the next run retries (git push is idempotent).

--dry-run computes the exact same plan and logs a structured report without moving, deleting, or committing anything — use it to preview what a run would retire.

Disabling it. Move-then-retire stays on by default, but you can turn it off two ways:

athenaeum run --no-retire          # one run: skip the retire pass entirely

# athenaeum.yaml — persistent opt-out
librarian:
  retire: false

The --no-retire CLI flag overrides the yaml toggle. When disabled, raw auto-memory is neither moved into the wiki nor git rm'd; it stays in the intake queue and is re-examined on every run.

Related destructive operation. athenaeum auto-memory prune --apply is a second opt-in command that git rms pages (operational wiki/auto-*.md), with the same git-only recovery story as move-then-retire. It is dry-run by default — see Maintenance & inspection commands below.

Maintenance & inspection commands

Two subcommands operate over the compiled wiki outside the nightly run loop — one read-only, one with an opt-in destructive --apply:

# Read-only: find claims restated across distinct wiki entities.
athenaeum claims --find
athenaeum claims --find --threshold 0.9 --path ~/knowledge

claims --find is a cross-entity recurring-claim detector. It scans the wiki, embeds each claim via the recall-index embedding provider, and prints a YAML report grouping claims that recur across two or more distinct entities (default cosine cutoff 0.85, override with --threshold). It never mutates wiki/ — it only reports. With no embedding backend available it degrades to an empty report rather than failing.

# Cluster compiled wiki pages against each other and propose merges.
athenaeum dedupe wiki-pages
athenaeum dedupe wiki-pages --dry-run --threshold 0.6

dedupe wiki-pages clusters already-compiled concept/reference/principle wiki/*.md entity pages by topic/embedding similarity (issue #290) — complementing the raw-intake clustering that runs during athenaeum run. True duplicates are routed through the existing wiki/_pending_merges.md / resolve_merge approval flow (never auto-applied). Writing a proposal is idempotent — rerunning for a source set already proposed is a no-op. --dry-run previews without writing; --threshold overrides librarian.cluster_threshold (default 0.55, the same cutoff the raw auto-memory cluster pass uses). athenaeum run also runs this pass automatically whenever wiki/ exists — there is currently no toggle to opt out of the run-embedded pass; failures are logged (wiki-page dedup pass failed; continuing run) and non-fatal to the run.

# Archive resolved (approved/rejected) blocks out of the live sidecar.
athenaeum ingest-merges --path ~/knowledge

resolve_merge does not archive on its own — like resolve_question, it only flips the checkbox in place. Run ingest-merges (issue #299) to move every resolved block out of wiki/_pending_merges.md into wiki/_pending_merges_archive.md (newest-first, append-only), keeping the live sidecar limited to genuinely open proposals. Idempotent — this must be scheduled (or run periodically); nothing else archives resolved merges, which is exactly how the live file grew to 5MB/67K lines in production before this command existed (#299).

# Dry-run (default): print the kill-list + retained-list, change nothing.
athenaeum auto-memory prune
# Apply: git rm the kill-list in one commit and rebuild the recall index.
athenaeum auto-memory prune --apply

auto-memory prune retires operational/ephemeral wiki/auto-*.md pages (throwaway scratch scopes, install-token boilerplate, and pages flagged ephemeral: true) using the same classifier the intake gate applies. Dry-run is the default: it prints the kill-list and retained-list with a reason per page and exits 2 when candidates exist (a CI / sign-off signal), 0 when there is nothing to prune. --apply git rms the kill-list in a single labeled commit and then rebuilds the recall index; the commit pathspec is scoped to the kill-list, so unrelated staged work is never swept in. Like move-then-retire, recovery is git-only — the command refuses to run outside a git repo and never hard-unlinks, so a pruned page is recoverable from history.

MCP memory server

Athenaeum ships an MCP server exposing remember and recall tools so AI agents can write to raw intake and search the compiled wiki:

pip install 'athenaeum[mcp]'
athenaeum serve --path ~/knowledge

# Smoke test the round-trip without a live session
athenaeum test-mcp

Claude Code integration. Add to your MCP config and it auto-starts with every session:

claude mcp add --scope user athenaeum -- athenaeum serve --path ~/knowledge

Example round-trip:

User: Tristan's partner is Amanda; they met at Stanford GSB.

(Claude calls remember(content="Tristan's partner is Amanda; they met at Stanford GSB.", source="claude-session"))

A raw observation lands in raw/claude-session/20260417T…-…md. On the next athenaeum run, the pipeline compiles it into Tristan's wiki entity (under "Key Contacts") and Amanda's own entity if she doesn't exist yet. Later sessions can ask "who is Amanda?" and recall returns the compiled page.

Answering pending questions

When Tier 3 can't resolve an ambiguity or a principled contradiction, the librarian escalates to wiki/_pending_questions.md. Each escalation lands as a block like:

## [2026-04-20] Entity: "Acme Corp" (from sessions/20240406T120000Z-aabb0011.md)
- [ ] Is Acme still Series A after the 2026 recapitalisation?
**Conflict type**: principled
**Description**: Prior wiki says Series A; the 2026-04 raw file implies Series B.

You resolve a question one of two ways — pick whichever fits your workflow:

Option 1 — Edit the file directly

Flip [ ] to [x] on the checkbox line and type your answer below the checkbox (above or below the conflict-type / description lines — either works; the parser strips those metadata lines when extracting the answer):

## [2026-04-20] Entity: "Acme Corp" (from sessions/20240406T120000Z-aabb0011.md)
- [x] Is Acme still Series A after the 2026 recapitalisation?

They closed Series B on 2026-03-12, led by Acme Growth Partners.
The 2026-04 raw file is correct; the prior wiki entry is stale.

**Conflict type**: principled
**Description**: Prior wiki says Series A; the 2026-04 raw file implies Series B.

Option 2 — Use the MCP tool

For containerized agents that can't touch the filesystem, athenaeum serve exposes two tools:

• list_pending_questions() returns unanswered blocks as JSON — each item carries a stable id derived from the header + question text.
• resolve_question(id, answer) flips the checkbox and writes the answer body under it. It does not archive on its own — archival runs on the next ingest-answers pass.

Step 2 — ingest the answers

Either way, run:

athenaeum ingest-answers --path ~/knowledge

Each [x] block is rewritten as a raw intake file under raw/answers/{timestamp}-{entity-slug}.md with frontmatter linking back to the original source, then moved into wiki/_pending_questions_archive.md (newest-first, append-only — answered blocks are never deleted, only moved). The next athenaeum run picks the raw file up like any other intake and folds the answer into the wiki entity.

Re-running with no new [x] blocks is a no-op. Malformed blocks are preserved in place and logged to stderr, so a corrupt single entry cannot poison the rest of the file.

Transparent sidecar (Claude Code hooks)

For a fully passive experience where Claude auto-recalls relevant context on every prompt and saves observations without explicit commands, configure Claude Code hooks:

1. Copy the example hooks from examples/claude-code/ to your scripts directory.
2. Add hook entries to ~/.claude/settings.json (see examples/claude-code/settings-snippet.json).
3. Add CLAUDE.md instructions for proactive memory (see examples/claude-code/CLAUDE.md.example).

This gives you:

• Auto-recall — an FTS5 index is built at session start (~300ms); each user message triggers a <50ms search that injects relevant wiki pages into context.
• Auto-remember — Claude proactively saves important facts without being asked.
• Context checkpointing — observations are saved before context-window compaction.

Full setup guide, smoke test, and environment-variable reference: examples/claude-code/README.md.

Integrations

• Claude Code auto-memory — bridge ~/.claude/projects/<scope>/memory/ into Athenaeum's raw/ intake so the librarian can cluster, merge, and contradiction-check Claude Code's durable memory alongside other sources. See docs/integrations/claude-code.md.
• Contradiction detection — pipeline overview, cross-scope modes, source-precedence taxonomy, configuration reference, and cost model for the auto-memory contradiction path. See docs/contradiction-detection.md.

Vector search (optional)

Athenaeum supports a vector search backend (chromadb + all-MiniLM-L6-v2) for semantic recall alongside the default FTS5 keyword backend. The recall hook runs a hybrid FTS5 + vector merge when vector is configured — each backend rescues a failure class the other has (short-query proper-noun collisions for vector; no-lexical-overlap semantic queries for FTS5).

pip install 'athenaeum[vector]'

Enable it in athenaeum.yaml:

search_backend: vector

Full walkthrough and the four invariants a future simplification must not remove: docs/recall-architecture.md.

Query-topic extraction (optional)

athenaeum query-topics "your prompt" runs a Haiku classifier that returns substantive topics and ignores meta-instructions:

$ athenaeum query-topics "Without calling any tools, quote the block about Return Path verbatim"
Return Path

The naive regex+stopword fallback returns block,calling,quote,return,tools,verbatim,without — burying "Return Path" behind meta-instruction tokens. The example recall hook uses query-topics to rescue named-entity recall on instruction-heavy prompts and falls back silently to the regex extractor if the API key or CLI is unavailable.

Environment variables

The table below covers the common knobs. The exhaustive list — every env var, yaml key, and CLI flag with its code default and precedence chain — lives in docs/configuration.md.

Variable	Required	Description
ANTHROPIC_API_KEY	Yes (unless --dry-run)	API key for Tier 2/3 LLM calls
ATHENAEUM_CLASSIFY_MODEL	No	Override Tier 2 model. Precedence: env > models.classify in athenaeum.yaml > default claude-haiku-4-5-20251001
ATHENAEUM_WRITE_MODEL	No	Override Tier 3 model. Precedence: env > models.write in athenaeum.yaml > default claude-sonnet-4-6
ATHENAEUM_RESOLVE_MODEL	No	Override the contradiction-resolver model (default: claude-opus-4-7)
ATHENAEUM_RESOLVE_MAX_PER_RUN	No	Cap resolver calls per ingest run (default: 250, raised from 50 in #187)
ATHENAEUM_MAX_API_CALLS	No	Run-level API call budget for athenaeum run. Precedence: --max-api-calls CLI flag > env > librarian.max_api_calls in athenaeum.yaml > default 800. Env 0 is valid and defers the entire intake (writes wiki/_deferred_work.md and logs the DEGRADED summary); the CLI flag rejects 0
ATHENAEUM_MAX_FILES	No	Per-run intake batch size for athenaeum run. Precedence: --max-files CLI flag > env > librarian.max_files in athenaeum.yaml > default 50. Env 0 is valid (defer-everything window); the CLI flag rejects 0
ATHENAEUM_BATCH_MODE	No	Opt-in Batch API mode for athenaeum run (#236): tier-2/tier-3 calls are submitted as batches at a 50% token discount. Latency-tolerant — most batches finish within an hour, 24h worst case — intended for the nightly run. Precedence: --batch-mode / --no-batch-mode CLI flags > env > librarian.batch_mode in athenaeum.yaml > default off (--no-batch-mode forces the synchronous path even when env/yaml turn batch mode on)
ATHENAEUM_RESOLVE_AUTO_APPLY	No	Auto-apply high-confidence resolutions (default: true). See docs/auto-resolve.md
ATHENAEUM_RESOLVE_AUTO_APPLY_THRESHOLD	No	Confidence floor for auto-apply, in [0.0, 1.0] (default: 0.90)
ATHENAEUM_RESOLVE_FULL_BODY_TOKEN_CAP	No	Per-side body cap for the resolver's full-body context, ~4 chars/token (default: 1500; must be positive)
ATHENAEUM_CROSS_SCOPE_MODE	No	Cross-scope contradiction detection: off / ancestor / similarity / both (default: ancestor). See docs/contradiction-detection.md
ATHENAEUM_RESOLVED_SIMILARITY_THRESHOLD	No	Cosine threshold for matching new detections against the resolved-decision log (default: 0.83)
ATHENAEUM_TIER4_DEDUP	No	Dedupe pending-question escalations by source-memory pair (default: true; set false/0/no/off for legacy always-append)
ATHENAEUM_CACHE_DIR	No	Cache root for the librarian's embedding/cluster pass (default: ~/.cache/athenaeum)
ATHENAEUM_TOPIC_MODEL	No	Override query-topic model. Precedence: env > models.topic in athenaeum.yaml > default claude-haiku-4-5-20251001
ATHENAEUM_OP_KEY_PATH	No	1Password path for the session-start ANTHROPIC_API_KEY bootstrap (default: op://Agent Tools/Anthropic API Key/credential)
ATHENAEUM_PQ_SNOOZE_HOURS	No	Snooze TTL in hours for pending-questions surfacing (default: 24; consumed by the resolve-questions skill)
ATHENAEUM_PYTHON	No	Python interpreter used by the example hooks (default: python3)
AUTO_RECALL	No	Per-turn recall on/off (hook shell env; overrides athenaeum.yaml's auto_recall). Default: true
SEARCH_BACKEND	No	fts5 or vector (hook shell env; overrides athenaeum.yaml's search_backend). Default: fts5
ATHENAEUM_HOOK_DEBUG	No	Set to 1 to log vector-backend errors from user-prompt-recall.sh to stderr

Shell-env overrides. AUTO_RECALL and SEARCH_BACKEND are read from the shell environment after the hook sources ~/.cache/athenaeum/config.env, so exports in your shell profile beat the cached config. Intentional (lets you A/B-test a backend without editing athenaeum.yaml), but it's the first thing to check when the hook "ignores" a config change.

Claude Code auth caveat. Claude Code's own CLAUDE_CODE_OAUTH_TOKEN is scoped to its inference endpoint, and the Anthropic Messages API rejects it with 401 OAuth authentication is currently not supported. The pipeline and example hooks need a separate console API key — see docs/recall-architecture.md for the 1Password bootstrap pattern.

Configuration

Settings are resolved in the order CLI flag > env var > <knowledge_root>/athenaeum.yaml > built-in default, so a one-off shell export beats the yaml without requiring an edit. The canonical reference for every knob — librarian budgets, model selection, contradiction/resolver tuning, recall/search, and hook environment — is docs/configuration.md. As one example, the contradiction-resolver knobs live under a top-level resolve: block:

resolve:
  model: claude-opus-4-7          # ATHENAEUM_RESOLVE_MODEL
  auto_apply: true                # ATHENAEUM_RESOLVE_AUTO_APPLY (default: true)
  auto_apply_threshold: 0.90      # ATHENAEUM_RESOLVE_AUTO_APPLY_THRESHOLD, [0.0, 1.0]
  full_body_token_cap: 1500       # ATHENAEUM_RESOLVE_FULL_BODY_TOKEN_CAP, per-side body cap (~4 chars/token)

When auto_apply is on and a proposal's confidence meets or exceeds auto_apply_threshold, the pending-question block is auto-flipped to answered with an Auto-resolved: true audit-trail tag. See docs/auto-resolve.md for the full lane, including how to disable, lower the threshold, or reverse an auto-resolution.

Alternative model gateways. All model calls go through the Anthropic SDK, which honors ANTHROPIC_BASE_URL — so a LiteLLM proxy or any Anthropic-compatible gateway can serve alternative models with zero code change. Only Claude models are first-party tested; see docs/configuration.md for the details and #234 for multi-provider tracking.

Data formats

Raw intake lives in raw/{source}/*.md with the naming convention {timestamp}-{uuid8}.md (e.g., 20240406T120000Z-aabb0011.md). Each file is a plain markdown document containing observations, notes, or session transcripts. The {source} directory identifies the origin (e.g., sessions, imports).

Wiki entity pages live in wiki/ with YAML frontmatter:

---
uid: a1b2c3d4
type: person
name: Alice Zhang
aliases: [Alice]
access: internal
tags: [active]
created: '2024-04-06'
updated: '2024-04-06'
---

Entities are indexed in wiki/_index.md grouped by type. Conflicts requiring human review are appended to wiki/_pending_questions.md. Each run logs token usage and estimated costs at the end.

Degraded runs. When a run exhausts its API call budget (see ATHENAEUM_MAX_API_CALLS above), it writes a wiki/_deferred_work.md manifest itemizing the raw files it did not process and ends with a warning-level Done (DEGRADED — budget exhausted) summary line — the machine-greppable signal that intake was deferred rather than completed. The deferred files stay on disk and are picked up automatically by the next run. The manifest is overwritten on every budget-tripped run and cleared by the next clean run (full, merge-only, or cluster-only). The cap is enforced at the entity-tier loop; merge-phase and re-resolve calls count toward the budget but do not themselves stop the run, so a merge-heavy run can overshoot the cap before enforcement kicks in. A degraded run still exits 0 by default; pass athenaeum run --strict-budget to make a budget-tripped run exit nonzero instead — opt-in, for exit-code-based alerting.

Known limitations

Athenaeum is pre-1.0. These trade-offs are intentional for the current release line:

• No retrieval benchmarks yet. The hybrid-search claim rests on concrete failure modes (proper-noun collision, no-overlap semantic queries) and production use — not a published eval against mem0 / Letta / Zep / Cognee. If you need benchmarked recall@k on a closed corpus, pick a tool that publishes numbers. If you want a knowledge base that survives your tool choices, this is for you. PRs adding an eval harness are very welcome.
• FTS5 index rebuilds are non-atomic and unlocked. A shell hook and the librarian run rebuilding simultaneously can race; the window is small and single-user wikis do not hit it in practice, but hardened multi-writer safety remains future work. Workaround: don't invoke athenaeum rebuild-index and athenaeum run concurrently on the same $KNOWLEDGE_ROOT.
• The keyword search backend is a scan-on-query fallback. It reads every wiki page on every query; fine under ~1,000 entities, painful past that. Use search_backend: fts5 (default in the CLI and hooks) for any non-trivial wiki. The keyword backend exists as a zero-dependency baseline for tests and bootstrap.
• Tier 4 (human escalation) is a file, not a workflow. Conflicts land in wiki/_pending_questions.md; you read it and decide. No PR-opening, no Slack integration, no UI — on purpose, for now.

Development

git clone https://github.com/Kromatic-Innovation/athenaeum.git
cd athenaeum
pip install -e ".[dev]"

pytest tests/ -v
ruff check src/ tests/

Branch flow

Athenaeum follows trunk-style development, with develop as the active branch and main as the released-revision pointer:

• develop is the active development branch and the GitHub default. All pull requests target develop.
• main carries the most recent released revision. Release tags (vX.Y.Z) live on main and trigger the PyPI release workflow.

Most users should install via pip install athenaeum (above). To work from source against the latest released revision instead of the active branch, clone and check out the latest tag:

git clone https://github.com/Kromatic-Innovation/athenaeum.git
cd athenaeum
git checkout "$(git describe --tags --abbrev=0)"

See CONTRIBUTING.md for the full promotion flow.

Getting help

Rolling this out on a team? Open an issue or reach out via kromatic.com. We talk to teams working through agent-memory rollouts often and are happy to point at whatever's useful.

License

Apache 2.0 — see LICENSE.