YantrikDB — A Cognitive Memory Engine for Persistent AI Systems

Your agent starts every session as a stranger. Bolting a vector store onto it doesn't fix that: nothing is ever forgotten, near-duplicate memories pile up, and two contradictory facts come back ranked side by side with no signal that they disagree.

YantrikDB is the engine that manages memories instead of just storing them — temporal decay, autonomous consolidation, contradiction detection, and a knowledge graph, in an embeddable Rust library with Python bindings.

Get Started in 60 Seconds

For AI agents (MCP — works with Claude, Cursor, Windsurf, Copilot)

pip install yantrikdb-mcp

Add to your MCP client config:

{
  "mcpServers": {
    "yantrikdb": {
      "command": "yantrikdb-mcp"
    }
  }
}

That's it. The agent auto-recalls context, auto-remembers decisions, and auto-detects contradictions — no prompting needed. See yantrikdb-mcp for full docs.

As a Python library

pip install yantrikdb

record_text() / recall_text() work out of the box — no sentence-transformers install, no ONNX runtime. Just one pip install.

A new file-backed store opens on potion-base-8M (256-dim), fetched once (~28 MB, SHA-256 pinned, cached under your user cache dir) and self-hosted from yantrikos/yantrikdb-models — no HuggingFace dependency. If it cannot be fetched (offline), the store is created on the bundled potion-base-2M (64-dim, ~7 MB, no download ever) and a warning is logged. An existing database always reopens at the dimension it already holds, so upgrading the library never strands your data.

import yantrikdb

# New store: potion-base-8M @ 256 dims (downloads once).
# Existing store: reopened at whatever dimension it already has.
db = yantrikdb.YantrikDB.with_default("memory.db")

db.record("Alice is the engineering lead", importance=0.8, domain="people")
db.record("Project deadline is March 30", importance=0.9, domain="work")
db.record("User prefers dark mode", importance=0.6, domain="preference")

results = db.recall("who leads the team?", top_k=3)
# → [{"text": "Alice is the engineering lead", "score": 1.0}, ...]

db.relate("Alice", "Engineering", "leads")
db.get_edges("Alice")

db.think()  # consolidate, detect conflicts, mine patterns

db.close()

Why the default is the 256-dim model

The embedder choice is measured on real agent memory, not a leaderboard. Public benchmarks rank on Wikipedia-shaped text; agent memory is dense operational notes with heavy internal vocabulary and near-duplicate records that supersede one another, and it ranks the models differently.

Measured on 5,035 real production memories with 12 questions whose correct record was pinned by id, retrieved through the engine's own recall() (not raw cosine — the engine's hybrid lexical lanes and composite scoring are part of what you actually get):

The miss rate is the reason, not the MRR. Under the smaller model a third of real questions had no correct answer anywhere in the first hundred results — which to a user is indistinguishable from the memory not being there at all.

Two honest caveats. Twelve probes is a small set, enough to separate 0.120 from 0.312 but not to rank models a few points apart. And the gain is corpus-specific: on conversational-paraphrase benchmarks the two models are indistinguishable at every k from 2 to 80. Expect the benefit on dense, vocabulary-heavy stores; do not assume it transfers.

Other embedder options

# Larger still — 512-dim, ~121 MB, downloads on first call.
db = yantrikdb.YantrikDB("memory.db", embedding_dim=512)
db.set_embedder_named("potion-base-32M")

# Bring your own (sentence-transformers, fastembed, custom object).
from sentence_transformers import SentenceTransformer
db = yantrikdb.YantrikDB("memory.db", embedding_dim=384)
db.set_embedder(SentenceTransformer("all-MiniLM-L6-v2"))

# Force the bundled model — no download, works fully offline.
db = yantrikdb.YantrikDB("memory.db", embedding_dim=64)

A store's dimension is fixed when it is created. Switching models later means re-embedding — db.reembed("potion-base-8M") does it in place, preserving graph edges, consolidation state and conflict metadata.

As a Rust crate

[dependencies]
yantrikdb = "0.7"

# NOTE: the crate defaults differ from the pip package on purpose.
# `embedder-download` is OFF here, so a default cargo build has NO
# network code path at all and `with_default()` uses the bundled
# 64-dim potion-base-2M. The Python wheel enables it, so pip users get
# the 256-dim potion-base-8M default described above.
#
# To get that default (and set_embedder_named) in Rust, opt in:
# yantrikdb = { version = "0.7", features = ["embedder-download"] }
#
# Why not on by default: it pulls ureq + sha2 + dirs + tar + flate2
# into every build of an embedded database. See the measured retrieval
# difference above and decide for your deployment.

# Slim build (no bundled embedder, no network code path):
# yantrikdb = { version = "0.7", default-features = false }

The Problem

Current AI memory is:

Store everything → Embed → Retrieve top-k → Inject into context → Hope it helps.

That's not memory. That's a search engine with extra steps.

Real memory is hierarchical, compressed, contextual, self-updating, emotionally weighted, time-aware, and predictive. YantrikDB is built for that.

Why Not Existing Solutions?

Benchmark: Selective Recall vs. File-Based Memory

At 500 memories, file-based exceeds 32K context windows. At 5,000, it doesn't fit in any context window — not even 200K. YantrikDB stays at ~70 tokens per query. Precision improves with more data — the opposite of context stuffing.

Evidence (reproducible)

Every claim here points at a runnable harness — not a static number. Each is gated in CI (.github/workflows/benchmark.yml) so a regression fails the build.

• Recall doesn't degrade as the corpus grows, and stays fast. python -m yantrikdb.eval.benchmark holds a fixed signal corpus while adding distractors and measures recall + latency at each scale. Sample run: recall@k 0.938 → 0.929 as memories grow 7×, with p95 recall latency under 3 ms. regression_check() is the CI gate.
• The knowledge graph earns its keep on connected data. python -m yantrikdb.eval.graph_lift measures recall with entity-expansion ON vs OFF. Verdict on the connected corpus: +2.5% recall, +1.7% MRR — graph expansion helps where memories are actually linked.
• Apples-to-apples vs other memory systems. python -m yantrikdb.eval.competitors scores YantrikDB, mem0, Zep, and Letta on the same corpus, same queries, same metrics, no per-system tuning. (Competitors run once their libraries are installed; results are not pre-tuned.)

These run dependency-free on the bundled embedder, so anyone can reproduce them with one command.

LongMemEval-S — retrieval, scored mechanically

LongMemEval asks a question against a haystack of ~48 chat sessions per user, most of them near-identical distractors, and names the sessions that actually hold the evidence. This measures retrieval only — no answerer, no LLM judge, so there is nothing to disagree about. A query counts as all@k only if every gold session is in the top k (queries average 1.7 golds, so missing one scores zero).

479 of 500 queries scored, 0 errors. k=40 is the shipped default, and it is load-bearing rather than generous: a paired 400-query BEAM run measured k=40 → k=20 costing 3.4 rubric points across 8 of 10 categories.

The whole curve is published rather than the best row, because "recall@5" is not a well-defined quantity without saying what pool it was selected from — retrieving 40 and keeping the best 5 documents scores 85.0%, while retrieving 5 directly scores 72.9%, on the same queries with the same metric.

Reproduce: python lme_recall_multik.py 500 24 (one retrieval per query at k=40, every prefix scored).

Architecture

Design Principles

• Embedded, not client-server — single file, no server process (like SQLite)
• Local-first, sync-native — works offline, syncs when connected
• Cognitive operations, not SQL — record(), recall(), relate(), not SELECT
• Living system, not passive store — does work between conversations
• Thread-safe — Send + Sync with internal Mutex/RwLock, safe for concurrent access

Five Indexes, One Engine

┌──────────────────────────────────────────────────────┐
│                   YantrikDB Engine                    │
│                                                      │
│  ┌──────────┬──────────┬──────────┬──────────┐       │
│  │  Vector  │  Graph   │ Temporal │  Decay   │       │
│  │  (HNSW)  │(Entities)│ (Events) │  (Heap)  │       │
│  └──────────┴──────────┴──────────┴──────────┘       │
│  ┌──────────┐                                        │
│  │ Key-Value│  WAL + Replication Log (CRDT)          │
│  └──────────┘                                        │
└──────────────────────────────────────────────────────┘

1. Vector Index (HNSW) — semantic similarity search across memories
2. Graph Index — entity relationships, profile aggregation, bridge detection
3. Temporal Index — time-aware queries ("what happened Tuesday", "upcoming deadlines")
4. Decay Heap — importance scores that degrade over time, like human memory
5. Key-Value Store — fast facts, session state, scoring weights

Decoupled Write Path (v0.6.6+)

The vector index is structured as a two-tier LSM: a small mutable delta and an immutable HNSW cold tier swapped atomically via ArcSwap. Foreground writes only touch the delta (brief lock, O(1) push); HNSW work amortizes on a dedicated compactor thread. This is what eliminated the production wedge where sustained writes starved readers — see CONCURRENCY.md and docs/decoupled_write_path_rfc.md.

flowchart LR
    subgraph CLIENT["Caller"]
        C1["record / record_with_rid"]
        C2["recall / recall_with_seq"]
    end

    subgraph FG["Foreground — P1, brief locks only"]
        F1["assign_seq<br/>vec_seq.fetch_add<br/>(or fetch_max for cluster seq)"]
        F2["DeltaIndex.append<br/>brief RwLock&lt;Vec&gt; push"]
        F3["bump_visible_seq<br/>DashMap + AtomicU64<br/>(lock-free)"]
        F4["log_op → SQLite WAL"]
    end

    subgraph IDX["DeltaIndex (per engine)"]
        D1[("delta<br/>RwLock&lt;Vec&lt;DeltaEntry&gt;&gt;<br/>cap = delta_max (256)")]
        D2[("cold<br/>ArcSwap&lt;HnswIndex&gt;<br/>lock-free read")]
    end

    subgraph BG["Background — P3, dedicated threads"]
        B1["Compactor (1s tick)<br/>fires when delta past half-cap<br/>OR oldest entry > max_dirty_age"]
        B2["Materializer pool<br/>N = cores / 2<br/>drains pending oplog ops"]
    end

    subgraph STORE["SQLite (WAL mode, single file)"]
        S1["memories"]
        S2["oplog"]
        S3["entity_edges, sessions, ..."]
    end

    C1 --> F1
    F1 --> F2
    F2 --> D1
    F1 --> F3
    F1 --> F4
    F4 --> S2

    C2 -.->|"optional<br/>wait_for_visible_seq"| F3
    C2 --> D1
    C2 --> D2

    B1 -->|"seal + clone + ArcSwap.store"| D1
    B1 --> D2
    B2 --> S2
    B2 --> S1
    B2 --> S3

The structural invariant. Foreground (P1) and background (P3) do not share a lock primitive that holds for non-O(1) work. The cold tier is read lock-free via ArcSwap; the delta's RwLock is held for the O(1) push only. This is what makes "no single background task can wedge reads, writes, or recovery" enforceable — see CONCURRENCY.md Rules 2 and 3 for the names and failure modes if violated.

Cluster Mode (RFC 010 + Phase 6 RYW)

For multi-node deployments, yantrikdb-server wraps the engine with openraft for leader-elected replication. The four cluster-mutation primitives take the openraft commit-log index as their seq, so all nodes agree on a single global monotonic sequence — read-your-writes works across the cluster, not just within a node.

flowchart LR
    L["Leader<br/>HTTP request"]
    LR["Leader engine<br/>record_with_rid(seq=Some(log_idx))"]
    OR["openraft<br/>commit log"]
    F1["Follower 1 applier<br/>record_with_rid(seq=Some(log_idx))"]
    F2["Follower 2 applier<br/>record_with_rid(seq=Some(log_idx))"]
    R["Reader on any node<br/>recall_with_seq(min_seq=log_idx)"]

    L --> LR
    LR --> OR
    OR -->|replicate + apply| F1
    OR -->|replicate + apply| F2
    F1 -.->|"visible_seq[ns] reaches log_idx"| R
    F2 -.->|"visible_seq[ns] reaches log_idx"| R
    LR -.->|"visible_seq[ns] reaches log_idx"| R

Each record_with_rid / tombstone_with_rid / upsert_entity_edge_with_id / delete_entity_edge_with_id accepts an optional seq: Option<u64>. Single-node callers pass None and the engine allocates; cluster appliers pass Some(commit_log_index) and the engine ratchets vec_seq up to at least that value via fetch_max. After apply, visible_seq[namespace] reaches the log index, so any subsequent recall_with_seq(min_seq=N) blocks just long enough for the local node to have applied through index N — and no longer.

Memory Types (Tulving's Taxonomy)

All memories carry importance, valence (emotional tone), domain, source, certainty, and timestamps — used in a multi-signal scoring function that goes far beyond cosine similarity.

Key Capabilities

Relevance-Conditioned Scoring

Not just vector similarity. Every recall combines:

• Semantic similarity (HNSW) — what's topically related
• Temporal decay — recent memories score higher
• Importance weighting — critical decisions beat trivia
• Graph proximity — entity relationships boost connected memories
• Retrieval feedback — learns from past recall quality

Weights are tuned automatically from usage patterns.

Conflict Detection & Resolution

When memories contradict, YantrikDB doesn't guess — it creates a conflict segment:

"works at Google" (recorded Jan 15) vs. "works at Meta" (recorded Mar 1)
→ Conflict: identity_fact, priority: high, strategy: ask_user

Resolution is conversational: the AI asks naturally, not programmatically.

Semantic Consolidation

After many conversations, memories pile up. think() runs:

1. Consolidation — merge similar memories, extract patterns
2. Conflict scan — find contradictions across the knowledge base
3. Pattern mining — cross-domain discovery ("work stress correlates with health entries")
4. Trigger evaluation — proactive insights worth surfacing

Proactive Triggers

The engine generates triggers when it detects something worth reaching out about:

• Memory conflicts needing resolution
• Approaching deadlines (temporal awareness)
• Patterns detected across domains
• High-importance memories about to decay
• Goal tracking ("how's the marathon training?")

Every trigger is grounded in real memory data — not engagement farming.

Multi-Device Sync (CRDT)

Local-first with append-only replication log:

• CRDT merging — graph edges, memories, and metadata merge without conflicts
• Vector indexes rebuild locally — raw memories sync, each device rebuilds HNSW
• Forget propagation — tombstones ensure forgotten memories stay forgotten
• Conflict detection — contradictions across devices are flagged for resolution

Sessions & Temporal Awareness

sid = db.session_start("default", "claude-code")
db.record("decided to use PostgreSQL")  # auto-linked to session
db.record("Alice suggested Redis for caching")
db.session_end(sid)
# → computes: memory_count, avg_valence, topics, duration

db.stale(days=14)    # high-importance memories not accessed recently
db.upcoming(days=7)  # memories with approaching deadlines

Importing history. created_at (epoch seconds) records an event at the time it happened rather than the time it was loaded — so a bulk import keeps its real timeline and every temporal surface stays meaningful:

db.record("joined the observatory team", created_at=1_600_000_000.0)
db.record_batch([{"text": "...", "created_at": ts} for ts in anchors])

db.recall_as_of(march, query="where do they work")  # what was true then

Without it, every imported record shares the ingest wall-clock: decay and recency become insertion-order noise, and recall_as_of / time_window filter on a timeline that never existed. Omit it and the engine stamps now(), exactly as before.

For timelines assembled from evidence across sessions, keep created_at as the time the synthesized item became available and store the earliest evidence time in metadata.first_mention_at. Recall still selects the relevant top-k; order="first_mention" (or order="chronological") then presents those items oldest-first. Records without first_mention_at fall back to created_at.

Query-independent topic and concern organization is available from yantrikdb.organize. organize_evidence accepts an application-owned topic discovery callback, completes bounded evidence assignments deterministically, and persists evidence-versioned rollups. organize_concerns applies the same trust boundary to answer-sized ConcernItem values: every item must cite known evidence, evidence reuse is bounded, and persist_concerns records the full first-mention timeline through record_synthesis. recall_organized returns rollups for summary queries and expands them to concern or evidence items for list and timeline queries. Its default order="auto" uses first_mention_turn for questions about when something was brought up in conversation, while real-world timelines use first_mention_at and then created_at as a fallback.

For applications that need a complete consolidation checklist after their raw evidence, load_persisted_topic_cards enumerates every active topic handle by namespace without similarity top-k loss. topic_card_document renders each handle with its evidence-backed recorded date and turn span. This path is explicit: callers retain control over when the extra summary context is useful.

Organized recall also records a local rollup outcome ledger. A surfaced rollup gets an immutable impression ID with its hashed query, rank, score, namespace, requested item count, and coarse query shape; expansion records the ordered children actually returned and their serve-time scores. Applications can explicitly mark a returned child as selected or corrected with note_rollup_selection, then close the interaction with finalize_rollup_outcome. Finalization supplies the complete selected/corrected set; only then can an omitted returned child count as an explicit non-selection. Consumers may also pass omitted_child_rids when the user explicitly identifies an answer item retrieval failed to return. These are stored separately as caller_false_negative observations: they must have been active, same-namespace records available when the impression was served, and they never rewrite served history. The organizer cannot infer these labels itself; the application that observes the user's correction must finalize the interaction. Generic point reads, unfinished interactions, and ordinary corrections never infer a rollup outcome. rollup_outcome_report is a read-only, namespace/time scoped coverage report. rollup_outcome_examples exports bounded finalized per-child examples for offline calibration, with hashed queries and immutable serve-time rank/score features only; it never exposes query text or rebuilds features from mutable memory state. The stable query hash is a local linkage identifier, not anonymization, so exported artifacts should remain scoped and must not be published as de-identified data. Unselected means only that a returned child was omitted from the exact finalized set, not that an unseen memory was globally irrelevant. The readiness gate requires enough finalized queries, rollups, positive and negative children, at least 80% telemetry completion, and no dominant query or rollup. ready_for_offline_evaluation means only that an offline test is credible: these observations remain measurement data and are not ranker labels until their predictive value has been validated. rollup_membership_report has an independent readiness gate for false-negative rescue. rollup_membership_examples emits complete impression groups, including returned and explicit omitted-positive rows, bounded by finalization time so a later correction cannot leak into an earlier evaluation window. Its query key is namespace-scoped but remains linkable and must not be treated as anonymized.

Full API

Technical Decisions

Ecosystem

This repo is the engine. The rest of the stack builds on it:

Roadmap

• V0 — Embedded engine, core memory model (record, recall, relate, consolidate, decay)
• V1 — Replication log, CRDT-based sync between devices
• V2 — Conflict resolution with human-in-the-loop
• V3 — Proactive cognition loop, pattern detection, trigger system
• V4 — Sessions, temporal awareness, cross-domain pattern mining, entity profiles
• V5 — Multi-agent shared memory, federated learning across users

Worked example: Wirecard (RFC 008 substrate — with honest limits)

For nearly a decade, Wirecard's filings and EY's audit attested to €1.9B in Philippine escrow accounts. In June 2020 both banks and the central bank formally denied the accounts existed.

When the source_lineage fields are hand-populated — EY as [wirecard, ey] to capture audit dependence on Wirecard-provided documents, BSP as [bsp, bpi, bdo] to capture restatement of the commercial banks — RFC 008's ⊕ discounts the dependent claims, and the contest operator's temporal split distinguishes present-tense contradictions from historical state changes. On this hand-populated data, the substrate produces useful annotations.

Honest limits (surfaced by Phase 2 empirical testing, Apr 2026):

• On naturalistic evidence where a real agent populates the fields, the substrate's gates don't reliably fire. Cases B and C of the Phase 2 eval need an extractor/canonicalizer (not yet built) to work; Case A exposed that ⊕ is mathematically incapable of flipping decisions at realistic N, regardless of coefficient tuning.
• Current claim: structured schema for evidence provenance/temporal/conflict annotation, useful for audit and inspection. The dependence-discount operator works on curated inputs but needs replacement before it can drive decisions.
• Not a current claim: "decision-improvement substrate for AGI-capable agents." That framing is withdrawn pending RFC 009.

See docs/showcase/wirecard.md for the full walkthrough including the Phase 2 negative result and the gold-state ablation that partitioned operator failure from extraction failure. Run the hand-populated demonstration directly:

cargo run --example showcase_wirecard

Research & Publications

📄 Skill as Memory, Not Document (May 2026)

Sarkar, P. (2026). Skill as Memory, Not Document: A Database-Native Substrate for Agent Skill Catalogs. Zenodo.

A measurement paper at 5K-skill scale: token cost vs filesystem catalogs (with the honest 1.49× ablation), retrieval latency (87.3 ms p50), and invalid-skill admission (0% YantrikDB vs 97% document-only baseline). Reproducible scripts + raw CSVs at yantrikdb-server/benchmarks/skill_recall/. Companion blog: yantrikdb.com/papers/skill-substrate.

Earlier work

• U.S. Patent Application 19/573,392 (March 2026): "Cognitive Memory Database System with Relevance-Conditioned Scoring and Autonomous Knowledge Management"
• Zenodo (software): YantrikDB: A Cognitive Memory Engine for Persistent AI Systems

Author

Pranab Sarkar — ORCID · LinkedIn · developer@pranab.co.in

License

Apache-2.0. See LICENSE for the full text.

The MCP server is MIT-licensed.