The Problem

Every LLM application that runs multi-turn conversations has a memory strategy — usually "keep everything in context and hope." Almost nobody measures what that strategy actually remembers after 50, 100, or 200 turns.

MemoryLens answers three questions:

• How much does an AI still recall after N conversation turns — and at what token cost?
• Which memory architecture (full history, RAG, tiered compression, entity store, knowledge graph…) retains facts most efficiently?
• When a user updates a fact ("I moved to Mumbai"), does the memory surface the new value, the stale one — or contradict itself with both?

Every core metric is content-based and deterministic: no API key, fully reproducible. Add any LLM key and a two-stage answer+judge pipeline measures what the model actually answers.

Install

pip install memorylens-bench

Python 3.10–3.13 · Linux, macOS, Windows (all tested in CI).

Quick Start

CLI — results in under a minute, no API key

memorylens                        # 100-turn benchmark, 3 backends
memorylens --seeds 5              # multi-seed: mean ± std across 5 personas
memorylens --scenario medical     # domain scenarios: default | edtech | support | medical
memorylens --backends naive rag_chunked cascading graph
memorylens --seeds 5 --fit-curves # fit Ebbinghaus + exponential forgetting curves
memorylens --llm --provider groq  # real answer+judge LLM evaluation

Python API

from memorylens import run_benchmark, results_to_display_dict

raw = run_benchmark(total_turns=100, backends=["naive", "rag", "cascading"])
results = results_to_display_dict(raw)
print(results["cascading"]["recall"])   # recall at each checkpoint

Dashboard and REST API

pip install "memorylens-bench[dashboard]"
streamlit run dashboard.py            # decay curves, run history, cost projection

pip install "memorylens-bench[server]"
uvicorn memorylens.api:app            # POST /v1/benchmarks → job id → poll results

Benchmark Results

All numbers below are reproduced from this exact code (v0.4.0) with: memorylens --seeds 5 --backends naive rag rag_chunked cascading summary entity graph faiss

Recall@T — 100 turns (mean ± std, n=5 personas)

Read this before quoting numbers. At 100 turns with 8 templated facts, every retrieval- and extraction-based backend saturates — the differentiation is token cost at equal recall (rag_chunked delivers the same recall as naive at ~1/21 the tokens) and naive's collapse once its context budget evicts early turns. Two structural caveats:

1. entity / graph exploit the fact templates. Facts are injected as "My X is Y", which the regex extractors match by construction. Their 100% shows structured stores never lose what they capture — it says nothing about free-form conversation. Harder paraphrased injections are on the roadmap.
2. summary is extractive in zero-key mode — it keeps fact-bearing lines verbatim, so substring recall is guaranteed. With an LLM key, compression becomes abstractive and honest decay appears.

Stress test — 200 turns

At 200 turns the bounded systems saturate and real decay separates the field (memorylens --turns 200 --checkpoints 10 50 100 150 200 --seeds 5 --backends ...):

The two production-realistic constraints fail in opposite ways: naive's fixed token budget evicts old turns wholesale, while rag_chunked's bounded FIFO vector index (200 chunks) silently drops the earliest facts even though each query costs only 70 tokens. Cascading survives because its cold tier compresses fact-bearing lines instead of discarding them.

Temporal drift and contradiction after fact updates

Two facts update mid-conversation (city at T=40, age at T=60). At T=100:

Drift is a retrieval-layer proxy (worst-case bound). For behavioral measurement — does the model answer with the stale value — run memorylens --llm.

How It Works

┌────────────────────────────────────────────────────────────────────────┐
│  LAYER 1 — SIMULATOR                                                    │
│  Injects facts at known turns, fires domain filler queries in between   │
│  Facts can be updated mid-conversation to test drift + contradiction    │
│                                                                         │
│  T=0  "My name is Arjun Sharma."                                        │
│  T=1  "My city is Bangalore."                                           │
│  T=40 "My city has changed to Mumbai."   ← update event                │
│  Scenarios: default (tech Q&A) · edtech · support · medical             │
└──────────────────────────────┬─────────────────────────────────────────┘
                               ▼
┌────────────────────────────────────────────────────────────────────────┐
│  LAYER 2 — MEMORY BACKENDS (8 implementations)                          │
│                                                                         │
│  naive        Full history, evict oldest at a token budget              │
│  rag          Embed every message, retrieve top-K (upper bound)         │
│  rag_chunked  Chunked + bounded FIFO index (production-realistic)       │
│  cascading    Hot/Warm/Cold tiers with Ebbinghaus temporal decay        │
│  summary      Rolling compression (LLM or extractive)                   │
│  entity       Structured key-value fact store                          │
│  graph        NetworkX knowledge graph, in-place fact updates           │
│  faiss        FAISS vector index (optional dependency)                  │
└──────────────────────────────┬─────────────────────────────────────────┘
                               ▼
┌────────────────────────────────────────────────────────────────────────┐
│  LAYER 3 — EVALUATOR (6 metrics, dual mode)                             │
│                                                                         │
│  Content mode (no API key): deterministic substring checks              │
│  LLM mode (any provider):   answer+judge pipeline — did the LLM         │
│                             actually answer correctly?                  │
└────────────────────────────────────────────────────────────────────────┘

The 6 Metrics

The 4 Temporal Decay Functions

The cascading backend's warm-tier scoring uses a pluggable forgetting curve — compare them with --decay:

--fit-curves fits both Ebbinghaus and exponential models to your measured Recall@T series and reports half-life, stability, and R².

LLM Provider Support

All content metrics work with no API key. Add any one key for the LLM answer+judge pass:

memorylens --list-providers
memorylens --llm                  # auto-detect
memorylens --llm --provider groq  # force one

How MemoryLens Compares

Most evaluation frameworks measure quality at a single point in time. MemoryLens measures how memory quality changes over conversation turns — a dimension the tools below don't cover (they solve different problems, and compose well with this one):

Details: docs/comparison-with-existing-tools.md

Project Structure

memorylens/              installable package
├── memory/              8 backends + decay functions (add yours here)
├── simulator/           facts, conversation generator, personas, scenario registry
├── evaluation/          metrics, benchmark runner, stats, experiment logger
├── utils/               local embeddings, LLM providers, SQLite run storage
├── api.py               FastAPI REST server
└── cli.py               `memorylens` command

tests/                   39 integration tests, no API key needed
dashboard.py             Streamlit dashboard (decay curves, run history, export)
docs/                    guides: adding a backend, comparisons, methodology

Contributing

A new memory backend is 3 methods and one registry line — the full guide with a worked example is in docs/adding-a-new-backend.md:

from memorylens.memory.base import BaseMemory

class YourMemory(BaseMemory):
    name = "your_backend"
    def add_message(self, role, content, turn): ...
    def get_context(self, query, current_turn): ...
    def reset(self): ...

New scenarios are a data file plus one registry entry. New metrics are plain functions.

git clone https://github.com/Neal006/memorylens && cd memorylens
pip install -e ".[server,dev]"
pytest tests -q          # all green before you start

Start here: good first issue · Guide: CONTRIBUTING.md · Plans: ROADMAP.md

Citation

@software{memorylens2026,
  author  = {Daftary, Neal},
  title   = {{MemoryLens}: A Temporal Decay Benchmark for {LLM} Memory Architectures},
  year    = {2026},
  url     = {https://github.com/Neal006/memorylens},
  version = {0.4.0}
}

License

MIT — free to use, modify, and distribute.