Skip to main content

MemJudge

An open, vendor-neutral benchmark for AI memory systems.

MemJudge measures what a memory system (Mem0, Zep, Letta, Supermemory, …) actually returns when an agent asks it something, and what that costs in tokens and time.

  • Judge-only scoring. No answering LLM sits between the memory system and the judge, so a clever answer prompt can't hide weak retrieval. The judge reads the raw retrieved context.
  • One pinned judge. One exact model snapshot, prompt hash and reasoning setting for every published run. Changing the judge starts a new season; stored contexts are re-judged.
  • Token-budget aware. One fixed tokenizer, a hard per-query budget, and accuracy always reported against tokens returned. Dumping the whole conversation does not win.
  • Honest statistics. 95% confidence intervals (cluster bootstrap over conversations) and sample sizes for every number; failures count as wrong, not as missing.
  • The real costs. Ingestion time until data is searchable, retrieval latency p50/p95 and error/timeout rate. Anything a system doesn't report is shown as "not reported", never guessed.
  • Better ground truth. Abstention and adversarial questions are scored properly, knowledge-update questions report a stale-fact rate, suspect gold answers are flagged (not dropped), and reviewed corrections ship as versioned errata.
  • Verifiable. Results are canonicalised (RFC 8785) and signed (Ed25519); the dataset, connector code, judge prompt and raw per-question outputs are all hashed into the result.

Built and maintained by the founder of Recalld, a memory system — disclosed here and on every Recalld run. See docs/scoring.md for exactly how results are computed.

Quick start

pip install memjudge              # core + BM25 baseline; add extras for vendor SDKs (below)
export GEMINI_API_KEY=...        # or ANTHROPIC_API_KEY / OPENAI_API_KEY / OPENROUTER_API_KEY
memjudge run --dataset locomo --connector baselines/bm25 --judge google:<model> --smallest 1

This writes runs/<run-id>/ with result.json, raw questions-top10.jsonl, questions-top30.jsonl and conversations.jsonl, and report-top10.html and report-top30.html you can open in a browser. Local runs are for private testing and are never published to the leaderboard.

Command What it does
memjudge datasets / memjudge connectors List what you can run
memjudge run Benchmark a connector on a dataset (--resume <dir> continues a run)
memjudge report <run> [<run> …] HTML report; 2–4 runs gives a side-by-side comparison
memjudge rejudge <run> --judge … Re-score stored contexts with another judge
memjudge run … --reuse-ingestion <run> Rerun only retrieval on data kept with --keep-data
memjudge cleanup <run> Delete data a --keep-data run left in the memory system
memjudge estimate --dataset … Typical judge cost and upper bound, and what the memory system must ingest
memjudge calibrate labels.jsonl --judge … --judge … Compare candidate judges: agreement, false-accept rate, cost
memjudge crosscheck <run> … --judge … Re-judge a sample with a second judge; report disagreement per system
memjudge verify result.signed.json Check a published result's signature (and artifacts)
memjudge validate-dataset my.json Validate a custom dataset in the canonical format

API keys are read from the environment or --env-file; they are never written to disk or logs, and connectors only receive the keys they declare.

Connectors

Connector System Install extra Keys Notes
baselines/bm25 BM25 keyword search over raw turns – – Reference point; runs locally
mem0/platform Mem0 Platform (hosted) mem0 MEM0_API_KEY Local only: Mem0's terms forbid benchmarking, so the hosted runner does not offer it
mem0/oss Mem0 open source, in-process mem0-oss MEM0_OSS_OPENAI_API_KEY Full multi-signal setup; also run python -m spacy download en_core_web_sm. Mem0 OSS has no timestamp, so the conversation date goes into Mem0's extraction prompt through its prompt argument. Refuses to run while OPENROUTER_API_KEY is set (Mem0 would use it)
zep/graphiti Zep Graphiti open source, in-process graphiti GRAPHITI_OPENAI_API_KEY Needs FalkorDB on localhost:6379: docker run -p 6379:6379 falkordb/falkordb. Same LLM and embedder as Mem0 OSS; the hosted runner starts its own FalkorDB, empty for every run
zep/cloud Zep Cloud zep ZEP_API_KEY Local only: Zep's terms forbid publishing benchmark results without its written consent, so the hosted runner does not offer it
supermemory/cloud Supermemory (hosted) supermemory SUPERMEMORY_API_KEY One document per session; waits until Supermemory has indexed it and built its memories (dreaming: instant, one extra billed operation per session)
letta/cloud Letta (hosted) letta LETTA_API_KEY Agent-managed memory with openai/gpt-6-luna and openai/text-embedding-3-small (as Mem0 OSS), checked on the created agent; one LLM step per turn, billed by Letta, or by OpenAI when the account has its own OpenAI key connected in Letta. Returns memory blocks, Letta's conversation search, then all archival memory (Letta Cloud has no working archival search). Runs 3 conversations at a time to fit Letta's free plan (3 agents)
recalld/cloud Recalld (hosted) – RECALLD_API_KEY Vendor-maintained by the MemJudge maintainer. Sends every model explicitly (openai-gpt-6-luna, gemini-embedding-002-vertex) and stops if Recalld reports others
offline/replay Replay of recorded search results – – No API calls

Every connector names the models its system runs, or says the vendor picks them (ADR 0014); results show them. Every run scores two result sizes from one ingestion (ADR 0018): Top 10 and Top 30. Each question is retrieved twice, once for up to 10 items and once for up to 30 (Zep and Graphiti: 7 + 3 and 20 + 10 facts + entities; Letta: see above), and each size is judged and scored on its own. The harness cuts every context to the same token budget.

Install one extra, several ("memjudge[mem0,zep]") or all of them ("memjudge[all]"). Each SDK is pinned to the version its connector was written against. Every hosted connector also takes a base_url setting for self-hosted or staging servers; such runs are marked as custom endpoints. Mem0 OSS uses its own key name so it never receives the judge's key.

Datasets

Name Source Licence
locomo snap-research/locomo CC BY-NC 4.0
longmemeval-s LongMemEval (cleaned) MIT
longmemeval-oracle LongMemEval, evidence sessions only (quick checks) MIT

Datasets are not redistributed: each is downloaded from a pinned upstream revision and its SHA-256 is checked before use. Custom datasets work locally with --dataset my.json.

Docs

Contributing

See CONTRIBUTING.md. Security issues: SECURITY.md.

Licence

Code: Apache-2.0. Published results (scores, verdicts, metrics): CC BY 4.0. Dataset text inside per-question records (questions, answers, conversation excerpts) keeps its dataset's licence: LoCoMo is CC BY-NC 4.0 (non-commercial use only), LongMemEval is MIT. MemJudge is a free, non-commercial project. The MemJudge name and logo are not covered by these licences; see TRADEMARK.md.

Metadata

Release files for memjudge 0.1.0

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for memjudge 0.1.0
File Size Uploaded Provenance Actions
memjudge-0.1.0.tar.gz 587.5 kB With provenance Details for memjudge-0.1.0.tar.gz

Built distribution (wheel)

Table of built distributions (wheels) for memjudge 0.1.0
File Interpreter ABI Platform Actions
memjudge-0.1.0-py3-none-any.whl Python 3 none any Details for memjudge-0.1.0-py3-none-any.whl

Total release size: 705.0 kB

Release files / memjudge-0.1.0.tar.gz

Download URL memjudge-0.1.0.tar.gz
Size 587.5 kB
Tags Source
SHA-256 checksum
How to use checksums
3293c0e39a0587f0ac6aa2061d4927bbb0ed6912098a2966cc1b393ef3e509e5
BLAKE2b-256 checksum
How to use checksums
0da18ab65f8024e2dcef6042c3b62917d57706ee878718301c5230807055a0ab
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
Yes
Uploaded via twine/7.0.0 CPython/3.13.14

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Nov 2, 2026.

Transparency log

Release files / memjudge-0.1.0-py3-none-any.whl

Download URL memjudge-0.1.0-py3-none-any.whl
Size 117.5 kB
Tags Python 3
SHA-256 checksum
How to use checksums
e432bbcfde29b73a718a76a198acab05989e87adbad621f01e99aa648cfee369
BLAKE2b-256 checksum
How to use checksums
f1f5e380b0c5074e3eb9a12e1b860d5480d666ee77d3431d8f277d56e528ecb9
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
Yes
Uploaded via twine/7.0.0 CPython/3.13.14

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Nov 2, 2026.

Transparency log

Security for memjudge 0.1.0

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

Full provenance

Both files in this release have been uploaded with provenance metadata.

Select from 2 files

Attestation for memjudge-0.1.0.tar.gz

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Nov 2, 2026.

Transparency log

A note on package security

PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.

To learn more about protecting yourself from malicious actors, visit our package security documentation.

Release history Release notifications | RSS feed

This release

0.1.0 This release

2 release files Full provenance

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page