june-bench

A pip-installable, reproducible benchmark suite for memory / QA systems — June + pluggable competitors — over LoCoMo, LongMemEval, HotpotQA/2Wiki/MuSiQue, and FinanceBench, with the same data and the same scorer.

pip install june-bench
june-bench list
june-bench run --system echo --dataset smoke --split smoke    # offline, no key, no download

Reproduce the June vs Cognee head-to-head

One command runs both systems over the same HotpotQA open-pool, the same answer model, and the same judge, and prints a side-by-side with the metered API cost:

pip install "june-bench[cognee,june-api]"     # bundles cognee + fastembed
june-bench reproduce-h2h --key <YOUR_ACCESS_KEY> --questions 100

• Access key — June's endpoint is hardware-limited (not yet funded), so runs are key-gated. Request one at access@januraine.ai; the reply includes your key and this exact command.
• Same-embedder by default — Cognee automatically embeds with bge-large-en-v1.5, the commodity open model June's dense lane uses, so it's a same-embedder matched run out of the box (nothing to export). This embedder is a disclosed benchmark parameter, not June's moat; pass --embedder <id> to swap it.
• You bring an OpenRouter key (prompted) — it pays for both systems' gpt-4o answers (~$21 for the chain-of-thought tier at n=100); the host never holds or pays for it.
• Cognee runs locally (needs RAM + a one-time ~1.3 GB fastembed download); June answers over its endpoint. The command batches the pool upload, blocks the $90 Opus-on-Cognee path, and meters real cost.

june-bench reproduce runs the June-only HotpotQA number the same way; reproduce-retrieval scores June's recall@k/nDCG/MRR. All three are plain-language and need no JUNE_BENCH_* env vars.

A benchmark is run(system, dataset) → records → score. Two typed ports are the only extension points:

• System — the thing benchmarked. JuneApiSystem (default; a thin HTTP client to June's /v1/answer, so no June source is shipped), JuneLocalSystem ([june-local] extra; a source-protected compiled wheel), CogneeSystem ([cognee] extra), or any future system as one adapter.
• Dataset — what it runs on. The four benchmarks behind a registry.

The scorer is the canonical SQuAD/HotpotQA EM/F1 + selective-accuracy/coverage/cost — Cognee-comparable. Tiny smoke fixtures ship in the wheel (offline wiring proof); full splits are fetched, sha-verified, from a pinned release. No score is ever baked into the package — every result row records dataset + scorer + system + model + cost, so a published number is reproducible by a stranger, with the exact command above.

Tokens saved by a real coding agent — june-bench tokens-saved

June's in-product receipts can only measure June's own chat. For Claude Code, Codex and any MCP client, June never sees the bill — so the only honest per-agent number comes from running the agent itself, twice, and reading its own session accounting:

pip install "june-bench[agents]"              # june-mcp for the with-June arm
june-bench tokens-saved --harness claude-code --june-url http://localhost:8000 --key <KEY> --runs 3
june-bench tokens-saved --harness both --noise 300 --mcp-readonly --out results/tokens-saved   # Claude Code + Codex, bigger repo

What it does: materialises a fixed corpus (a fictional company's internal docs, 60 files) into a scratch workspace, ingests the same corpus into a fresh June canvas, and runs every task in the set on the real agent with june-mcp attached and without it — same workspace, same prompt, N runs each. Usage is read from the harness's own report (Claude Code's JSON result, Codex's turn.completed event): prompt tokens fed = input + cache read + cache creation, every turn. Two single-call baselines run beside them when you give a model key: naive-read v1 k=3 (the same grep-shaped procedure June's in-product A/B uses) and cited-read (the task's gold documents in full — the baseline most flattering to the agent). Money is printed only with its provenance (OpenRouter credits delta, or the harness's own list-price figure); the report carries a control line (the fixed overhead of each arm, including June's tool manifest), a correct line per arm, June's own receipts as a cross-check, and every unmeasured run by name. Nothing is estimated or extrapolated; a fake harness (--harness fake) exists for wiring only and stamps its report NOT A MEASUREMENT.

Task classes: doc-lookup, multi-doc, multi-turn (the second turn is where an agent re-reads), past-session (the record is a transcript file), and unfiled-session (a decision June captured that was never written to the repository — without June the agent has nothing to read).

What the first live runs showed (Claude Code 2.1, September 2026, smoke-sized): on a small, grep-able folder of markdown files the agent's own Grep+Read is already a precise retriever, and June's tool manifest (~6k tokens read-only, ~10k with writes, per turn) outweighs what June serves — the with-June arm fed the model more. On unfiled-session tasks the without-June agent burned ~48k tokens searching and answered 0/2; with June it answered 2/2 for fewer tokens. Read the correct line with the ratio, always. Both are in the report; neither is hidden.

Does an agent pick the right tool? — tool-surface

pip install "june-bench[tool-surface]"
python -m june_bench.tool_surface run --arm oracle --profile compact --out results/ts-oracle      # no model: proves the suite
ENABLE_TOOL_SEARCH=false python -m june_bench.tool_surface run --arm claude-code --model claude-sonnet-5 \
    --profile compact --runs 3 --out results/ts-claude-code
python -m june_bench.tool_surface report --md results/ts-*/results.jsonl

124 scenarios against June's MCP connector, judged from the tool-call log and the engine's state afterwards. The run checks the engine first and stops rather than scoring an outage; --resume picks up where a run stopped. Point it at a June engine with --engine / --key (or JUNE_BASE_URL / JUNE_API_KEY).

Honest-measurement notes

Both systems get the same documents, questions, gold answers, scorer, and embedder. Databases are reset before runs so results are never contaminated by prior state. Costs are measured from provider billing deltas, not estimated.

Protocol notes (read before comparing numbers)

june-bench runs a matched-pair protocol: identical evidence pool, answer model, and judge for every system, scored with EM/F1 plus a fixed LLM judge. Two deliberate differences from the official benchmark settings: the default mode pools QA over the corpus (the official LoCoMo/LongMemEval settings are per-conversation), and default runs use 100-question slices. This makes results directly comparable between systems run here — and NOT comparable to published leaderboard numbers, which use different protocols. Compare systems, not leaderboards.

Note on difficulty: pooling is the harder direction. The official settings give each question its own haystack (e.g. ~40 sessions in LongMemEval_S); the pool is the union of every conversation in the run, so each question faces strictly more distractors — including cross-conversation confusables the official design never tests. Both systems face the same pool.

Dataset licenses

Full splits are fetched from their official sources, sha-verified (see june-bench fetch). The small bundled reproduce/smoke fixtures are subsets of HotpotQA (CC BY-SA 4.0), LongMemEval (MIT), LoCoMo (CC BY-NC 4.0), and FinanceBench (CC BY-NC 4.0) — see DATA_LICENSES.md for attribution and modification notes.

Links

• Junê: https://june.januraine.ai
• Published results: https://june.januraine.ai (benchmarks section)
• Releases (desktop apps): https://github.com/Junemind/June_releases

Serving platform matters (measured, July 2026)

The answer model's serving platform is part of the experiment, and June is the system honest enough to show it. June answers only what its evidence supports and refuses the rest — it does not gamble. An aggregator (OpenRouter) routes each request to an unpinned, changing provider mix, and when that serving drifts conservative, June's honest refusals rise; guess-style systems have no refusal channel, so the same drift hides inside silently-changed guesses instead.

Measured on an identical engine and identical questions (2026-07-27):

Use OpenRouter for one-key convenience and real-time cost metering. For accuracy-representative or publishable numbers, choose a direct platform in the menu — every result stamps the platform it ran on (As-run:), so numbers from different serving paths are never conflated.