infolang-bench 0.1.0

Memory-benchmark harness (LoCoMo, LongMemEval, BEAM 1M) for InfoLang, built on the infolang Python SDK.

infolang-bench

Memory-benchmark harness for InfoLang. Runs LoCoMo, LongMemEval, and BEAM 1M through the official infolang SDK — same ingest, same questions, same optional LLM judge (see docs/JUDGING.md). Competitor lanes (Mem0 / Zep) share that harness (docs/FAIRNESS.md). Public claim formulas: docs/METHODOLOGY.md. Claims registry: docs/CLAIMS.md. Publishing / wandb: docs/PUBLISHING.md. Corrections: docs/CORRECTIONS.md.

Reproduce a published number (one command)

Published figures come from frozen recipes (il-bench repro). Knobs (top_k, candidate_pool, max_per_session, judge models, strict mode) are byte-identical to summary.meta.provenance.repro.

Hosted (public path)

The runtime Docker image is not public (it embeds core-IP packages and a private model checkpoint). Anyone can still reproduce recall quality, judged accuracy, and hosted-path latency against api.infolang.ai:

# Free API key (no credit card) from https://infolang.ai
export INFOLANG_API_KEY=il_live_...

# Option A — Docker (only Docker + git clone required)
docker compose run --rm bench repro locomo

# Option B — local pip (from source; not on PyPI yet)
git clone https://github.com/InfoLang-Inc/infolang-bench.git && cd infolang-bench
pip install -e .
il-bench repro locomo

Optional judged phase: also set OPENAI_API_KEY (or ANTHROPIC_API_KEY). Without those keys, repro still writes retrieval *.summary.json / *.per_q.json and prints why judging was skipped.

Hosted mode cannot reproduce self-hosted / on-box runtime latency.

Local runtime (evaluation / private image holders)

export IL_RUNTIME_IMAGE=your-registry/infolang-runtime:tag
export INFOLANG_BASE_URL=http://il-runtime:8766
export INFOLANG_DEV_KEY=devsecret:default
docker compose --profile local up -d il-runtime
docker compose --profile local run --rm \
  -e INFOLANG_BASE_URL=http://il-runtime:8766 \
  -e INFOLANG_DEV_KEY=devsecret:default \
  bench repro locomo --local

Smoke

il-bench repro locomo --limit 1
# or
docker compose run --rm bench repro locomo --limit 1

Artifacts land in results/repro_<benchmark>_<ts>.infolang.{summary,per_q,judged}.json.

Expected cost / wall-time / ranges

Benchmark	Recipe	Wall (full, hosted, approx.)	Judge tokens (if keys set)	Notes
LoCoMo	top_k=50, pool=500, max_per_session=6, judge gpt-4o-mini	tens of minutes	~answer+judge calls × questions	Report both overall_excl_adversarial and overall_incl_adversarial
LongMemEval (s)	same pool / session caps, top_k=50	tens of minutes–hours	same	Abstention types (*_abs) use abstain-is-correct
BEAM 1M	top_k=100, pool=500	hours (all chats)	same	Prefer --limit / --chats for smoke

Exact expected metric bands for a given recipe version are those printed in the matching published *.summary.json / *.judged.json under results/ (never hand-entered). provenance.repro.recipe_sha256 must match.

Install

infolang-bench isn't on PyPI yet — install from source:

git clone https://github.com/InfoLang-Inc/infolang-bench.git
cd infolang-bench
pip install -e .                              # pulls infolang>=0.2.0 (on PyPI)
pip install -e ".[tracking]"                  # optional W&B
pip install -e ".[backends]" -c constraints-backends.txt  # Mem0/Zep

For development:

pip install -e ".[dev]"

Connect to a runtime

Scenario	Configuration
Hosted repro (recommended)	INFOLANG_API_KEY=il_live_...
Local self-hosted	INFOLANG_DEV_KEY + runtime on :8766 (or INFOLANG_BASE_URL)
Custom host	INFOLANG_BASE_URL=...

il-bench health

Download datasets

Data is downloaded on demand (il-bench repro auto-downloads if missing). Checksums are verified before every run.

il-bench download locomo
il-bench download longmemeval --variant s
il-bench download beam --data-dir data/beam-1m

Advanced: il-bench run

For ad-hoc sweeps (non-published knobs), backends, or WandB streaming:

il-bench run locomo --limit 1 --backend infolang
il-bench run locomo --backend mem0 --judge --answer-model … --judge-model …
il-bench judge results/….per_q.json --answer-model … --judge-model …
il-bench compare results/*.judged.json --out compare.md

See docs/JUDGING.md, docs/FAIRNESS.md.

run options

Flag	Meaning
--backend	infolang | mem0 | zep
--limit N	Limit units / sample size
--top-k K	Recall depth (default: max K)
--pool N	Hybrid candidate pool (default 500)
--judge	Chain the answerer + judge
--no-strict	Allow hybrid→plain fallback (published runs stay strict)
--wandb	Stream to W&B (tracking extra); WANDB_MODE=offline for CI

Development

pip install -e ".[dev]"
ruff check .
mypy
pytest

Nightly compose smoke: .github/workflows/repro-nightly.yml.

Where we lose

Public honesty: categories where a competitor or alternative beats InfoLang. Same table style as the wins. Extend this section from same-harness artifacts when full judged runs land (CLAIMS.md).

Category	Who wins	Evidence	Why
LoCoMo multi-hop	Mem0 (published LLM-judge; same-harness pending)	Legacy internal signal: retrieval finds evidence turns, but we lack a chain-synthesis step	Closing the gap needs an LLM chain-synthesis pass on top of recall — not a bigger top_k
STS-B @ 192-byte budget	PCA-96	METHODOLOGY.md#size: 0.815 PCA vs 0.708 IL cosine	IL record is a memory primitive (reconstruct + structured query), not an STS-B embedding competitor — publish the trade, don't hide it

Do not cite quarantined workspace benchmarks/ proxy tables in public copy.

License

Apache-2.0.