Munibench 0.1.0

A callback-based benchmark runner for evaluating memory systems

Munibench

Munibench is a small callback-based benchmark runner for memory systems. It keeps the benchmark implementation independent from the system being tested: you provide callbacks for resetting, feeding, and asking your system.

Install from PyPI

py -m pip install Munibench

Install locally

From this directory:

py -m pip install --editable .

From the thesis repository root:

.venv\Scripts\python.exe -m pip install --editable .\package

Download a dataset

from munibench import download_dataset, list_datasets

print(list_datasets())
path = download_dataset("longmemeval-s")
print(path)

The first download of longmemeval-s is approximately 277 MB. Hugging Face caches it locally, so subsequent calls reuse the downloaded file.

Run the benchmark

from munibench import benchmark

bench = benchmark(
    benchmarks="all",
    dataset="longmemeval-s",
    results_path="results/benchmark.jsonl",
)

bench.run(
    system=memory_system,
    reset=reset,
    feed=feed,
    ask=ask,
    new_instance=new_instance,
    feed_format="text",
    limit=5,
    verbose=True,
)

You can bypass downloading and use a local file instead:

bench = benchmark(data_path="path/to/longmemeval_s.json")

Do not pass both dataset and data_path.

Public API

• Benchmark and its backwards-compatible alias benchmark
• LongMemEval
• download_dataset()
• list_datasets()
• dataset_info()
• normalize_text() and is_correct()

The PyPI distribution is named Munibench; Python imports are lowercase:

import munibench