Bench'd Harness

The neutral benchmark harness for AI memory systems. Every score is independently run, cryptographically signed, and verifiable by anyone.

Leaderboard | Docs | Methodology | Submit Results

Quick Start

pip install benchd-harness

# Generate signing keys
benchd keys generate --out ./keys

# Set your LLM API key (for the judge)
export OPENROUTER_API_KEY=sk-or-...

# Run LongMemEval against your MCP-compatible memory system
benchd run -a mcp -b longmemeval-v1 --judge --key ./keys/private.key \
  --adapter-config '{"endpoint": "http://localhost:3000/mcp"}'

# Submit results to the leaderboard
benchd submit ./runs/run_xxx/manifest.signed.json

MCP Systems: Zero-Code Testing

If your memory system exposes an MCP server with ingest and query tools, you don't need to write any adapter code:

benchd run -a mcp -b longmemeval-v1 --judge \
  --adapter-config '{"endpoint": "http://localhost:3000/mcp"}'

The MCP adapter auto-discovers your tools and maps them to Bench'd's interface.

Available Benchmarks

Built-in Adapters

Writing a Custom Adapter

from benchd_harness.adapters.base import BaseAdapter

class MyAdapter(BaseAdapter):
    @property
    def name(self) -> str:
        return "my-system"

    def setup(self) -> None:
        self.client = MyMemoryClient()

    def ingest(self, turns: list[dict]) -> None:
        for turn in turns:
            self.client.add(role=turn["role"], content=turn["content"])

    def recall(self, query: str) -> str:
        return self.client.search(query).text

    def reset(self) -> None:
        self.client.clear()

Register in benchd_harness/adapters/__init__.py and run with benchd run -a my-system.

Commands

Signing & Verification

Every run produces an Ed25519-signed manifest containing all inputs, outputs, scores, and failure traces. Anyone can verify:

benchd verify ./runs/run_xxx/manifest.signed.json

Current Results (May 2026)

Full results at benchd.ai/leaderboard.

Links

• Website: benchd.ai
• Leaderboard: benchd.ai/leaderboard
• Docs: benchd.ai/docs
• Submit: benchd.ai/submit