Murphy — Longevity Benchmark CLI

Evaluate any LLM on aging-biology tasks using an Estimathon-style benchmark. Models submit intervals [min, max] for numerical questions, receive only binary feedback (GOOD / BAD), and manage a shared submission budget across all problems. Non-numerical tasks (binary, multiclass, ternary, generation) are scored with standard accuracy / F1.

Install

pip install murthy-bench

Quick start

murthy

On first launch, Murphy runs a setup wizard to configure your API keys and verify dataset access. Keys are saved to ~/.longevity/config.json and masked on input.

The wizard walks through:

1. Anthropic API key — required for the chat interface and --provider anthropic runs
2. HuggingFace token — required for LongeBench dataset; wizard verifies live access
3. OpenAI API key — optional

LongeBench is a gated dataset. Before your HF token will work, visit huggingface.co/datasets/insilicomedicine/longebench and click Request access. Approval is usually instant. Then re-run /setup to re-verify.

You can re-run setup at any time from inside the chat:

/setup

Set keys manually

murthy config set anthropic.api_key  sk-ant-...
murthy config set hf.token           hf_...
murthy config list

Interactive chat

murthy

Type naturally — Claude calls the right tools. Type / to see all commands with Tab autocomplete.

Running benchmarks (CLI)

Full LongeBench — mixed mode (recommended)

murthy run \
  --model claude-sonnet-4-6 \
  --provider anthropic \
  --tasks longebench \
  --mode mixed \
  --limit 50

Estimathon only (numerical tasks)

murthy run \
  --model claude-sonnet-4-6 \
  --provider anthropic \
  --tasks sample \
  --mode estimathon \
  --think

Against the L-LLM endpoint

murthy run \
  --model longevity-llm \
  --provider endpoint \
  --endpoint https://saujlffcxf20v74m.us-east-2.aws.endpoints.huggingface.cloud \
  --api-key <hf-token> \
  --tasks longebench \
  --mode mixed \
  --limit 50

Estimathon rules

score = (10 + Σ floor(max/min) for GOOD final answers) × 2^(N − # good final answers)

• Only the last submission per problem counts
• Refining a GOOD interval is a voluntary bet — if the new interval misses, you lose that problem
• Feedback is binary only: GOOD or BAD — no "too high / too low"
• Default budget: floor(18/13 × N) slips across all N problems (matching real Estimathon's 18-slip / 13-problem ratio)
• Lower score is better

Refinement accuracy — the key signal: of all voluntary bets on GOOD intervals, what fraction paid off? Random guessing wins ~50%. Significantly above 50% means genuine biological reasoning.

Two-track scoring in mixed mode

Providers

Task sources

Output

Results written to results.jsonl. Fields include:

Estimathon track

• final_score — Estimathon score (lower is better)
• n_good_final / n_problems — problems solved
• slips_used / total_budget
• refinement_accuracy — fraction of refinement bets that succeeded
• slip_log — every submission with GOOD/BAD, width factor, score delta
• think — per-slip chain-of-thought trace (with --think)

One-shot track

• correct — boolean per task
• f1 — for generation tasks
• by_format — accuracy breakdown per format

Project structure

longivity_hack/
├── cli.py                  Typer entry point (run / chat / status / tasks / config / group / compare)
├── hf_llm_models.csv       300+ HuggingFace models for /model search and /add refresh
├── idea.md                 Benchmark design rationale
├── devlog.md               Development log
├── CLAUDE.md               Developer guide for contributors
└── benchmark/
    ├── chat.py             Interactive chat UI (first-run wizard, slash commands)
    ├── runner.py           Estimathon session, one-shot eval, run_mixed()
    ├── loader.py           Task loading — sample / LongeBench / local JSONL
    ├── client.py           Unified model client (all providers)
    ├── config.py           ~/.longevity/config.json
    ├── results.py          JSONL writer / reader
    └── model_manager.py    CSV-backed model browser (/model, /add, /batch)

For a local dev setup (cloning and running from source), see SETUP.md.