munch-bench

Retrieval + Inference benchmark for LLM-powered codebase Q&A.

munch-bench measures how well different LLMs answer questions about real codebases when given retrieval-augmented context from jCodeMunch.

What it measures

Corpus

110 questions across 11 repos spanning Python, JavaScript, TypeScript, Go, Rust, and Java:

Questions are categorized by difficulty (easy/medium/hard) and type (api/architecture/debugging/refactoring).

Quick start

pip install munch-bench

Prerequisites

1. Index the repos you want to benchmark against using jCodeMunch:

   jcodemunch-mcp index pallets/flask
   jcodemunch-mcp index tiangolo/fastapi
   # ... etc
   

2. Set API keys for your chosen provider:

   export GROQ_API_KEY=gsk_...        # for Groq
   export OPENAI_API_KEY=sk-...       # for OpenAI
   export ANTHROPIC_API_KEY=sk-ant-...  # for Anthropic
   

Run a benchmark

# Run with Groq (default — fastest + cheapest)
munch-bench run --provider groq

# Run with a specific model
munch-bench run --provider groq --model llama-3.3-70b-versatile

# Run with OpenAI
munch-bench run --provider openai --model gpt-4o-mini

# Run with Anthropic
munch-bench run --provider anthropic --model claude-sonnet-4-6

# Filter to specific repos or difficulty
munch-bench run --provider groq --repo pallets/flask --difficulty hard

# Custom token budget for retrieval
munch-bench run --provider groq --token-budget 16000 -v

Results are saved to results/<provider>_<model>_<date>.json.

Compare runs and generate leaderboard

# Compare multiple runs
munch-bench compare results/*.json -o leaderboard.html

# View corpus statistics
munch-bench corpus-stats

Leaderboard

The leaderboard is a static HTML page with interactive Chart.js visualizations, deployed to GitHub Pages on every push to main.

Architecture

munch-bench/
  corpus/              # YAML question files (one per repo)
  src/munch_bench/
    cli.py             # CLI entrypoint: run, compare, corpus-stats
    corpus.py          # YAML corpus loader + filtering
    retrieval.py       # jCodeMunch retrieval wrapper
    inference.py       # Provider dispatch (Groq, OpenAI, Anthropic)
    evaluate.py        # Metrics: P@k, recall, exact match, LLM judge
    runner.py          # Orchestrator with rich progress
    leaderboard.py     # Static HTML + Chart.js generator
  tests/
  results/             # JSON benchmark outputs (gitignored)

Adding questions

Create a YAML file in corpus/:

repo: owner/name
questions:
  - id: unique-id-001
    question: "How does X work?"
    ground_truth_answer: "X works by..."
    ground_truth_symbols: ["function_name", "ClassName"]
    difficulty: medium  # easy, medium, hard
    category: architecture  # api, architecture, debugging, refactoring
    tags: [optional, tags]

License

Apache 2.0

Powered by jCodeMunch + Groq