MCP Tool Selection Benchmark

Benchmark how accurately different LLM models select the correct MCP tool given natural language instructions, powered by the GitHub Copilot SDK.

How It Works

1. Load a tool registry (tools.json) and ground-truth test suite (test_suite.json).
2. Generate 5 query variations per instruction at different ambiguity levels (explicit → misleading) using a Copilot SDK model call.
3. Evaluate each variation against every selected model — the model is presented with the tools and must pick one or more.
4. Score selections via exact-match and partial-credit against ground truth.
5. Report per-model accuracy, confusion matrices, and optional description suggestions in JSON and HTML.

Prerequisites

• Python ≥ 3.10
• GitHub Copilot CLI installed and in PATH
• An active Copilot subscription (each query counts as a premium request)
• Authenticated via copilot login, or GH_TOKEN / GITHUB_TOKEN env var

Installation

From PyPI:

pip install mcp-tool-selection-bench

Or for development:

cd mcp-tool-selection-bench
pip install -e ".[dev]"

Usage

Auto-generate a test suite (optional)

Don't have a test_suite.json? Generate one from your tool definitions:

mcp-bench generate --tools tools.json --output test_suite.json

This uses an LLM to create 3–5 test cases per tool (mix of single-tool and multi-tool, varying ambiguity). Review and edit the output before benchmarking.

Run a benchmark

mcp-bench run \
  --tools samples/tools.json \
  --test-suite samples/test_suite.json \
  --models gpt-5 claude-sonnet-4 \
  --output results.json

The run subcommand is the default — you can omit it for backward compatibility.

CLI Arguments (run)

Input Schemas

tools.json

[
  {
    "name": "search_issues",
    "description": "Search for issues in GitHub repositories",
    "parameters": {
      "type": "object",
      "properties": {
        "query": { "type": "string", "description": "Search query" }
      },
      "required": ["query"]
    },
    "metadata": { "category": "github", "tags": ["issues", "search"] }
  }
]

test_suite.json

Supports both single-tool and multi-tool test cases:

[
  {
    "instruction": "Find all open bugs in the react repo",
    "expected_tools": ["search_issues"]
  },
  {
    "instruction": "Find open bugs in react and check the CI logs",
    "expected_tools": ["search_issues", "get_job_logs"]
  }
]

For multi-tool cases, use expected_tools (ordered list). Single-tool cases using expected_tool (string) remain supported for backward compatibility.

Output

The report (results.json) contains:

• run_metadata — timestamp, models tested, query counts
• per_model_results — per-model accuracy (exact-match and partial-credit), per-instruction breakdown with each variation's selected tools and exact_match result
• summary — best model, accuracy ranking, best exact-match and partial-credit scores
• confusion_matrices — per-model confusion matrix (expected vs. selected tool counts)
• suggestions — tool-description improvement suggestions (when --suggest is used)

Scoring Metrics

• Exact-match accuracy: Full ordered sequence of selected tools must match expected tools exactly
• Partial-credit accuracy: Ordered prefix matching — counts how many tools in sequence match from the start (e.g., expected [A, B, C], selected [A, B, X] → 2/3 credit)

HTML Report

Use --html report.html to generate a visual report with:

• Model ranking table
• Confusion-matrix heatmaps (green = correct, red = misselected)
• Description improvement suggestions (if --suggest was used)

Regression Tracking (Diff)

Compare two benchmark runs to see what improved or regressed:

mcp-bench diff baseline.json current.json
mcp-bench diff baseline.json current.json --html diff.html
mcp-bench diff baseline.json current.json --fail-under 0.05  # fail if any model regressed >5%

The diff output shows per-model and per-instruction accuracy changes with ↑/↓/= indicators. Changes beyond ±5% are flagged as improved (green) or regressed (red).

For MCP Server Maintainers

Want to benchmark tool selection accuracy in your MCP server's CI? Add a workflow that runs on every tool change:

1. Add tools.json and test_suite.json to your repo (see Input Schemas)
2. Copy the template workflow from examples/mcp-server-workflow.yml into .github/workflows/
3. Create a GH_TOKEN repository secret with a Copilot-licensed PAT
4. Customize the model list, paths, and thresholds in the workflow

The workflow will:

• Run the benchmark whenever tool files change
• Automatically diff against the previous run's results
• Fail the build if accuracy drops below the configured threshold
• Upload JSON + HTML reports as GitHub Actions artifacts

Running Tests

pytest tests/

Project Structure

mcp-tool-selection-bench/
├── pyproject.toml
├── README.md
├── .github/workflows/
│   ├── ci.yml              # CI pipeline: test & build on every push/PR
│   └── publish.yml         # Publish to PyPI on GitHub Release
├── examples/
│   └── mcp-server-workflow.yml  # Template workflow for MCP server repos
├── src/mcp_bench/
│   ├── cli.py              # CLI entry point (run + diff + generate subcommands)
│   ├── models.py           # Pydantic data models
│   ├── prompts.py          # All prompt templates (centralised)
│   ├── generator.py        # Auto-generate test suites from tool definitions
│   ├── query_generator.py  # Generate query variations via Copilot SDK
│   ├── evaluator.py        # Run queries against models
│   ├── scorer.py           # Score, rank, and build confusion matrices
│   ├── advisor.py          # Description improvement suggestions
│   ├── diff.py             # Regression tracking (diff two runs)
│   ├── visualize.py        # HTML report with heatmaps & diff views
│   └── report.py           # Write JSON report
├── samples/
│   ├── tools.json          # Example tool registry (8 tools)
│   └── test_suite.json     # Example test suite (19 instructions)
└── tests/

CI

Every push to master and every pull request triggers the CI pipeline (.github/workflows/ci.yml):

1. Test — runs pytest across Python 3.10, 3.12, and 3.13
2. Build — builds sdist + wheel and uploads as a GitHub Actions artifact

Publishing

Creating a GitHub Release triggers .github/workflows/publish.yml, which builds and publishes to PyPI via Trusted Publishers (OIDC — no API tokens needed).

# Create a release via CLI
gh release create v0.2.0 --generate-notes

License

MIT