mcp-scoring-engine

A standalone scoring engine for evaluating the quality of Model Context Protocol (MCP) servers. Pure Python, no framework dependencies — just dataclasses, scoring logic, and network probes.

Used in production by MCP Scoreboard to grade thousands of MCP servers.

Installation

pip install mcp-scoring-engine

Quick Start

Score a server from its GitHub repo (static analysis)

from mcp_scoring_engine import ServerInfo, analyze_repo, compute_score

server = ServerInfo(
    name="my-mcp-server",
    description="A tool server for doing useful things",
    repo_url="https://github.com/owner/my-mcp-server",
)

static = analyze_repo(server.repo_url)
result = compute_score(server, static_result=static)

print(result.composite_score)  # 0–100
print(result.grade)            # "A+", "B", "D", etc.
print(result.score_type)       # "partial" (1 tier) or "full" (2+ tiers)

Probe a running server

from mcp_scoring_engine import (
    ServerInfo, probe_server, deep_probe_server, compute_score
)

# Fast health check (~10s) — connection, initialize, ping
fast = probe_server("https://my-server.example.com/mcp")
print(fast.is_reachable, fast.connection_ms)

# Deep protocol probe (~30s) — schema validation, error handling, fuzz testing
deep = deep_probe_server("https://my-server.example.com/mcp")
print(deep.tools_count, deep.schema_valid, deep.fuzz_score)

# Score with the probe results
server = ServerInfo(name="my-server", description="...", repo_url="...")
static = analyze_repo(server.repo_url)
result = compute_score(server, static_result=static, deep_probe=deep)
print(result.grade)

Probe a stdio server

from mcp_scoring_engine import probe_server_stdio, deep_probe_server_stdio

fast = probe_server_stdio(["npx", "-y", "@modelcontextprotocol/server-memory"])
deep = deep_probe_server_stdio(["python", "-m", "my_mcp_server"])

Classify a server

from mcp_scoring_engine import classify_server, ServerInfo

server = ServerInfo(
    name="stripe-mcp",
    description="MCP server for Stripe payment processing",
    repo_url="https://github.com/stripe/stripe-mcp",
)

category, targets = classify_server(server)
print(category)  # "finance"
print(targets)   # ["Stripe"]

Detect entry points for stdio servers

from mcp_scoring_engine import detect_entry_point, make_github_file_reader

# With a GitHubPublicClient (from your own GitHub API code)
file_reader = make_github_file_reader(client)
tree = client.get_tree()

result = detect_entry_point(tree, file_reader)
# {"language": "python", "run_cmd": ["python", "-m", "my_server"],
#  "install_cmd": "uv pip install -e .",
#  "source": "pyproject.toml [project.scripts]", "confidence": "high"}

Entry point detection parses build metadata to infer how to run an MCP server:

• Python: pyproject.toml scripts, setup.cfg/setup.py console_scripts, __main__.py
• Node: package.json bin field, scripts.start, main field

When called via analyze_repo(), detection piggybacks on the already-fetched file tree at zero extra API cost. The result is stored in StaticAnalysis.details["entry_point"].

Detect red flags

from mcp_scoring_engine import detect_flags, ServerInfo

server = ServerInfo(
    name="sketchy-server",
    description="A MCP server",
    repo_url="",
    remote_endpoint_url="http://localhost:3000/mcp",
)

flags = detect_flags(server)
for flag in flags:
    print(f"[{flag.severity}] {flag.label}: {flag.description}")
    # [critical] No Source Code: No repository URL or source link provided
    # [warning] Staging Artifact: Endpoint URL contains localhost or staging reference

Architecture

The engine evaluates servers across three data tiers:

The composite score is a weighted blend of five categories:

Score types:

• partial — Only 1 data tier available. Numeric score but no letter grade.
• full — 2+ data tiers. Graded A+ through F.

API Reference

Core

Probes

Types

All inputs and outputs are plain dataclasses:

• ServerInfo — Input server metadata (name, description, repo_url, etc.)
• ScoreResult — Complete scoring output (composite_score, grade, category scores, flags, badges)
• FastProbeResult — Health check results (is_reachable, timing)
• DeepProbeResult — Protocol compliance results (schema, error handling, fuzz)
• StaticAnalysis — Repo analysis results (7 metric scores + GitHub metadata)
• ReliabilityData — Pre-computed reliability metrics (uptime, latency)
• Flag — Red flag (key, severity, label, description)
• Badge — Display badge (key, label, level)

License

MIT