mcp-trust-check

One GitHub Action that runs the full MCP trust & safety trilogy — mcp-doctor, mcp-fuzz, mcp-reality-check — against an MCP server and posts one combined score, instead of three separate installs and three separate reports. Also a Python package (pip install mcp-trust-check) doing the same job for a live agent session instead of a CI run — see Live session wrapper below.

Each of the three tools already works standalone and does something the other two structurally can't:

• mcp-doctor reads the server's source and checks spec conformance/security/documentation — no server execution needed.
• mcp-fuzz actually launches the server and calls every tool with schema-derived inputs, checking crash resilience, latency, response size, and (opt-in) concurrency and resource-lifecycle correctness.
• mcp-reality-check also launches the server, but checks whether a tool's successful response is actually trustworthy — no disguised refusals, no empty content, no output that quietly ignores its own schema.

None of them use an LLM judge anywhere — every check here is deterministic and reproducible, same as the tools it wraps.

Demo

35 seconds, live run against the official MCP reference server — real output, no cherry-picking: 86%/B code quality, 100%/A crash resilience, 100%/A output fidelity, 95%/A combined.

Use

Static-only — zero extra config, works on any checked-out repo:

- uses: actions/checkout@v4
- uses: vishalhabib99/mcp-trust-check@v1
  with:
    path: .

Full trilogy — also add run with the command that actually starts your server over stdio:

- uses: actions/checkout@v4
- uses: vishalhabib99/mcp-trust-check@v1
  with:
    path: .
    run: "python server.py"
    # or: run: "npx -y some-mcp-server"
    fail-under: 80

Remote server — if your server ships over Streamable HTTP (a hosted/SaaS MCP server, or one your workflow starts in an earlier step), pass url instead of run:

- uses: actions/checkout@v4
- uses: vishalhabib99/mcp-trust-check@v1
  with:
    path: .
    url: "https://example.com/mcp"
    headers: |
      Authorization=Bearer ${{ secrets.MCP_TOKEN }}
    fail-under: 80

Set run or url, not both — the Action fails fast if it gets both. If both are left empty, only the mcp-doctor static check runs — the runtime checks are skipped and the report says so explicitly, not silently dropped. This is a real tradeoff, not a default to route around: mcp-doctor can statically analyze any checked-out repo path with no configuration, but mcp-fuzz and mcp-reality-check have to actually launch (or connect to) your server, and there's no way to derive a correct launch command from a repo checkout in general — you have to supply it.

Inputs

Input	Default	Meaning
path	.	Path to statically audit with mcp-doctor.
run	""	Command that launches the server over stdio, for the mcp-fuzz/mcp-reality-check runtime checks. Empty = runtime checks skipped.
url	""	URL of an already-running server to test over Streamable HTTP instead of launching one with run. Set one or the other.
headers	""	HTTP headers for url, one KEY=VALUE per line (values may contain spaces — e.g. Authorization=Bearer ...). Pass tokens from secrets.
env	""	Space-separated KEY=VALUE pairs passed through to the launched server (many real servers need an API key to start at all).
include-destructive	false	Also let mcp-fuzz and mcp-reality-check test tools without readOnlyHint: true. Only turn this on against a server you're confident is safe to call blindly — see mcp-fuzz's README Safety section before using it.
fail-under	0	Fail the workflow if the combined score is below this percent. 0 disables gating.
fail-on	never	Fail the workflow on the release decision: block fails on BLOCK, fix fails on FIX-FIRST or BLOCK, review fails unless it's SHIP with HIGH confidence, never disables it.
comment	true	Post the combined report as a PR comment.

Outputs

doctor-score / doctor-grade, fuzz-score / fuzz-grade, reality-score / reality-grade (empty if neither run nor url was set), combined-score / combined-grade, and decision (SHIP, FIX-FIRST, or BLOCK) with blocker-count / fix-count, confidence (HIGH, MEDIUM, or LOW), needs-human-review (true/false) and coverage-percent.

Release decision: SHIP, FIX-FIRST, or BLOCK

A score answers "how good is it on average?" The question a release actually asks is "can this go out?", and an average is the wrong tool for that. A server that crashes on one bad input, or tells an agent a failed call worked, can still average an A.

So next to the score, the report gives a decision, built from a short list of explicit rules over the raw findings. The worst finding wins, and every rule that fires is listed as a reason:

Decision	Fires when
⛔ BLOCK	mcp-doctor security error · the server crashed (process died) on any fuzz input · the server never connected, or the run ended early · any tool returned a disguised refusal (an agent reads it as success and acts on it)
🟡 FIX-FIRST	mcp-doctor security warning or spec/documentation error · fuzz timeouts · tools that misbehave under concurrent calls · resources still readable after delete, or orphaned after a parent delete · empty content, or output that breaks its own schema
✅ SHIP	none of the above

Quality warnings from mcp-doctor (a vague description, say) lower the score but never change the decision. If the runtime checks were skipped, the report says the decision covers static analysis only. Use fail-on: block to gate PRs on it. Same design as everything else here: no LLM, no weights, deterministic, and the rules live in one short file (decision.py) you can read in two minutes.

Checked against real runs, not just unit tests: the official memory reference server (runtime checks) gets SHIP, and the test fixture server, which has a tool that quietly refuses, gets BLOCK with that tool named.

Confidence: should you act on the decision?

The decision says what. Confidence says whether to act on it, the same split TypeSafe's Jev model makes for agent decisions. Jev learns its confidence from training data. Here it's plainer: how much of the server was actually exercised. mcp-fuzz only calls tools annotated read-only unless you pass include-destructive, so a SHIP can rest on a small slice of the server.

Confidence	When
HIGH	mcp-fuzz called at least 80% of the tools, and mcp-reality-check got a real response to check. Also every BLOCK: a crash seen once is a crash, and untested tools can only add to it.
MEDIUM	50–79% of tools called, or no successful response for mcp-reality-check to check
LOW	under 50% called, or the runtime checks didn't run

needs-human-review is true for everything except SHIP with HIGH confidence. Pair it with fail-on: review to auto-pass the clear cases and send the rest to a person.

It's a level with its reasons listed, never a number like 0.82. A calibrated probability would need past releases with known outcomes to fit against, and there aren't any, so a decimal here would claim a precision the rules don't have.

On five official or widely used servers, run locally with the defaults, all five came out SHIP. Confidence is what told them apart:

Server	Tools called	Decision
server-sequential-thinking	1 / 1	SHIP, HIGH
server-filesystem	10 / 14	SHIP, MEDIUM
server-everything	9 / 13	SHIP, MEDIUM
server-memory	3 / 9	SHIP, LOW
chrome-devtools-mcp (52.6K★)	8 / 30	SHIP, LOW

What the combined score means — and doesn't

The combined score is a plain, unweighted average of whichever of the three scores actually ran (one or three — never two, since fuzz and reality-check both run when run or url is set, or neither does). No tool is weighted more heavily than another; that would require a judgment call about which failure mode matters more that this project isn't going to make for you. Treat it as a single skim-friendly number for a PR check, not a substitute for reading the three sections underneath it — each retains its own real caveats (mcp-fuzz's crash-resilience score, for instance, deliberately doesn't grade whether a successful call's output was actually correct; that's what the reality-check section is for).

Apart from the decision rules above, which only read the three tools' existing findings, this repo contains no new detection logic of its own — it's orchestration over the three published tools, each independently dogfooded against 40+ real-world MCP servers (see each tool's own README for that history). If a check here is wrong, the bug is almost certainly in the underlying tool, not in the combining step.

Hosted servers: a metadata-only survey

The runtime checks are meant for servers you run yourself. For 10 public hosted MCP servers (Hugging Face, Microsoft Learn, AWS Knowledge, Cloudflare Docs, Context7, Exa, Svelte, Kiwi.com, DeepWiki, GitMCP), docs/hosted-survey-2026-09 records only what every client reads on connect, with no tool calls. It covers annotations, the context cost per tool, and what gets sent vs. what the model sees. Headline: 2 of 10 servers mark no tool as read-only, so under the spec's defaults, 8 tools that only read look destructive.

Live session wrapper

The Action above runs once, in CI. By now, each tool in the trilogy also has its own live counterpart, usable directly in an agent's own code: mcp-doctor's registration gate, mcp-fuzz's latency gate, mcp-reality-check's correctness gate. GuardedSession is the same idea as this Action, applied there: one wrapper around a real ClientSession that runs all three at their natural point, instead of three separate imports wired by hand.

from mcp import ClientSession
from mcp_trust_check import GuardedSession

session: ClientSession = ...  # your own, already-connected session
gs = GuardedSession(session)

await gs.list_tools()  # runs mcp-doctor's checks once, at discovery
for name, result in gs.registration_results.items():
    if result.flagged:
        ...  # missing/vague description, or a real annotation conflict

result = await gs.call_tool("some_tool", {"arg": "value"})
if result.outcome != "ok":
    ...  # crashed or timed out
elif result.flagged:
    ...  # slow/bloated relative to this tool's own history, a disguised
         # refusal, empty content, or a schema violation

A decision on every call: ACT, ESCALATE, or BLOCK

The release decision above runs once, in CI. Live, the question is narrower: what should the agent do with this one response? Every call_tool result carries its own decision, the per-call version of SHIP / FIX-FIRST / BLOCK:

result = await gs.call_tool("some_tool", {"arg": "value"})
if result.decision == "BLOCK":        # crash, timeout, or a disguised refusal
    ...                               # don't build on this response
elif not result.should_act:           # ESCALATE, or an unchecked tool
    ...                               # hand it to a person
print(result.decision, result.confidence, result.reasons)

Decision	When
BLOCK	the call crashed or timed out, or the response is a disguised refusal (an agent would read the failure as success and build on it)
ESCALATE	empty content, output that breaks its own schema, or a slow or large response for this tool
ACT	none of the above. An honest isError is an ACT: the tool told the truth, and the agent can handle it.

Confidence is how much of this call could actually be checked:

• HIGH: the tool is registered, and there are at least 3 earlier calls to compare speed and size against.
• MEDIUM: the tool has fewer than 3 earlier calls (no speed/size baseline yet), or mcp-doctor flagged it at registration.
• LOW: the tool never went through list_tools, so there was no schema to check its output against.

should_act is True for an ACT unless confidence is LOW. That's looser than the release decision on purpose. Live, MEDIUM mostly means "one of the first few calls to this tool," and escalating every early call would make the gate useless; the correctness checks run on every call either way. The rules are one pure function, decide_call, with no model and no network.

Dogfooded on the official @modelcontextprotocol/server-memory server: a read_graph before list_tools came back ACT/LOW. After registration, create_entities came back ACT/MEDIUM (first call). read_graph stayed MEDIUM until its third earlier call, then moved to HIGH.

One real call per call_tool, not three — the reason this exists as its own composed wrapper rather than "just call all three gates yourself": mcp_fuzz.gate.LatencyGate.timed_call and mcp_reality_check.gate.guarded_call each make their own real call to the tool. Calling both back to back would mean two real invocations per logical call — wasteful for an idempotent tool, actively wrong for a non-idempotent or destructive one. GuardedSession calls the tool exactly once and fans the single real response out to each sibling package's own pure, already-tested per-response functions instead — verified directly: a real test wraps the underlying session's call_tool with a call counter and asserts it fires exactly once per GuardedSession.call_tool.

Same scope discipline as its three parts: correctness and latency, not security — see each sibling's own README for why that's a deliberate boundary, not an oversight. Dogfooded live against the official @modelcontextprotocol/server-memory reference server — registration, a real create_entities call, and a real read_graph call, all clean.

License

MIT