agentcheckup

Diagnose your agent setup before your agent does.

Every other MCP tool looks at one server. None of them look at what happens when you load five.

$ agentcheckup

agentcheckup 0.4.0  |  3 server(s)  |  4 tool(s)  |  ~442 tokens per request

  Grade C    Score  79/100   1 error  2 warning  1 info

  CONFLICTS (1)
    AGT201  Tool name 'read_file' is exposed by both 'files' and 'git'.
            [files/read_file <-> git/read_file]
            fix: Hard collision: the client keeps one and the other is
                 unreachable. Rename one, or use a gateway that prefixes.

  RISKS (2)
    AGT101  Server 'github' runs unpinned npx package
            '@modelcontextprotocol/server-github'.
    AGT107  Server 'github' stores credential(s) inline:
            GITHUB_PERSONAL_ACCESS_TOKEN.

  NUDGES (1 across 1 rule(s), no score impact)
    AGT105  Server did not return a tool list  (1)
            github

The problem nobody is testing for

Two servers cannot see each other. Your model can. It gets a flat list of 47 tools from five servers and picks by description similarity — with no way to know that files.read_file and git.read_file are different things.

Three failure modes follow, and no existing tool catches them:

1. Hard collision. Two servers expose the same tool name. The client keeps one, the other is silently unreachable. You spend an afternoon debugging "why won't it read the file" when the answer is "it's calling the other one".

2. Soft collision. github.search and docs.search with 70% overlapping descriptions. Selection becomes a coin flip that looks exactly like model flakiness. It isn't — it's your manifest.

3. Context tax. An agent pays for the full tool manifest on every request. 47 tools with 200-character descriptions is thousands of tokens before the user says anything. Nobody budgets for it because nobody measures it.

agentcheckup measures all three, in one command, from the configs you already have.

What it checks

36 rules in total. Full reference: docs/rules.md.

Install

pip install agentcheckup
# or run without installing
uvx agentcheckup

Python 3.10+. Zero runtime dependencies — no SDK, no tokenizer, no embeddings, no network calls. It reads JSON, starts processes, prints text.

Usage

One command, no arguments

agentcheckup

Finds your configs automatically — 27 known locations across Claude Desktop, Claude Code (including its per-project mcpServers), Cursor, VS Code, Cline, Roo Code, Kilo Code, Windsurf, Zed, Codex CLI and Continue, plus project-local .mcp.json / .cursor/mcp.json / .vscode/mcp.json — starts each server, collects its tools, and reports.

Servers that fail to start are reported as AGT105 and skipped, not fatal. A broken server should not stop you auditing the other four.

Common flags

agentcheckup --no-spawn          # configs only, no processes started
agentcheckup --config path.json  # add a config the discovery missed
agentcheckup --no-discover       # audit only what you passed with --config
agentcheckup --timeout 40        # slow server? give it longer
agentcheckup --format json       # machine-readable
agentcheckup --fail-on warning   # CI gate

One server at a time

agentcheckup scan tools.json                    # {"tools": [...]}, {"result": {...}} or a log
agentcheckup scan --cmd "python my_server.py"   # live stdio handshake
agentcheckup config ~/.claude.json              # config rules only
agentcheckup rules                              # list every rule

As a GitHub Action

The fastest way to keep a manifest from regressing, and the version you pin is the version you run — the action installs from its own checkout.

- uses: wanjiaqing/agentcheckup@v0.3.0
  with:
    config: .mcp.json
    fail-on: warning

Full reference, including SARIF upload and score gates: docs/github-action.md.

Raw CLI in CI

- name: Diagnose agent setup
  run: |
    pip install agentcheckup
    agentcheckup --no-spawn --fail-on warning

For GitHub code scanning, emit SARIF:

- run: agentcheckup --format sarif --fail-on never > agentcheckup.sarif
- uses: github/codeql-action/upload-sarif@v4
  with:
    sarif_file: agentcheckup.sarif

One run can produce all three outputs:

agentcheckup --format text --json-out report.json --sarif-out report.sarif

Exit codes: 0 clean, 1 findings at or above --fail-on, 2 usage/load error.

How conflicts are detected

No embeddings, no model, no network — because a linter that needs a 200MB wheel is a linter that stops running on a plane.

Each tool pair across servers is scored on two signals:

• name overlap — normalised token Jaccard (read_files ≡ read_file)
• description overlap — token Jaccard over stopword-stripped, plural-stemmed words

A pair is flagged when the descriptions overlap ≥ 65% and the names share a head noun, or when the blended score clears 0.55. The shared-noun requirement is what keeps search_repositories and search_documents apart — without it the false-positive rate made the report untrustworthy, which is worse than no report.

It is a heuristic, and the docs say so. AGT203 is a warning, not a verdict.

How the score works

Most linters fail here: a rule that fires on 18 of 18 tools is a preference, not a defect, and charging one point each scores the official MCP servers 0/100. That number is useless, so the model is calibrated instead of invented:

Budget is 20 points per tool, so a 200-tool manifest is not punished for being large, and a 4-tool manifest with one hard collision is not let off easy.

Calibration against four real inputs:

Repeated findings are collapsed: 12 tools missing parameter descriptions print as one line naming all twelve, and info findings are summarised one line per rule. A report nobody can read is a report nobody acts on.

Design principles

• Zero runtime dependencies. A diagnostic that breaks because a transitive dep broke is worse than no diagnostic.
• Stable rule IDs. AGT201 never changes meaning. Suppressions and CI thresholds keep working across releases.
• Every finding has a fix. A rule that only says "this is bad" trains people to ignore the output.
• Degrade, don't die. A server that will not start is one finding, not an aborted run.
• No network calls, ever. Your config never leaves the machine. This is the one thing a commercial scanner structurally cannot promise.
• Heuristics are labelled as heuristics. info means "a human should decide". The score is a conversation starter, not a certification.

What this is not

• Not a sandbox. It tells you the manifest looks risky; it cannot stop the server.
• Not a runtime guard. Nothing here runs during an agent session.
• Not an MCP protocol validator. It checks quality and safety, not conformance — the official modelcontextprotocol/conformance suite does that.

Contributing

Adding a rule takes about ten minutes and one small file — see CONTRIBUTING.md. Rules and client adapters are the whole point of this project, so those PRs are the most welcome kind.

License

MIT