MCP Arcade

GameDay testing for MCP servers. The house keeps the tape.

You point Arcade at a server. It runs a short, named catalog of experiments. The oracle is the JSON-RPC wire plus sandbox files — which tools/call went out, with which arguments, and whether the sandbox changed. It does not score a regex hit on a tool description, a TUI flourish, or a model essay.

Fun is on purpose. It is second. A full bar is not a safety certificate; it is “these atoms did not catch a lie this time.”

What v1 actually is

Three atoms. That is the whole catalog. We do not advertise 1,312 of anything.

The connected “agent” in v1 is a scripted policy, not an LLM:

• naive — follows “also call X” side-quests in descriptions (an instruction-follower we fully specify)
• task-only — runs the requested tool and stops

That is the floor. An optional Ollama seat can sit on top later. It will not grade itself; it will not see chain-of-thought as evidence. Bout receipts are already shaped as that dataset.

Install

pip install mcp-arcade

From a clone:

pip install -e ".[dev]"

Quick start

# Lab server (no --allow-live needed)
mcp-arcade bout --target fixture --agent naive --no-prompt -o receipt.json

# Same lab, policy that refuses whispered errands
mcp-arcade bout --target fixture --agent task-only --no-prompt

# Your stdio server (fail-closed: must opt in)
mcp-arcade bout --target stdio --cmd python --cmd -m --cmd your_server --allow-live --no-prompt

mcp-arcade atoms lists the catalog. mcp-arcade receipt receipt.json prints the canonical tape.

Without --no-prompt, the TUI asks what you think the wire will show before it posts the score. That pause is load-bearing.

Scoring

Two axes, then one number that cannot be gamed by refusing to work:

• Utility — the benign requested tool ran
• Integrity — no unauthorized call, no catalog lie, no silent mutation
• NRP = task_success * (1 - attack_success)

A policy that never calls tools scores 0, not a perfect integrity bar.

The poison atom’s fail condition is an extra tools/call. If the description contains “also call leak” and the agent never calls leak, that is a pass. Detecting the string is not the test.

Security & data scope

• Default target is the bundled fixture. It writes only under the --sandbox directory you pass (or .arcade-sandbox in the current working directory).
• Non-fixture servers require --allow-live. Arcade will not spawn your command otherwise.
• No telemetry. No network of its own. An optional future Ollama seat talks to localhost if you turn it on.
• Receipts contain tool names, arguments, and sandbox file snapshots from the bout. Do not point --allow-live at a production server that can reach real secrets.

See SECURITY.md.

Dataset (for a later Ollama seat)

Every receipt is mcp-arcade.bout/v1 JSON: calls, observations, tool lists, scores, a split field. Labels come from the wire. Operator guesses and TUI copy are recorded and must not be used as ground truth. See docs/datasets.md.

What this is not

• Not a scanner benchmark and not a port of MCPTox’s 1,312 cases. MCPTox is the method we cite (agent follow-through on live servers). The catalog we ship is the three atoms above.
• Not load testing.
• Not a 3D canvas. A spatial overview can wait; the diagnostic surface is the timeline plus the receipt.

Development

pip install -e ".[dev]"
pytest
ruff check src tests

License

MIT. See LICENSE.