NinjaEval MCP server

Lets your coding agent (Claude Code, Cursor, Copilot, any MCP client) test an AI change with NinjaEval before it calls the change done.

The agent edits your bot's system prompt, model or tools, then calls test_change. NinjaEval replays your test cases on production and on the change and grades every answer. It returns ready or blocked, with the cause of each regression and a fix that was re-run to check it. The agent applies the fix and tests again. Shipping stays with you: the server can't promote a version to production.

Set up

1. In NinjaEval, open ⚙ Workspace & API keys and create a key for the agent (e.g. coding-agent).

2. Add the server to your MCP client. For Claude Code:

   claude mcp add ninjaeval --env NINJAEVAL_API_KEY=efk_... -- uvx ninjaeval-mcp
   

   Any other client (mcpServers in its JSON config):

   { "mcpServers": { "ninjaeval": { "command": "uvx", "args": ["ninjaeval-mcp"],
                                    "env": { "NINJAEVAL_API_KEY": "efk_..." } } } }
   

   Or pip install ninjaeval-mcp and run ninjaeval-mcp.

It runs over stdio on your machine and opens no network port. It talks only to api.ninjaeval.com.

Tools

Each response has a next field telling the agent what to do. When the cause is TEST, the test itself is wrong, so the agent should tell you rather than bend the product to fit it.

Settings

Each change test runs your real app and an AI judge, which takes minutes and counts toward the workspace's daily limits. Endpoint projects call your service with the system prompt NinjaEval sends. If your service ignores it, test code changes by deploying them where the endpoint runs and passing code_sha.