toolfit

toolfit finds the specific places your MCP server confuses models, rewrites the tool descriptions to fix them, and proves the fix with a before-and-after eval.

uvx toolfit scan "npx -y @modelcontextprotocol/server-github"     # free, static, seconds
uvx toolfit eval examples/toy_server.py --seeds 10 --fix --badge   # model-graded, minutes, your key

Every existing MCP grader stops at a number and a list of complaints. toolfit measures which tool pairs a model actually confuses, proposes a description rewrite for each failing tool, re-runs the same tasks against the rewrite, and reports the delta with a p-value — accepted and rejected. The rejected ones are what make the accepted ones believable.

What a run looks like

toolfit eval examples/toy_server.py --seeds 10 --fix --badge, model under test claude-sonnet-5 (full output: docs/examples/toy-server/). The toy server has two deliberately confusable pairs: create_task/update_task share the description "Add a new task.", and list_tasks/count_tasks share "Get tasks by status."

## Confusion Matrix

| Intended \ Called | count_tasks | create_reminder | create_task | list_tasks | update_task |
|---|---|---|---|---|---|
| count_tasks       | 0  | 0  | 0  | 10 | 0  |
| create_reminder   | 0  | 10 | 0  | 0  | 0  |
| create_task       | 0  | 0  | 10 | 0  | 0  |
| list_tasks        | 0  | 0  | 0  | 10 | 0  |
| update_task       | 0  | 0  | 0  | 0  | 10 |

## Pass Rates
- count_tasks: 0/10 (0%), 95% CI [0%, 28%]
- create_reminder: 8/10 (80%), 95% CI [49%, 94%]
- create_task: 9/10 (90%), 95% CI [60%, 98%]
- list_tasks: 10/10 (100%), 95% CI [72%, 100%]
- update_task: 10/10 (100%), 95% CI [72%, 100%]

## Proposed Fixes

### create_task — REJECTED
- Before: 'Add a new task.'
- After:  'Creates a brand-new task by specifying its title and priority, distinct from
           update_task (which modifies existing tasks) or create_reminder (...)'
- Pass rate: 9/10 → 7/10, p-value 1.0000
- Reason: rejected: made things worse

### count_tasks — REJECTED
- Before: 'Get tasks by status.'
- After:  'Return the number of tasks matching a given status, rather than the tasks
           themselves, by accepting a required status argument.'
- Pass rate: 0/10 → 0/10, p-value 1.0000
- Reason: rejected: no change

Three things this run shows, none of them flattering, all of them the point:

• The matrix names the real problem. count_tasks is called zero times out of ten; every request meant for it goes to list_tasks. The solvability check flagged the identical create_task/update_task descriptions on 14 of 20 tasks — yet Sonnet 5 still picked the right one every time, because it reads task_id in the schema. Descriptions aren't the whole story, and toolfit measures what the model does, not what the text says.
• The fix loop refused to claim a win. Three rewrites were proposed and measured; one made things worse, two changed nothing. They are printed anyway. Every failure here is at the argument level or in the generator's "count vs. list" phrasing — things a description rewrite can't fix — and the numbers say so instead of the tool saying so.
• Every number carries its uncertainty. 9/10 and 7/10 overlap almost entirely at n=10; the exact test gives p=1.0 for a change in the wrong direction. Nothing gets reported as a finding because it looked good once.

On real servers

Same command against three public servers, model under test claude-sonnet-5, 10 seeds per tool (full reports in docs/examples/):

The git and filesystem results share a cause that a description can't fix: the model takes a correct precondition step — stage before commit, check allowed directories before touching a path — and a single-call grader scores it as the wrong tool. Those numbers were 0.1.x.

0.2.0 grades the sequence. The model may make up to --max-steps (default 3) calls, each answered with a synthetic result, and the task passes if the intended tool is called correctly at any step. Same git server, same 10 seeds (docs/examples/mcp-server-git-multistep/):

- git_commit: 10/10 (100%), 95% CI [72%, 100%]        # was 4/10

## Preconditions (observed)
- git_add → git_commit: 7/10 trials
- git_status → git_commit: 1/10 trials

## Undeclared Preconditions
- git_commit: models call git_add first in 7/10 trials, but git_commit's
  description never mentions git_add

That last line is the deliverable. It is the dependency graph the model believes in, built from behaviour rather than declared by anyone, diffed against what your catalog says. You fix it one of two ways — state the precondition in git_commit's description, or make the tool stage for you — and re-run: --mutate/--fix report reached via an earlier call: before → after so either choice is verifiable. The confusion matrix still shows the first call, so 0.1.x matrices stay comparable; --max-steps 1 reproduces them exactly. Cost: roughly 2× the wall time on servers where the model actually chains.

Both fixes, measured on the same server at 20 seeds (docs/examples/mcp-server-git-precondition/):

--mutate "git_commit:Records staged changes ... call git_add first ..."
- Reached via an earlier call: 13/20 → 19/20

--mutate "git_commit:... Automatically stages all modified and new files ..."
- Reached via an earlier call: 13/20 → 0/20

Pass rate stayed 20/20 both ways, so the pass-rate verdict is "not significant" and the precondition line is the result. The second description is only honest if the server really auto-stages; the point is that either claim is now checkable in numbers. Re-measuring one tool this way doesn't need the whole catalog re-run: --only git_commit generates tasks for that tool alone while still offering the model all twelve. The same two mutations that way, 10 seeds, 36 s against 2551 s for the full run, with the p-value the precondition line now carries:

- Reached via an earlier call: 5/10 → 9/10 (two-sided p=0.1250; informational, not part of the verdict)
- Reached via an earlier call: 5/10 → 0/10 (two-sided p=0.0625; informational, not part of the verdict)

Same direction as the 20-seed run, but 10 paired trials can't get a 5-trial swing under 0.05. That is what --seeds 20 buys.

The filesystem server tells the other half of the story (docs/examples/server-filesystem-multistep/): 55% → 79%, but not uniformly. Tools that were losing to a precondition went to 5/5 (create_directory, get_file_info, move_file, directory_tree); tools failing on arguments stayed put (read_multiple_files 0/5), and the deprecated read_file stayed at 0. The graph shows list_allowed_directories feeding eleven tools and search_files feeding three as a lookup step. That's the grader telling precondition problems apart from description problems — which is what you need to know before rewriting anything.

Twelve rewrites were proposed for the filesystem server. Five improved the number (8→10, 6→8); none were accepted, because one Bonferroni correction across twelve proposals at n=10 sets α=0.004 and the report says so. Run --only list_directory --only move_file --fix-tool list_directory --fix-tool move_file --seeds 20 on the tools the matrix names, not --fix on the whole catalog at once. --only generates tasks just for those tools while still offering the model the whole catalog, so the re-measure costs minutes instead of the full run.

At the size teams actually ship. examples/ops_server.py is 49 tools across users, tickets, deployments, alerts, on-call, docs, flags and config, with planted problems listed in its docstring. Sonnet 5, 5 seeds, 48 min (docs/examples/ops-server/): 84%, no tool excluded. The declared precondition (promote_release says to run validate_release first) was observed 5/5 and correctly not flagged. The near-neighbour pairs mostly did not confuse Sonnet; what failed was arguments (create_flag 2/5 with every call routed right) and no-calls (snooze_alert 4/5 replied with a question instead), so the report now prints what the model said in a No-Call Replies section. The id-lookup preconditions planted there never fired: generated tasks carry the id, so the model has no reason to look it up. Precondition findings are about state (stage before commit), not id lookups.

With a model that actually gets confused. Llama 3.1 8B via OpenRouter on the toy server: 64%, count_tasks 0/10, 6 malformed tool-call payloads in 50 tasks. That run is why ambiguous tasks are now regenerated, why unparseable arguments count as an argument failure on the named tool rather than a no-call, and why fixer proposals are capped at 25 words. With those in, the fixer's rewrite of create_task helped every time it was measured (5/10 → 8/10, then 14/20 → 17/20 alone at 20 seeds) and was still rejected: p=0.23. A 15-point effect needs about 40 paired trials to clear 0.05, and toolfit says so rather than round up (docs/examples/toy-server-llama/).

Across ten models, not one. docs/models.md runs the same command over 10 models — 70B/32B/24B open weights through commercial small/mini tiers — on three servers, 10 seeds each. Zero malformed/duplicated-JSON trials across the 21 completed combinations: every model here forms JSON fine, so what's left is almost entirely which-tool and which-argument. The toy server's count_tasks/list_tasks pair (identical descriptions) fails 0/10 on every single model tested, best and worst alike — the cleanest evidence that a description gap doesn't shrink with model strength. The report now says this directly: a Failure Attribution section splits every failure into four buckets (description confusion, fixable arguments, model-mechanics no description can move, and correct deprecated-tool avoidance), plus a Mechanics Floor baseline, so you know before rewriting anything whether the catalog can actually fix what's failing.

Two commands, two budgets

They are deliberately separate. scan is the zero-config front door; on mature servers it finds little (1 finding across 166 tools on 15 public servers) because the bug it catches is the copy-paste class. The confusion is what eval is for.

Install

uvx toolfit --help          # zero-install, one-off
pipx install toolfit        # persistent / CI

Python 3.10+ — on an older interpreter (macOS system python3 is 3.9) pip only says No matching distribution found for toolfit; use uvx, or uv venv --python 3.12. Talks to servers over the MCP protocol via the official mcp SDK, so the server can be in any language. Agents: see AGENTS.md for the full operating manual.

Pointing it at a server

toolfit scan path/to/server.py                                  # run via `uv run`
toolfit scan "npx -y @modelcontextprotocol/server-filesystem ."   # any command line
toolfit scan https://your-host/mcp                              # Streamable HTTP

The subprocess gets your environment (minus toolfit's own *_API_KEYs, which a third-party server binary has no business seeing), so servers that read a token from GITHUB_TOKEN or STRIPE_SECRET_KEY work unchanged. toolfit never calls a tool — it only ever asks for the catalog (tools/list), then asks a model what it would call. Nothing touches your backend.

How the measurement works

The credibility problem with LLM-generated evals is circularity: if one model writes the task and also decides the expected answer, you are measuring agreement with that model. toolfit inverts it:

1. Sample a concrete, schema-valid argument set for one tool (gen/schema_sampler.py), honouring required, enums, formats, and nullables.
2. Ask a generator model to write the sentence a user would type that leads to exactly those arguments, without naming the tool. Ground truth is the sampled tuple, not the model's opinion.
3. Send that sentence plus the whole catalog to the model under test.
4. Grade structurally (grade/grader.py): right tool, and arguments equal after canonicalising dates, case, whitespace, and array order. No LLM judge, ever.

Guardrails: a second pass checks each task for tool-name leakage and for solvability against the catalog; both are reported as warnings, never silently dropped. Tools whose schema the sampler can't handle are excluded and listed — a partial number is never printed as a complete one.

Mutation testing is the same grader run twice. --mutate 'tool:new description' re-runs a tool's own base tasks against a catalog where only that description is patched (protocol-level; your source is never touched) and compares pass rates on paired trials. --fix does the same with a proposed rewrite for every failing tool.

Significance is an exact one-sided McNemar test on the discordant pairs, Bonferroni-corrected across everything re-measured in one run. It is deterministic and honest at small n: with 5 seeds the smallest attainable p-value is 1/32, so --seeds 10 is the practical floor for a verdict — the CLI warns if you go lower. Every rate carries n and a Wilson 95% interval.

CI

- uses: sreshtalluri/toolfit@main
  with:
    server: "npx -y @modelcontextprotocol/server-github"
    eval: true                      # omit for the free scan only
    anthropic-api-key: ${{ secrets.ANTHROPIC_API_KEY }}

scan --strict exits 1 on any finding; eval --strict exits 1 if any tool's pass rate is below --strict-threshold (default 0.9). Tools excluded by a schema warning are named on stderr and not counted — they don't fail the gate, but you'll see them.

--badge writes toolfit-badge.svg with the pass rate (or the before→after delta of a single mutation or accepted fix), coloured by rate, with the model, generator, seed count, and a hash of the task suite embedded so the number is never separable from what produced it.

Models

--model picks the model under test and the provider is inferred from the name: claude* → Anthropic, gpt*/o* → OpenAI, vendor/model → OpenRouter. Keys come from ANTHROPIC_API_KEY, OPENAI_API_KEY, OPENROUTER_API_KEY. Task generation and fix proposals always use Anthropic, so that key is required for eval regardless of --model.

Data handling. The tool catalog and the generated task text are sent to whichever provider you configure, with your key. toolfit itself stores nothing and phones nowhere. If your server is internal, that is the one place its schema leaves your machine.

Landscape

toolfit is not a model leaderboard and does not test for prompt injection. One server per run.

Development

uv sync --extra dev
uv run pytest -q
uv run toolfit scan examples/toy_server.py

examples/toy_server.py has two deliberately confusable tool pairs — create_task/update_task share a description, list_tasks/count_tasks share a vague one — so the fix loop has something real to find. Design history and the methodology decisions behind every number are in docs/designs/toolfit-v0-scope.md.

MIT.