prompt-to-mcp 0.1.0

Describe the MCP server you want, point at an OpenAPI spec, and get a small curated FastMCP server (a few workflow-level tools, not hundreds of endpoint clones) plus a built-in eval harness that measures it.

prompt-to-mcp

Describe the MCP server you want, point at an OpenAPI spec, and get a small curated FastMCP server plus an eval harness that measures it.

Stripe API (2026-06-24)     587 operations   →   6 described tools
Tool-definition cost         ~462,960 tokens  →   ~956 tokens

The same spec can become a different server depending on what you ask for. You choose the shape; the model designs the tools; the output is editable code, not a black box.

Hero example

One command turns the 587-operation Stripe spec into a server scoped to a single job:

prompt-to-mcp plan specs/spec3.json \
    "a server for support agents to manage customers, invoices and refunds" \
    -o stripe-support.toolplan.yaml --max-tools 8 \
    --provider ollama --model gpt-oss:120b-cloud

The description drove the curation. The model kept the customer, invoice, and refund surface and left everything else (subscriptions, products, prices, balance, connect, webhooks, reporting) out:

Tool	Read-only	What it does
customer_lookup	yes	Find or retrieve customer information
customer_manage	no	Create, update, or delete a customer
invoice_lookup	yes	Find or retrieve invoice information
invoice_manage	no	Create, update, finalize, pay, void, or send an invoice
refund_lookup	yes	List or retrieve refunds
refund_manage	no	Create, update, or cancel a refund

Six tools, about 956 tokens of tool definitions, against 587 operations and roughly 462,960 tokens for the naive one-tool-per-endpoint conversion. Full artifacts: stripe-support.toolplan.yaml and generated/stripe-support/.

For comparison, curating the whole Stripe API with --auto (no description) yields a broader 11-tool general server, stripe.toolplan.yaml. Same spec, two servers: one shaped to a task, one shaped to the API.

Why context is the headline

Every request an agent makes carries the full tool list. Drawn to scale against a 32k-token model window, for the GitHub REST API (1,206 operations):

naive      1,206 tools │ ████████████████████████████████████████  414,263 tok   12.6× over ✗
32k window   (ceiling) │ ███                                         32,768 tok
distilled      9 tools │ ▏                                            1,758 tok   18.6× under ✓

The naive tool list alone is about 13 times larger than the entire window. It cannot fit before the agent reads a single word of the task. Providers differ in how they fail, and the harness records both:

Provider behavior on an oversized tool list	What the harness does
Strict (Anthropic, OpenAI): reject the request	records provider_error with the provider's message
Lenient (Ollama): silently truncate the list and run anyway	a pre-flight --model-context guard refuses the run, so a mangled tool list is never passed off as a result

The --auto mode and the benchmarks

--auto curates the whole API without a description, which is the mode every benchmark below used. It is a fair, description-free comparison: naive one-tool-per-endpoint against an automatically curated server, same agent, same scenarios.

API	Operations	Naive tool-def cost	Distilled	Ratio	Eval (qwen2.5:7b, live)
GitHub	1,206	~414,263 tok	9 tools / ~1,758 tok	236×	13/15 (87%)
Stripe	587	~462,960 tok	11 tools / ~1,793 tok	258×	12/15 (80%)

The agent is qwen2.5:7b, local via Ollama. The naive side was refused by the context guard in every case (its tool list does not fit a 32k model). The distilled side ran live against api.github.com and api.stripe.com (test mode). Full artifacts: eval_runs/headline/ and eval_runs/stripe/.

For GitHub, the per-scenario picture:

xychart-beta
    title "Distilled GitHub server: trials passed per scenario (3 trials each)"
    x-axis ["repo-overview", "repo-metadata", "list-branches", "repo-language", "search-repo"]
    y-axis "trials passed" 0 --> 3
    bar [3, 3, 3, 3, 1]

The weak scenario is also the expensive one. search-repo passed 1 of 3 trials and consumed about 2.7 times the input tokens of any other task. A small model that cannot decide between similar options flails, and flailing costs tokens:

xychart-beta
    title "Mean input tokens per task, by scenario"
    x-axis ["repo-overview", "repo-metadata", "list-branches", "repo-language", "search-repo"]
    y-axis "input tokens" 0 --> 10000
    bar [3582, 4062, 3448, 3401, 9569]

Stripe tells the same story from the other side: the four read scenarios passed 3 of 3 each; the one write scenario (create a customer) failed all three trials. The 7B model could not reliably drive a mutating call to completion. That is a model limit, not a curation limit, and the report records it plainly.

What the numbers show, and what they don't

The context result is decisive. 414k or 462k tokens of tool definitions do not fit a 32k model, no matter how capable the agent is. On small-context models, curation is what makes the API usable at all.

The 87% and 80% show the pipeline works end to end: an LLM-designed tool surface, compiled to code, drove live API calls well enough for a 7B model to answer correctly. It is not yet proof that curation beats a naive server that fits. That comparison (cap the naive baseline with --naive-max-tools on a large-context model, add a no-tools control) is supported by the harness and is the next experiment worth running. See Methodology and limitations.

Coverage tradeoff

Curation drops long-tail endpoints on purpose. The hero server above cannot touch subscriptions or payouts, because the description did not ask for them. That is the point: with a description, you decide what is covered, and the tool surface stays small enough to fit and cheap enough to reason over.

Nothing is locked in. The ToolPlan is YAML you can read and edit, and build is free and deterministic. A missing capability is one YAML edit and a rebuild away: add the endpoint to a tool (or add a tool), run build, and the server has it. The model proposes the starting point; you own the final shape.

How it works

Four commands. Two of them cost tokens.

flowchart LR
    S["OpenAPI spec<br/>(587 ops)"] --> I["inspect<br/><i>damage report</i>"]
    I --> P["plan<br/><i>1 LLM pass</i>"]
    P --> Y[["ToolPlan YAML<br/>editable, diff-friendly"]]
    Y --> B["build<br/><i>deterministic codegen</i>"]
    B --> M["FastMCP server<br/>(6 tools)"]
    M --> E["eval<br/><i>naive vs distilled</i>"]
    style Y fill:#fde68a,stroke:#b45309,color:#000
    style P fill:#bfdbfe,stroke:#1e40af,color:#000

Command	Cost	What it does
inspect	free	Parse the spec, print the naive-conversion damage report.
plan	1 LLM pass	Curate the spec (for your description, or --auto) into an editable ToolPlan YAML.
build	free	Compile the ToolPlan into a runnable FastMCP package.
eval	LLM per trial	Run scenarios against both servers, emit a report.

The ToolPlan YAML is the design decision the rest hangs on. The model proposes the curation; a human reviews and hand-edits the YAML; build turns it into code with no model in the loop. Nothing the LLM produces is a black box, and iterating on a plan is free because only plan and eval spend tokens.

# one of the 6 tools from the hero server, as it appears in the editable plan
- name: refund_lookup
  intent: List or retrieve refunds
  read_only: true
  endpoints: [GET /v1/refunds, GET /v1/refunds/{refund}]
  dispatch: { mode: select, selector: mode }   # "list" or "get" picks the endpoint
  response_filter:
    fields: [id, amount, currency, status, reason, created, charge]

Quickstart

git clone <this-repo> && cd prompt-to-mcp && uv sync

The petstore spec ships in the repo, so this runs end to end out of the box. inspect, plan, and build also accept an http(s) URL in place of a file path.

# 1. what a naive conversion would cost you
prompt-to-mcp inspect tests/fixtures/petstore.yaml

# 2a. curate it for a described server (the one paid step) into an editable ToolPlan
prompt-to-mcp plan tests/fixtures/petstore.yaml \
    "a server for looking up pets and their orders" \
    -o petstore.toolplan.yaml --provider anthropic --model claude-sonnet-5
#     then open petstore.toolplan.yaml and edit anything you disagree with

# 2b. or curate the whole API automatically
prompt-to-mcp plan tests/fixtures/petstore.yaml --auto \
    -o petstore.toolplan.yaml --provider anthropic --model claude-sonnet-5

# 3. compile the plan into a server (free, deterministic)
prompt-to-mcp build petstore.toolplan.yaml --spec tests/fixtures/petstore.yaml \
    -o generated/petstore

p2m is a short alias for the same CLI (p2m plan ..., p2m build ...).

To reproduce the Stripe and GitHub numbers you need their OpenAPI descriptions (not checked in), a running Ollama, and an API token:

# Stripe (test mode): distilled ran live, naive refused by the guard
PYTHONUTF8=1 prompt-to-mcp eval specs/spec3.json --server-dir generated/stripe \
    --scenarios scenarios/stripe.yaml -o eval_runs/stripe --only distilled --trials 3 \
    --provider ollama --model qwen2.5:7b --model-context 32768

PYTHONUTF8=1 prompt-to-mcp eval specs/spec3.json --server-dir generated/stripe \
    --scenarios scenarios/stripe.yaml --only naive --trials 1 \
    --provider ollama --model qwen2.5:7b --model-context 32768

# GitHub: same shape against generated/github and scenarios/github.yaml
mkdir -p specs && curl -L -o specs/api.github.com.json \
  https://raw.githubusercontent.com/github/rest-api-description/main/descriptions/api.github.com/api.github.com.json

Any model, any provider

The LLM layer speaks Anthropic-native or the OpenAI chat protocol. The two ends of this project ran on different vendors with zero code changes: curation on a hosted model, the eval on a local 7B via Ollama.

Preset (--provider)	Protocol	Base URL
anthropic	anthropic	SDK default
glm	anthropic	api.z.ai/api/anthropic
kimi	anthropic	api.moonshot.ai/anthropic
openrouter	openai	openrouter.ai/api/v1
ollama	openai	localhost:11434/v1

Custom gateways work with --base-url <url> --api-protocol openai|anthropic. Configuration is read from flags, then environment variables, then a .env in the CWD, then preset defaults. The environment variables are P2M_API_KEY, P2M_MODEL, P2M_BASE_URL, P2M_API_PROTOCOL, and P2M_MAX_OUTPUT_TOKENS; a bare ANTHROPIC_API_KEY or OPENAI_API_KEY is picked up as a key fallback. Run prompt-to-mcp providers for the full picture.

Running on real providers surfaced two quirks, both handled in prompt_to_mcp/llm.py:

• Gemini 3 attaches an opaque thought_signature to each tool call and rejects the next turn unless it is echoed back verbatim. The neutral ToolCallRequest.extra field carries it through the agent loop.
• Some gateways count a model's hidden reasoning tokens against max_tokens, so a modest cap starves the visible answer. The output budget is configurable (32k default) with a truncation-aware retry.

Methodology and limitations

For each (server, scenario, trial) the harness runs an agent loop and records the outcome, task success, tool-call count, token usage, and wall time. Outcomes are typed so a failed run can never masquerade as a real one:

• completed: the agent answered; success says whether it passed the checks.
• provider_error: the provider rejected the request, or the --model-context guard refused an oversized tool surface before running it.
• harness_error: an exception inside one trial. It never crashes the whole run.

Success is checked cheapest-first: substring and regex checks on facts that don't drift, with an LLM-as-judge rubric as fallback for open-ended answers.

Known limits:

Limitation	Effect
chars/4 token estimator	context figures are order-of-magnitude, not a tokenizer
Small N, 1 local model	success rates are noisy and model-specific
No no-tools control	some scenarios are partly answerable from model memory
Naive never run fitted	the win leans on the context ceiling, not proven tool quality
One write scenario, missed by the 7B	mutating paths are generated but not yet passed by this agent
LLM-as-judge fallback	can misjudge open-ended answers

Repo map

Path	What
prompt_to_mcp/ingest.py, analyze.py	Spec loading and the deterministic damage report
prompt_to_mcp/curate.py, toolplan.py	The LLM curation pass and the ToolPlan schema
prompt_to_mcp/emit.py, templates/	ToolPlan to FastMCP codegen (Jinja2)
prompt_to_mcp/evalrun.py, evalspec.py, evalreport.py	The eval harness
prompt_to_mcp/llm.py	Provider-pluggable LLM layer (both protocols)
prompt_to_mcp/cli.py	The inspect / plan / build / eval CLI
stripe-support.toolplan.yaml, generated/stripe-support/	The hero prompt-first server
scenarios/, generated/, eval_runs/	Scenarios, the curated servers, the runs behind the numbers

License

MIT. See LICENSE.