Tool-Semantics

Know when an MCP change breaks the agent — not just the schema.

Behavioral compatibility testing for MCP tools and AI-agent interfaces.

Why Tool-Semantics?

AI agents do not call tools the way typed clients do. They choose tools from descriptions, invent arguments from schemas, and infer side effects from naming and prose. A change that remains JSON-Schema-valid can still:

• steer the model toward the wrong tool
• drop a required argument the model used to omit
• rename enums the model still emits
• quietly escalate from read-only to write/destructive behavior

Tool-Semantics captures normalized tool-interface snapshots and diffs them for structural and semantic risk — so teams can gate MCP and tool-API changes before agents ship broken workflows.

Compatibility layers

The MVP implements deterministic interface snapshots and structural comparison (layers 1–2, with warnings that point at 3–5), plus local MCP capture over stdio. Remote MCP transport and model-based behavioral testing are on the roadmap.

How it works

flowchart LR
  A[MCP / JSON manifest] --> B[Scanner]
  B --> C[Normalized snapshot]
  C --> D[Diff engine]
  D --> E[Compatibility report]
  E --> F[CLI / CI exit codes]

Quick start

python -m venv .venv
source .venv/bin/activate
pip install -e ".[dev]"

# Capture two interface versions
tool-semantics capture examples/github_server_v1.json -o .tool-semantics/v1.json
tool-semantics capture examples/github_server_v2.json -o .tool-semantics/v2.json

# Compare — exits 1 on breaking/critical changes
tool-semantics compare .tool-semantics/v1.json .tool-semantics/v2.json \
  --markdown-output .tool-semantics/report.md

Demo

Example output

Comparing the included GitHub demo manifests surfaces removals, renames via addition, description drift, and newly required parameters:

Tool-Semantics: 1.0.0 → 2.0.0
┌──────────┬────────────────────────────┬─────────────────────────┬──────────────────────────────────────────┐
│ Severity │ Code                       │ Subject                 │ Change                                   │
├──────────┼────────────────────────────┼─────────────────────────┼──────────────────────────────────────────┤
│ breaking │ tool.removed               │ search_issues           │ Tool 'search_issues' was removed.        │
│ info     │ tool.added                 │ find_work_items         │ Tool 'find_work_items' was added; …      │
│ warning  │ tool.description_changed   │ create_issue            │ Tool description changed; …              │
│ breaking │ parameter.added_required   │ create_issue.repository │ Required parameter 'repository' was …    │
└──────────┴────────────────────────────┴─────────────────────────┴──────────────────────────────────────────┘
Result: breaking

Install (library)

pip install tool-semantics
# or from source: pip install -e .

from pathlib import Path
from tool_semantics.scanner import capture_manifest
from tool_semantics.diff import compare_snapshots
from tool_semantics.report import render_markdown

baseline = capture_manifest(Path("examples/github_server_v1.json"))
candidate = capture_manifest(Path("examples/github_server_v2.json"))
report = compare_snapshots(baseline, candidate)
print(render_markdown(report))
print("compatible:", report.is_compatible)

Exit codes

CLI reference

tool-semantics --version
tool-semantics capture <manifest.json> [-o .tool-semantics/snapshot.json] \
  [--provenance-output snapshot.provenance.json] [-v]
tool-semantics capture-mcp -o snap.json \
  [--provenance-output snap.provenance.json] -- python my_mcp_server.py
tool-semantics compare <baseline.json> <candidate.json> \
  [--json-output report.json] \
  [--markdown-output report.md] \
  [--config .tool-semantics.toml] \
  [-v]

• --verbose / -v logs paths, tool counts, and change totals to stderr (default Rich UX unchanged).
• --config loads ignore rules; if omitted, .tool-semantics.toml in the cwd is used when present.
• capture-mcp speaks MCP JSON-RPC over stdio; secrets-like keys are redacted by default.

Approved baselines and provenance

Capture the approved interface into a Git-tracked baseline. The snapshot is the contract that compare uses; review and commit it when an interface change is intentional.

tool-semantics capture examples/github_server_v1.json \
  -o .tool-semantics/baselines/github.json \
  --provenance-output .tool-semantics/baselines/github.provenance.json

The optional provenance sidecar records capture context and a digest of the snapshot. It is separate from the snapshot and never affects compatibility comparisons.

JSON reports include changes, is_compatible, and counts by severity.
Change-code catalog: docs/change-codes.md.
Ignore-config schema: docs/config.md.
GitHub Action: docs/github-action.md.
Publishing: docs/publishing.md.
Migration adapters: docs/adapters.md.

Optional risk field

MCP does not standardize risk. Tool-Semantics accepts an optional tool-level risk value (read_only, external_write, destructive, unknown). Missing values default to unknown (no false escalation). See the GitHub demo manifests for examples.

Offline probes

from pathlib import Path
from tool_semantics.scanner import capture_manifest
from tool_semantics.probes import Probe, ProbeKind, evaluate_probes

snapshot = capture_manifest(Path("examples/github_server_v1.json"))
report = evaluate_probes(
    snapshot,
    [
        Probe(
            id="search",
            intent="find issues",
            expected_tool="search_issues",
            required_params=["query"],
            kind=ProbeKind.POSITIVE,
        )
    ],
)
assert report.passed

Project layout

src/tool_semantics/   # scanner, models, diff engine, report, CLI
examples/             # demo MCP-style manifests
tests/                # pytest suite
docs/assets/          # README visuals

Roadmap

See ROADMAP.md for milestones. Live MCP supports stdio and SSE capture; model-backed probes / metrics / stability are available via the library API (docs/probes.md). Downstream consumers (myelinmesh, dogfood capture): docs/downstream.md.

Contributing

We welcome issues and PRs — especially documentation fixes, tests, and compatibility-rule ideas.

• Read CONTRIBUTING.md
• Follow the Code of Conduct
• Browse good first issues

Security

Do not auto-execute discovered MCP tools. See SECURITY.md for reporting guidance.

License

Apache License 2.0 — see LICENSE.