Skip to main content

SafeSink

SafeSink checks whether an MCP server's tools do what their annotations claim.

MCP tool annotations (readOnlyHint, destructiveHint, idempotentHint, openWorldHint) are untrusted hints: nothing stops a server from declaring a tool read-only while it writes files, or idempotent while every retry charges the customer twice. SafeSink calls the tool, observes the state it actually changed inside an explicit filesystem, SQLite, or HTTP JSON boundary, and reports where the declaration and the observed effect disagree.

One command, no manifest, no server changes:

safesink mcp-check -- npx -y @modelcontextprotocol/server-filesystem "$PWD/sandbox"

Install

python -m pip install safesink

Python 3.10+; no runtime dependencies. To work on it, from a clone:

git clone https://github.com/virajsabhaya23/effectfence.git
cd effectfence
python -m pip install -e .

The repository ships a dependency-free PEP 517 build backend, so the build works offline.

Quick start

Point it at any stdio MCP server:

mkdir -p sandbox
safesink mcp-check \
  --sandbox sandbox \
  --out out/mcp/report.json \
  --junit out/mcp/junit.xml \
  --sarif out/mcp/report.sarif \
  -- npx -y @modelcontextprotocol/server-filesystem "$PWD/sandbox"

mcp-check starts the server, reads tools/list, synthesizes minimal arguments from each tool's inputSchema (reusing any quoted example the description gives, such as '/mongodb/docs'), seeds the sandbox with probe fixtures those arguments can refer to, skips tools declaring destructiveHint: true, and verifies what changed. Servers that need configuration take --env KEY=VALUE, repeatable. The command goes after -- so its own flags are not parsed by safesink.

For a server that needs a curated argument per tool, write a manifest and use mcp-verify instead:

safesink mcp-verify my-manifest.json --out out/mcp/report.json

The honest fixture and the deliberately dishonest one show both outcomes:

safesink mcp-verify examples/mcp-conformance/passing.json   # exit 0
safesink mcp-verify examples/mcp-conformance/failing.json   # exit 2

From Python:

from safesink import verify_manifest

report = verify_manifest("manifest.json")
print(report["verdict"], report["certificateSha256"])

Verdicts and exit codes

Verdict Exit Meaning
pass 0 every case was exercised and no declared effect was contradicted
inconclusive 3 nothing was contradicted, but at least one case could not be verified
fail 2 a declared effect was contradicted, or the run could not be trusted

inconclusive is a first-class outcome, not a soft pass. It covers rejected synthesized arguments, observers whose state changes on its own, and ambiguous results the observation boundary cannot classify. A case is only pass when it has no violations and no unresolved evidence, so CI never reports an unverified case as verified.

What it checks

Finding Meaning
READ_ONLY_HINT_MISMATCH a tool declared readOnlyHint: true changed observed state
NON_DESTRUCTIVE_HINT_MISMATCH a tool declared destructiveHint: false removed state
CLOSED_WORLD_HINT_MISMATCH a tool declared openWorldHint: false changed an open-world observer
IDEMPOTENT_HINT_MISMATCH repeating the identical call changed observed state again
TOOL_ERROR_EXPECTATION_MISMATCH tools/call returned isError contrary to the case
TOOL_NOT_LISTED a case targets a tool that tools/list never advertised
TOOL_COVERAGE_BELOW_MINIMUM too few advertised tools are covered by cases

A tool that publishes no annotations at all has nothing to contradict, so it is reported as NO_DECLARED_ANNOTATIONS and is never called: a pass there would claim the tool was verified when no check existed. Add a contract to a case if you want to verify an unannotated server against your own policy.

Two behaviours keep false findings out of reports:

  • Observer control rounds. Every case (and every ambiguous-result trial) runs matched no-tool observation rounds first. A delta that reproduces without the tool is reported as OBSERVER_OR_BACKGROUND_INTERFERENCE and is not blamed on the tool.
  • Argument-synthesis limits are inconclusive. When mcp-scan generated the arguments and the server rejects them, the result is TOOL_ERROR_ON_GENERATED_ARGUMENTS, not a conformance finding. Hand-written cases that are rejected still fail.

Ambiguous-result schedules (ambiguousResultFault) repeat a call under drop-result-after-response or timeout-before-send and classify the outcome as committed-result-lost, no-effect-timeout, duplicate-on-retry, or explicitly unclassified.

Validation against real servers

Measured on 2026-11-21 with the released upstream servers, safesink mcp-check, default settings (destructive tools skipped):

Server Version Tools Cases Findings Inconclusive Verdict
@modelcontextprotocol/server-filesystem 2.4.0 16 13 0 1 inconclusive (exit 3)
@modelcontextprotocol/server-memory 2.2.1 9 6 0 0 pass (exit 0)
@modelcontextprotocol/server-sequential-thinking 2.2.0 1 1 0 0 pass (exit 0)

The single inconclusive case is copy_file, whose second identical call is rejected because the first call created the destination; the tool's idempotentHint: true therefore remains unverified rather than violated.

Servers outside the official set, same date, same defaults:

Server Version Tools Cases Findings Inconclusive Verdict
@upstash/context7-mcp 4.7.3 2 2 0 0 pass (exit 0)
duckduckgo-mcp-server (PyPI) 0.7.0 3 3 0 3 inconclusive (exit 3)
mcp-server-sqlite-npx 0.8.0 5 5 0 5 inconclusive (exit 3)

Context7 publishes full annotations on both tools; its readOnlyHint, idempotentHint and openWorldHint claims held. The other two publish no annotations on any tool, so every case is NO_DECLARED_ANNOTATIONS and no tool is called: there is nothing to compare against. That is the intended outcome, and it is why an unannotated tool is not reported as a pass.

These are trusted servers, so zero findings is the expected result: the table shows that the scanner does not manufacture findings, not that it can catch a liar. Detection is covered by the deliberately dishonest fixture, where the suite asserts READ_ONLY_HINT_MISMATCH, IDEMPOTENT_HINT_MISMATCH, and NON_DESTRUCTIVE_HINT_MISMATCH.

An earlier revision reported 5 of these 13 cases as conformance failures because synthesized arguments pointed at directories and files that did not exist. That false-positive rate is why generated arguments are now seeded from probe fixtures and why a rejected generated call is inconclusive.

CI

- uses: actions/checkout@v7
- uses: virajsabhaya23/effectfence@main
  with:
    manifest: examples/mcp-conformance/passing.json
    out: out/mcp/report.json
    junit: out/mcp/junit.xml
    sarif: out/mcp/report.sarif

Pin a full commit SHA in high-assurance environments. The action verifies a manifest; use mcp-check in the job directly if you have no manifest yet. JUnit marks unverifiable cases as skipped, not failed, matching the verdicts above.

Security model

  • Commands are executed without a shell, as arrays, with a restricted environment (PATH plus explicitly inherited variables).
  • HTTP observers are host-allowlisted and follow no redirects.
  • Argument and output values are redacted by default, and sensitive observer digests are excluded from reports.
  • mcp-check only ever mutates the sandbox directory you give it, and skips tools that declare themselves destructive unless you pass --include-destructive.
  • Only configured observers define the observable boundary. Snapshot comparison cannot detect unobserved side effects, and it cannot prove that an external read occurred.
  • Annotations come from the server under test, so a dishonest server can lie to the checker itself; SafeSink reports what it observed, within that boundary.

See the MCP conformance guide for the versioned manifest, the observer boundary, and the full security model.

Frozen crash/retry simulator

Earlier versions also shipped a deterministic crash/retry schedule simulator with a SQLite effect ledger, reference sinks, a 30-case benchmark, and a live Kafka/PostgreSQL killpoint adapter. It is frozen: kept for reproducibility of existing artifacts, with no new features, adapters, or benchmark cases.

safesink-legacy verify examples/crash_after_effect.json --out out/report.json
safesink-legacy explore examples/crash_after_effect.json --out out/exploration.json
safesink-legacy benchmark benchmark/corpus.json --out benchmark/results.json

The implementation lives in safesink/legacy/; python -m safesink.legacy.cli is equivalent. See docs/FROZEN_CRASH_RETRY.md for its status, scope, and known limitations (notably: its benchmark is graded by the same model that generates the schedules, so its precision/recall figures measure corpus authoring, not detection of real systems).

The deprecated effectfence import path and console script still work and forward to safesink.

Metadata

Release files for safesink 0.4.3

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for safesink 0.4.3
File Size Uploaded Provenance Actions
safesink-0.4.3.tar.gz 90.7 kB With provenance Details for safesink-0.4.3.tar.gz

Built distribution (wheel)

Table of built distributions (wheels) for safesink 0.4.3
File Interpreter ABI Platform Actions
safesink-0.4.3-py3-none-any.whl Python 3 none any Details for safesink-0.4.3-py3-none-any.whl

Total release size: 152.3 kB

Release files / safesink-0.4.3.tar.gz

Download URL safesink-0.4.3.tar.gz
Size 90.7 kB
Tags Source
SHA-256 checksum
How to use checksums
b999770b163cd7f1ef0877174a37b1e556e3c0e09b044730a4d1512d70f502d7
BLAKE2b-256 checksum
How to use checksums
f1b963aae0118b7b6832a28d5afb3268bd7127663880eef2d948125df20be162
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
Yes
Uploaded via twine/7.0.0 CPython/3.13.14

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Nov 21, 2026.

Transparency log

Release files / safesink-0.4.3-py3-none-any.whl

Download URL safesink-0.4.3-py3-none-any.whl
Size 61.6 kB
Tags Python 3
SHA-256 checksum
How to use checksums
02e9a360d3d733494e4bc0f8d7aa88eb71962ef9f0472cc1d3271dd2b9a1be08
BLAKE2b-256 checksum
How to use checksums
efd8885b45dc764aa8160143c8236879f93c5e18470a221ae5d20949f9488ebd
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
Yes
Uploaded via twine/7.0.0 CPython/3.13.14

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Nov 21, 2026.

Transparency log

Security for safesink 0.4.3

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

Full provenance

Both files in this release have been uploaded with provenance metadata.

Select from 2 files

Attestation for safesink-0.4.3.tar.gz

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Nov 21, 2026.

Transparency log

A note on package security

PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.

To learn more about protecting yourself from malicious actors, visit our package security documentation.

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page