Skip to main content

SafeSink

SafeSink checks whether an MCP server's tools do what their annotations claim.

MCP tool annotations (readOnlyHint, destructiveHint, idempotentHint, openWorldHint) are untrusted hints: nothing stops a server from declaring a tool read-only while it writes files, or idempotent while every retry charges the customer twice. SafeSink calls the tool, observes the state it actually changed inside an explicit filesystem, SQLite, or HTTP JSON boundary, and reports where the declaration and the observed effect disagree.

One command, no manifest, no server changes:

safesink mcp-check -- npx -y @modelcontextprotocol/server-filesystem "$PWD/sandbox"

Install

python -m pip install safesink

Python 3.10+; no runtime dependencies. To work on it, from a clone:

git clone https://github.com/virajsabhaya23/effectfence.git
cd effectfence
python -m pip install -e .

The repository ships a dependency-free PEP 517 build backend, so the build works offline.

Quick start

Point it at any stdio MCP server:

mkdir -p sandbox
safesink mcp-check \
  --sandbox sandbox \
  --out out/mcp/report.json \
  --junit out/mcp/junit.xml \
  --sarif out/mcp/report.sarif \
  -- npx -y @modelcontextprotocol/server-filesystem "$PWD/sandbox"

mcp-check starts the server, reads tools/list, synthesizes minimal arguments from each tool's inputSchema, seeds the sandbox with probe fixtures those arguments can refer to, skips tools declaring destructiveHint: true, and verifies what changed. The command goes after -- so its own flags are not parsed by safesink.

For a server that needs a curated argument per tool, write a manifest and use mcp-verify instead:

safesink mcp-verify my-manifest.json --out out/mcp/report.json

The honest fixture and the deliberately dishonest one show both outcomes:

safesink mcp-verify examples/mcp-conformance/passing.json   # exit 0
safesink mcp-verify examples/mcp-conformance/failing.json   # exit 2

From Python:

from safesink import verify_manifest

report = verify_manifest("manifest.json")
print(report["verdict"], report["certificateSha256"])

Verdicts and exit codes

Verdict Exit Meaning
pass 0 every case was exercised and no declared effect was contradicted
inconclusive 3 nothing was contradicted, but at least one case could not be verified
fail 2 a declared effect was contradicted, or the run could not be trusted

inconclusive is a first-class outcome, not a soft pass. It covers rejected synthesized arguments, observers whose state changes on its own, and ambiguous results the observation boundary cannot classify. A case is only pass when it has no violations and no unresolved evidence, so CI never reports an unverified case as verified.

What it checks

Finding Meaning
READ_ONLY_HINT_MISMATCH a tool declared readOnlyHint: true changed observed state
NON_DESTRUCTIVE_HINT_MISMATCH a tool declared destructiveHint: false removed state
CLOSED_WORLD_HINT_MISMATCH a tool declared openWorldHint: false changed an open-world observer
IDEMPOTENT_HINT_MISMATCH repeating the identical call changed observed state again
TOOL_ERROR_EXPECTATION_MISMATCH tools/call returned isError contrary to the case
TOOL_NOT_LISTED a case targets a tool that tools/list never advertised
TOOL_COVERAGE_BELOW_MINIMUM too few advertised tools are covered by cases

Two behaviours keep false findings out of reports:

  • Observer control rounds. Every case (and every ambiguous-result trial) runs matched no-tool observation rounds first. A delta that reproduces without the tool is reported as OBSERVER_OR_BACKGROUND_INTERFERENCE and is not blamed on the tool.
  • Argument-synthesis limits are inconclusive. When mcp-scan generated the arguments and the server rejects them, the result is TOOL_ERROR_ON_GENERATED_ARGUMENTS, not a conformance finding. Hand-written cases that are rejected still fail.

Ambiguous-result schedules (ambiguousResultFault) repeat a call under drop-result-after-response or timeout-before-send and classify the outcome as committed-result-lost, no-effect-timeout, duplicate-on-retry, or explicitly unclassified.

Validation against real servers

Measured on 2026-11-21 with the released upstream servers, safesink mcp-check, default settings (destructive tools skipped):

Server Version Tools Cases Findings Inconclusive Verdict
@modelcontextprotocol/server-filesystem 2.4.0 16 13 0 1 inconclusive (exit 3)
@modelcontextprotocol/server-memory 2.2.1 9 6 0 0 pass (exit 0)
@modelcontextprotocol/server-sequential-thinking 2.2.0 1 1 0 0 pass (exit 0)

The single inconclusive case is copy_file, whose second identical call is rejected because the first call created the destination; the tool's idempotentHint: true therefore remains unverified rather than violated.

These are trusted servers, so zero findings is the expected result: the table shows that the scanner does not manufacture findings, not that it can catch a liar. Detection is covered by the deliberately dishonest fixture, where the suite asserts READ_ONLY_HINT_MISMATCH, IDEMPOTENT_HINT_MISMATCH, and NON_DESTRUCTIVE_HINT_MISMATCH.

An earlier revision reported 5 of these 13 cases as conformance failures because synthesized arguments pointed at directories and files that did not exist. That false-positive rate is why generated arguments are now seeded from probe fixtures and why a rejected generated call is inconclusive.

CI

- uses: actions/checkout@v7
- uses: virajsabhaya23/effectfence@main
  with:
    manifest: examples/mcp-conformance/passing.json
    out: out/mcp/report.json
    junit: out/mcp/junit.xml
    sarif: out/mcp/report.sarif

Pin a full commit SHA in high-assurance environments. The action verifies a manifest; use mcp-check in the job directly if you have no manifest yet. JUnit marks unverifiable cases as skipped, not failed, matching the verdicts above.

Security model

  • Commands are executed without a shell, as arrays, with a restricted environment (PATH plus explicitly inherited variables).
  • HTTP observers are host-allowlisted and follow no redirects.
  • Argument and output values are redacted by default, and sensitive observer digests are excluded from reports.
  • mcp-check only ever mutates the sandbox directory you give it, and skips tools that declare themselves destructive unless you pass --include-destructive.
  • Only configured observers define the observable boundary. Snapshot comparison cannot detect unobserved side effects, and it cannot prove that an external read occurred.
  • Annotations come from the server under test, so a dishonest server can lie to the checker itself; SafeSink reports what it observed, within that boundary.

See the MCP conformance guide for the versioned manifest, the observer boundary, and the full security model.

Frozen crash/retry simulator

Earlier versions also shipped a deterministic crash/retry schedule simulator with a SQLite effect ledger, reference sinks, a 30-case benchmark, and a live Kafka/PostgreSQL killpoint adapter. It is frozen: kept for reproducibility of existing artifacts, with no new features, adapters, or benchmark cases.

safesink-legacy verify examples/crash_after_effect.json --out out/report.json
safesink-legacy explore examples/crash_after_effect.json --out out/exploration.json
safesink-legacy benchmark benchmark/corpus.json --out benchmark/results.json

The implementation lives in safesink/legacy/; python -m safesink.legacy.cli is equivalent. See docs/FROZEN_CRASH_RETRY.md for its status, scope, and known limitations (notably: its benchmark is graded by the same model that generates the schedules, so its precision/recall figures measure corpus authoring, not detection of real systems).

The deprecated effectfence import path and console script still work and forward to safesink.

Metadata

Release files for safesink 0.4.2

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for safesink 0.4.2
File Size Uploaded Provenance Actions
safesink-0.4.2.tar.gz 88.2 kB With provenance Details for safesink-0.4.2.tar.gz

Built distribution (wheel)

Table of built distributions (wheels) for safesink 0.4.2
File Interpreter ABI Platform Actions
safesink-0.4.2-py3-none-any.whl Python 3 none any Details for safesink-0.4.2-py3-none-any.whl

Total release size: 148.7 kB

Release files / safesink-0.4.2.tar.gz

Download URL safesink-0.4.2.tar.gz
Size 88.2 kB
Tags Source
SHA-256 checksum
How to use checksums
edf7ae1c00763e530db3d0b3c2ceb022d25dc535dd23534229162388b47bd05d
BLAKE2b-256 checksum
How to use checksums
2cc30f0c946388e13b60157d4e3ce1b668d01fbc112e727959aefe3d9b6cfdc3
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
Yes
Uploaded via twine/7.0.0 CPython/3.13.14

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Nov 21, 2026.

Transparency log

Release files / safesink-0.4.2-py3-none-any.whl

Download URL safesink-0.4.2-py3-none-any.whl
Size 60.5 kB
Tags Python 3
SHA-256 checksum
How to use checksums
0972279c0bc8b4640c2d406037f7f04a15ae38d6d08d4bc7e7b31dc32edeca95
BLAKE2b-256 checksum
How to use checksums
2805881240ca127d1231993e6d2168a66c19ce34d4f629d5b8333331f123778d
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
Yes
Uploaded via twine/7.0.0 CPython/3.13.14

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Nov 21, 2026.

Transparency log

Security for safesink 0.4.2

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

Full provenance

Both files in this release have been uploaded with provenance metadata.

Select from 2 files

Attestation for safesink-0.4.2.tar.gz

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Nov 21, 2026.

Transparency log

A note on package security

PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.

To learn more about protecting yourself from malicious actors, visit our package security documentation.

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page