Skip to main content

SafeSink

SafeSink checks whether an MCP server's tools do what their annotations claim.

MCP tool annotations (readOnlyHint, destructiveHint, idempotentHint, openWorldHint) are untrusted hints: nothing stops a server from declaring a tool read-only while it writes files, or idempotent while every retry charges the customer twice. SafeSink calls the tool, observes the state it actually changed inside an explicit filesystem, SQLite, or HTTP JSON boundary, and reports where the declaration and the observed effect disagree.

One command, no manifest, no server changes:

safesink mcp-check -- npx -y @modelcontextprotocol/server-filesystem "$PWD/sandbox"

Install

safesink is not on PyPI yet, so pip install safesink fails. Install from the repository instead:

python -m pip install "git+https://github.com/virajsabhaya23/effectfence.git"

or, from a clone:

git clone https://github.com/virajsabhaya23/effectfence.git
cd effectfence
python -m pip install .

Python 3.10+; no runtime dependencies. The repository ships a dependency-free PEP 517 build backend, so the build works offline.

Quick start

Point it at any stdio MCP server:

mkdir -p sandbox
safesink mcp-check \
  --sandbox sandbox \
  --out out/mcp/report.json \
  --junit out/mcp/junit.xml \
  --sarif out/mcp/report.sarif \
  -- npx -y @modelcontextprotocol/server-filesystem "$PWD/sandbox"

mcp-check starts the server, reads tools/list, synthesizes minimal arguments from each tool's inputSchema, seeds the sandbox with probe fixtures those arguments can refer to, skips tools declaring destructiveHint: true, and verifies what changed. The command goes after -- so its own flags are not parsed by safesink.

For a server that needs a curated argument per tool, write a manifest and use mcp-verify instead:

safesink mcp-verify my-manifest.json --out out/mcp/report.json

The honest fixture and the deliberately dishonest one show both outcomes:

safesink mcp-verify examples/mcp-conformance/passing.json   # exit 0
safesink mcp-verify examples/mcp-conformance/failing.json   # exit 2

From Python:

from safesink import verify_manifest

report = verify_manifest("manifest.json")
print(report["verdict"], report["certificateSha256"])

Verdicts and exit codes

Verdict Exit Meaning
pass 0 every case was exercised and no declared effect was contradicted
inconclusive 3 nothing was contradicted, but at least one case could not be verified
fail 2 a declared effect was contradicted, or the run could not be trusted

inconclusive is a first-class outcome, not a soft pass. It covers rejected synthesized arguments, observers whose state changes on its own, and ambiguous results the observation boundary cannot classify. A case is only pass when it has no violations and no unresolved evidence, so CI never reports an unverified case as verified.

What it checks

Finding Meaning
READ_ONLY_HINT_MISMATCH a tool declared readOnlyHint: true changed observed state
NON_DESTRUCTIVE_HINT_MISMATCH a tool declared destructiveHint: false removed state
CLOSED_WORLD_HINT_MISMATCH a tool declared openWorldHint: false changed an open-world observer
IDEMPOTENT_HINT_MISMATCH repeating the identical call changed observed state again
TOOL_ERROR_EXPECTATION_MISMATCH tools/call returned isError contrary to the case
TOOL_NOT_LISTED a case targets a tool that tools/list never advertised
TOOL_COVERAGE_BELOW_MINIMUM too few advertised tools are covered by cases

Two behaviours keep false findings out of reports:

  • Observer control rounds. Every case (and every ambiguous-result trial) runs matched no-tool observation rounds first. A delta that reproduces without the tool is reported as OBSERVER_OR_BACKGROUND_INTERFERENCE and is not blamed on the tool.
  • Argument-synthesis limits are inconclusive. When mcp-scan generated the arguments and the server rejects them, the result is TOOL_ERROR_ON_GENERATED_ARGUMENTS, not a conformance finding. Hand-written cases that are rejected still fail.

Ambiguous-result schedules (ambiguousResultFault) repeat a call under drop-result-after-response or timeout-before-send and classify the outcome as committed-result-lost, no-effect-timeout, duplicate-on-retry, or explicitly unclassified.

Validation against real servers

Measured on 2026-11-21 with the released upstream servers, safesink mcp-check, default settings (destructive tools skipped):

Server Version Tools Cases Findings Inconclusive Verdict
@modelcontextprotocol/server-filesystem 2.4.0 16 13 0 1 inconclusive (exit 3)
@modelcontextprotocol/server-memory 2.2.1 9 6 0 0 pass (exit 0)
@modelcontextprotocol/server-sequential-thinking 2.2.0 1 1 0 0 pass (exit 0)

The single inconclusive case is copy_file, whose second identical call is rejected because the first call created the destination; the tool's idempotentHint: true therefore remains unverified rather than violated.

These are trusted servers, so zero findings is the expected result: the table shows that the scanner does not manufacture findings, not that it can catch a liar. Detection is covered by the deliberately dishonest fixture, where the suite asserts READ_ONLY_HINT_MISMATCH, IDEMPOTENT_HINT_MISMATCH, and NON_DESTRUCTIVE_HINT_MISMATCH.

An earlier revision reported 5 of these 13 cases as conformance failures because synthesized arguments pointed at directories and files that did not exist. That false-positive rate is why generated arguments are now seeded from probe fixtures and why a rejected generated call is inconclusive.

CI

- uses: actions/checkout@v7
- uses: virajsabhaya23/effectfence@main
  with:
    manifest: examples/mcp-conformance/passing.json
    out: out/mcp/report.json
    junit: out/mcp/junit.xml
    sarif: out/mcp/report.sarif

Pin a full commit SHA in high-assurance environments. The action verifies a manifest; use mcp-check in the job directly if you have no manifest yet. JUnit marks unverifiable cases as skipped, not failed, matching the verdicts above.

Security model

  • Commands are executed without a shell, as arrays, with a restricted environment (PATH plus explicitly inherited variables).
  • HTTP observers are host-allowlisted and follow no redirects.
  • Argument and output values are redacted by default, and sensitive observer digests are excluded from reports.
  • mcp-check only ever mutates the sandbox directory you give it, and skips tools that declare themselves destructive unless you pass --include-destructive.
  • Only configured observers define the observable boundary. Snapshot comparison cannot detect unobserved side effects, and it cannot prove that an external read occurred.
  • Annotations come from the server under test, so a dishonest server can lie to the checker itself; SafeSink reports what it observed, within that boundary.

See the MCP conformance guide for the versioned manifest, the observer boundary, and the full security model.

Frozen crash/retry simulator

Earlier versions also shipped a deterministic crash/retry schedule simulator with a SQLite effect ledger, reference sinks, a 30-case benchmark, and a live Kafka/PostgreSQL killpoint adapter. It is frozen: kept for reproducibility of existing artifacts, with no new features, adapters, or benchmark cases.

safesink-legacy verify examples/crash_after_effect.json --out out/report.json
safesink-legacy explore examples/crash_after_effect.json --out out/exploration.json
safesink-legacy benchmark benchmark/corpus.json --out benchmark/results.json

The implementation lives in safesink/legacy/; python -m safesink.legacy.cli is equivalent. See docs/FROZEN_CRASH_RETRY.md for its status, scope, and known limitations (notably: its benchmark is graded by the same model that generates the schedules, so its precision/recall figures measure corpus authoring, not detection of real systems).

The deprecated effectfence import path and console script still work and forward to safesink.

Metadata

Release files for safesink 0.4.0

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for safesink 0.4.0
File Size Uploaded Provenance Actions
safesink-0.4.0.tar.gz 87.9 kB With provenance Details for safesink-0.4.0.tar.gz

Built distribution (wheel)

Table of built distributions (wheels) for safesink 0.4.0
File Interpreter ABI Platform Actions
safesink-0.4.0-py3-none-any.whl Python 3 none any Details for safesink-0.4.0-py3-none-any.whl

Total release size: 148.4 kB

Release files / safesink-0.4.0.tar.gz

Download URL safesink-0.4.0.tar.gz
Size 87.9 kB
Tags Source
SHA-256 checksum
How to use checksums
2ad540950ba9ab9fed29020ae7bf1cdb305cc294d6f365a1eaaf4b2031e2f31f
BLAKE2b-256 checksum
How to use checksums
60c9d338df4ba4db1b51fe63dc7fb5cb93b28b8cc7a904cd929a1ecd15c679ff
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
Yes
Uploaded via twine/7.0.0 CPython/3.13.14

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Nov 21, 2026.

Transparency log

Release files / safesink-0.4.0-py3-none-any.whl

Download URL safesink-0.4.0-py3-none-any.whl
Size 60.5 kB
Tags Python 3
SHA-256 checksum
How to use checksums
761bc0c1b463ad3771e7059c4c7e8b96147627571ee91a2561f180b9de629cf7
BLAKE2b-256 checksum
How to use checksums
8d73c486707f60ef33f97bd734865e112fa0c7423a9990a348f8d7ee8ad769d8
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
Yes
Uploaded via twine/7.0.0 CPython/3.13.14

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Nov 21, 2026.

Transparency log

Security for safesink 0.4.0

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

Full provenance

Both files in this release have been uploaded with provenance metadata.

Select from 2 files

Attestation for safesink-0.4.0.tar.gz

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Nov 21, 2026.

Transparency log

A note on package security

PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.

To learn more about protecting yourself from malicious actors, visit our package security documentation.

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page