SafeSink
SafeSink checks whether an MCP server's tools do what their annotations claim.
MCP tool annotations (readOnlyHint, destructiveHint, idempotentHint,
openWorldHint) are untrusted hints: nothing stops a server from declaring a
tool read-only while it writes files, or idempotent while every retry charges the
customer twice. SafeSink calls the tool, observes the state it actually changed
inside an explicit filesystem, SQLite, or HTTP JSON boundary, and reports where
the declaration and the observed effect disagree.
One command, no manifest, no server changes:
safesink mcp-check -- npx -y @modelcontextprotocol/server-filesystem "$PWD/sandbox"
Install
python -m pip install safesink
Python 3.10+; no runtime dependencies. To work on it, from a clone:
git clone https://github.com/virajsabhaya23/effectfence.git
cd effectfence
python -m pip install -e .
The repository ships a dependency-free PEP 517 build backend, so the build works offline.
Quick start
Point it at any stdio MCP server:
mkdir -p sandbox
safesink mcp-check \
--sandbox sandbox \
--out out/mcp/report.json \
--junit out/mcp/junit.xml \
--sarif out/mcp/report.sarif \
-- npx -y @modelcontextprotocol/server-filesystem "$PWD/sandbox"
mcp-check starts the server, reads tools/list, synthesizes minimal arguments
from each tool's inputSchema, seeds the sandbox with probe fixtures those
arguments can refer to, skips tools declaring destructiveHint: true, and
verifies what changed. The command goes after -- so its own flags are not
parsed by safesink.
For a server that needs a curated argument per tool, write a manifest and use
mcp-verify instead:
safesink mcp-verify my-manifest.json --out out/mcp/report.json
The honest fixture and the deliberately dishonest one show both outcomes:
safesink mcp-verify examples/mcp-conformance/passing.json # exit 0
safesink mcp-verify examples/mcp-conformance/failing.json # exit 2
From Python:
from safesink import verify_manifest
report = verify_manifest("manifest.json")
print(report["verdict"], report["certificateSha256"])
Verdicts and exit codes
| Verdict | Exit | Meaning |
|---|---|---|
pass |
0 | every case was exercised and no declared effect was contradicted |
inconclusive |
3 | nothing was contradicted, but at least one case could not be verified |
fail |
2 | a declared effect was contradicted, or the run could not be trusted |
inconclusive is a first-class outcome, not a soft pass. It covers rejected
synthesized arguments, observers whose state changes on its own, and ambiguous
results the observation boundary cannot classify. A case is only pass when it
has no violations and no unresolved evidence, so CI never reports an
unverified case as verified.
What it checks
| Finding | Meaning |
|---|---|
READ_ONLY_HINT_MISMATCH |
a tool declared readOnlyHint: true changed observed state |
NON_DESTRUCTIVE_HINT_MISMATCH |
a tool declared destructiveHint: false removed state |
CLOSED_WORLD_HINT_MISMATCH |
a tool declared openWorldHint: false changed an open-world observer |
IDEMPOTENT_HINT_MISMATCH |
repeating the identical call changed observed state again |
TOOL_ERROR_EXPECTATION_MISMATCH |
tools/call returned isError contrary to the case |
TOOL_NOT_LISTED |
a case targets a tool that tools/list never advertised |
TOOL_COVERAGE_BELOW_MINIMUM |
too few advertised tools are covered by cases |
Two behaviours keep false findings out of reports:
- Observer control rounds. Every case (and every ambiguous-result trial) runs
matched no-tool observation rounds first. A delta that reproduces without the
tool is reported as
OBSERVER_OR_BACKGROUND_INTERFERENCEand is not blamed on the tool. - Argument-synthesis limits are inconclusive. When
mcp-scangenerated the arguments and the server rejects them, the result isTOOL_ERROR_ON_GENERATED_ARGUMENTS, not a conformance finding. Hand-written cases that are rejected still fail.
Ambiguous-result schedules (ambiguousResultFault) repeat a call under
drop-result-after-response or timeout-before-send and classify the outcome as
committed-result-lost, no-effect-timeout, duplicate-on-retry, or explicitly
unclassified.
Validation against real servers
Measured on 2026-11-21 with the released upstream servers, safesink mcp-check,
default settings (destructive tools skipped):
| Server | Version | Tools | Cases | Findings | Inconclusive | Verdict |
|---|---|---|---|---|---|---|
@modelcontextprotocol/server-filesystem |
2.4.0 | 16 | 13 | 0 | 1 | inconclusive (exit 3) |
@modelcontextprotocol/server-memory |
2.2.1 | 9 | 6 | 0 | 0 | pass (exit 0) |
@modelcontextprotocol/server-sequential-thinking |
2.2.0 | 1 | 1 | 0 | 0 | pass (exit 0) |
The single inconclusive case is copy_file, whose second identical call is
rejected because the first call created the destination; the tool's
idempotentHint: true therefore remains unverified rather than violated.
These are trusted servers, so zero findings is the expected result: the table
shows that the scanner does not manufacture findings, not that it can catch a
liar. Detection is covered by the deliberately dishonest fixture, where the
suite asserts READ_ONLY_HINT_MISMATCH, IDEMPOTENT_HINT_MISMATCH, and
NON_DESTRUCTIVE_HINT_MISMATCH.
An earlier revision reported 5 of these 13 cases as conformance failures because synthesized arguments pointed at directories and files that did not exist. That false-positive rate is why generated arguments are now seeded from probe fixtures and why a rejected generated call is inconclusive.
CI
- uses: actions/checkout@v7
- uses: virajsabhaya23/effectfence@main
with:
manifest: examples/mcp-conformance/passing.json
out: out/mcp/report.json
junit: out/mcp/junit.xml
sarif: out/mcp/report.sarif
Pin a full commit SHA in high-assurance environments. The action verifies a
manifest; use mcp-check in the job directly if you have no manifest yet. JUnit
marks unverifiable cases as skipped, not failed, matching the verdicts above.
Security model
- Commands are executed without a shell, as arrays, with a restricted
environment (
PATHplus explicitly inherited variables). - HTTP observers are host-allowlisted and follow no redirects.
- Argument and output values are redacted by default, and sensitive observer digests are excluded from reports.
mcp-checkonly ever mutates the sandbox directory you give it, and skips tools that declare themselves destructive unless you pass--include-destructive.- Only configured observers define the observable boundary. Snapshot comparison cannot detect unobserved side effects, and it cannot prove that an external read occurred.
- Annotations come from the server under test, so a dishonest server can lie to the checker itself; SafeSink reports what it observed, within that boundary.
See the MCP conformance guide for the versioned manifest, the observer boundary, and the full security model.
Frozen crash/retry simulator
Earlier versions also shipped a deterministic crash/retry schedule simulator with a SQLite effect ledger, reference sinks, a 30-case benchmark, and a live Kafka/PostgreSQL killpoint adapter. It is frozen: kept for reproducibility of existing artifacts, with no new features, adapters, or benchmark cases.
safesink-legacy verify examples/crash_after_effect.json --out out/report.json
safesink-legacy explore examples/crash_after_effect.json --out out/exploration.json
safesink-legacy benchmark benchmark/corpus.json --out benchmark/results.json
The implementation lives in safesink/legacy/; python -m safesink.legacy.cli
is equivalent. See docs/FROZEN_CRASH_RETRY.md for
its status, scope, and known limitations (notably: its benchmark is graded by
the same model that generates the schedules, so its precision/recall figures
measure corpus authoring, not detection of real systems).
The deprecated effectfence import path and console script still work and
forward to safesink.
Metadata
Release files for safesink 0.4.2
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | Provenance | Actions |
|---|---|---|---|---|
| safesink-0.4.2.tar.gz | 88.2 kB | With provenance | Details for safesink-0.4.2.tar.gz |
Built distribution (wheel)
| File | Interpreter | ABI | Platform | Actions Reset |
|---|---|---|---|---|
| safesink-0.4.2-py3-none-any.whl | Python 3 | none | any | Details for safesink-0.4.2-py3-none-any.whl |
Total release size: 148.7 kB
Release files / safesink-0.4.2.tar.gz
| Download URL | safesink-0.4.2.tar.gz |
|---|---|
| Size | 88.2 kB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
edf7ae1c00763e530db3d0b3c2ceb022d25dc535dd23534229162388b47bd05d
|
|
BLAKE2b-256 checksum How to use checksums |
2cc30f0c946388e13b60157d4e3ce1b668d01fbc112e727959aefe3d9b6cfdc3
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
Yes |
| Uploaded via |
twine/7.0.0 CPython/3.13.14
|
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on Nov 21, 2026.
Transparency logRelease files / safesink-0.4.2-py3-none-any.whl
| Download URL | safesink-0.4.2-py3-none-any.whl |
|---|---|
| Size | 60.5 kB |
| Tags | Python 3 |
|
SHA-256 checksum How to use checksums |
0972279c0bc8b4640c2d406037f7f04a15ae38d6d08d4bc7e7b31dc32edeca95
|
|
BLAKE2b-256 checksum How to use checksums |
2805881240ca127d1231993e6d2168a66c19ce34d4f629d5b8333331f123778d
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
Yes |
| Uploaded via |
twine/7.0.0 CPython/3.13.14
|
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on Nov 21, 2026.
Transparency logSecurity for safesink 0.4.2
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
Full provenance
Both files in this release have been uploaded with provenance metadata.
Attestation for safesink-0.4.2.tar.gz
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on Nov 21, 2026.
Transparency logA note on package security
PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.
To learn more about protecting yourself from malicious actors, visit our package security documentation.