fetchgate 0.1.0

A deterministic gate at the fetch boundary: a 200 is transport success, not a read.

FetchGate

A 200 is not a read.

When an AI agent fetches a page and it quietly comes back empty (a JavaScript app, a 403, a Cloudflare wall), the agent often does not stop. It answers from memory and never tells you it did not actually read the page. FetchGate is a small gate at the fetch boundary that refuses to let that happen: the agent gets the content only when the page was really read, and an honest refusal otherwise.

It is deterministic (no model, no API keys), stdlib-only at its core, and free.

Quickstart

git clone https://github.com/MoAz06/FetchGate
cd FetchGate
python demo.py https://example.com https://httpbin.org/status/403

https://example.com
  verdict     : RETRIEVED  (clean)
  answerable  : ALLOW
  bytes       : raw 559 / extracted 127
  tiers       : static
  gate-confirmed content (127 chars): Example Domain This domain is for use ...

https://httpbin.org/status/403
  verdict     : FAILED  (transport.http_403)
  answerable  : STOP
  refused: the model gets no content and cannot answer as if it read this page.

Run the tests (no network, no model):

pip install -e .
python -m unittest discover -s tests

Use it in Claude Desktop or Cursor

pip install ".[mcp]"                 # the fetch tool
pip install ".[browser]"             # optional: render JavaScript pages
python -m playwright install chromium

Add to your MCP config (details in docs/mcp.md):

{ "mcpServers": { "fetchgate": { "command": "fetchgate-mcp" } } }

Your agent now has a fetch_url tool that returns content only for a confirmed read, and a refusal otherwise.

Use it in your own agent

from fetchgate import default_gate

gate = default_gate()   # renders JS pages if the [browser] extra is installed

def read_url(url: str) -> str:
    out = gate.guarded_fetch(url)
    content = gate.content_for(out.handle)
    if content is None:
        return f"NOT READ: {url} ({out.result.reason}). Do not answer as if you read it."
    return content

How it decides

Every fetch gets one verdict:

• RETRIEVED: a real read. The agent may answer and cite it.
• FAILED: a definite non-read (a 4xx or 5xx, a timeout, an https to http downgrade, an empty body). Stop.
• UNKNOWN: fetched, but not enough readable content to confirm a read (often a JavaScript page). Stop, or render and try again.

Three layers feed the verdict, and the result is the strictest of them:

• transport: status and redirect chain.
• extraction: readable text kept separate from raw bytes. A 17 byte price JSON is a read, a 400 KB article at 2 percent text is a read, a JS shell with no text is not.
• content-validity: a best-effort fingerprint for Cloudflare, consent, paywall, and soft-404 walls. It can only downgrade to UNKNOWN, never fake a read.

When a page looks thin or empty and the render tier is on, FetchGate escalates to a real headless browser and re-checks. It renders, it does not evade: a page still walled after an honest render stays a non-read.

Stated limits

• content_sha256 is a receipt anchor, not tamper-evidence. A cloaked page hashes cleanly.
• Provenance is not authenticity. The gate closes fail-silent-on-empty, not fetch-succeeds-on-wrong-content.
• A read is not an answer. RETRIEVED means enough content arrived, not that it held the fact you need. Claim support is a downstream job.
• Fail-closed has an availability cost. Anyone who can make your fetch fail can force a refusal. That is the trade for correctness.

What is built

The deterministic core, the offline evaluation with a signed manifest, the build-breaking cheat-test, the MCP server, and the Playwright render tier. Optional extras: [mcp], [browser], [extract] (trafilatura), [sign] (Ed25519).

MIT licensed.