Skip to main content

crawlcheck (Python)

Before your agent reads, cites, connects to or transacts with a domain, ask CrawlCheck. No dependencies; Python 3.8+.

pip install https://crawlcheck.io/sdk/crawlcheck-1.0.1-py3-none-any.whl
from crawlcheck import CrawlCheck

cc = CrawlCheck()                      # CrawlCheck(key="cc_...") for licensed detail
answer = cc.resolve("example.com")     # signed ResolveV1: policy, delivery, files, capabilities, entity, freshness, actions
print(answer["actions"]["read"]["decision"])

receipt = cc.preflight("example.com", action="cite", agent="GPTBot", template="safe_citation")
print(receipt["decision"], [s["why"] for s in receipt["steps"]])
assert cc.verify(receipt)["verified"]  # Ed25519, checked locally; the key must be in the published directory

ok, receipt = cc.guard("shop.example", action="transact", confirm=lambda r: input(r["meaning"] + " y/n? ") == "y")

Command line: python -m crawlcheck preflight example.com cite GPTBot, python -m crawlcheck verify receipt.json.

Decisions: allow, warn, require_confirmation, block, unsupported. A policy (max_age_hours, on_warn, require_entity, human_approval, allow, deny) can only make a decision stricter.

LangChain, LlamaIndex, CrewAI and the OpenAI Agents SDK (CrewAI and OpenAI Agents 1.11.0)

from crawlcheck import CrawlCheck
from crawlcheck.langchain import crawlcheck_tools        # needs langchain-core
# from crawlcheck.llama_index import crawlcheck_tools   # needs llama-index-core
# from crawlcheck.crewai import crawlcheck_tools        # needs crewai: Agent(..., tools=tools)
# from crawlcheck.openai_agents import crawlcheck_tools # needs openai-agents: Agent(..., tools=tools)

tools = crawlcheck_tools(CrawlCheck(key="cc_..."))       # or CrawlCheck() with no key

Three tools, the same in every framework: crawlcheck_check_site (a signed allow / warn / block decision before an agent reads, cites, connects to, pays or administers a site), crawlcheck_verify_fact (one phone, address, hours or service area checked against what the business publishes) and crawlcheck_fetch (fetches a page only after its domain and every redirect hop proceed). They fail closed. The package itself still has no dependencies; each adapter imports its framework only when called.

A free key of your own: curl -s -X POST -H "content-type: application/json" -d '{"email":"you@example.com"}' https://crawlcheck.io/api/v1/keys/free

Trust

verify_document(doc) and CrawlCheck.verify(doc) return verified, integrity_valid, issuer_trusted and accepted. Act on accepted: offline, issuer_trusted is None and accepted is False, because a document signed with someone else's key can be self-consistent. CrawlCheck.verify and guard check the published key directory, so accepted is decided there.

Since 1.9.1 a key the rotation record lists as retired still counts for its own signing window when the expected purpose is worked out: after a decision-key rotation, receipts the old decision key signed inside its window verify, and no other key may sign a decision made while that key was live (key-policy suite cases retired-decision-key-*).

Since 1.9.0 log_check reads the chain of signed heads 200 per request (/api/v1/log/sths) instead of one per batch, so a first check on a client holding no heads takes one or two requests; an older server is still walked one head per request.

Since 1.8.0 an MCP connect decision names the server and, with a lockfile, its approved tools_sha256 (preflight(domain, "connect", mcp={"url": ..., "tools_sha256": ...}); with_preflight sends it for you). CrawlCheck reads the live tools from its own network: a changed tool list comes back as block, and the signed receipt's assessed block says exactly which endpoint and tool state it covered.

Since 1.8.0 they also apply the signing policy (https://crawlcheck.io/keys#policy): which key purpose may sign which object at which time (decision receipts move to the decision key at 2026-11-09 17:00 UTC), retired keys (valid only for documents signed before they stopped) and compromised keys (never valid on their signature alone). The result carries key_policy_valid, and accepted needs it. Offline, build the same input from copies you saved: kp = key_policy_from(saved_directory, saved_rotation_record), then apply_key_policy(result, doc, kp) or client.verify(doc, key_policy=kp). A saved copy cannot know a key retired or compromised after you saved it.

accepted says who signed a document. It is not permission to act: a valid receipt can say block, warn or require_confirmation. guard() decides that part.

Since 1.12.0 guard() leaves the split in client.last_guard["acceptance"]: integrity_valid, issuer_trusted, key_policy_valid, freshness_valid, subject_matches, operation_matches, replay_safe, baseline_matches, assessment_complete, policy_source (which policy produced the decision), evidence_level (A record integrity, B reproducible decision, C corroborated reading) and policy_result (proceed, pause or deny). acceptance_of(receipt, verification, decision_scope(receipt, domain, action)) computes it for a receipt you hold. It is the same contract as the JS SDK's acceptanceOf, pinned by the shared golden file tests/fixtures/continuity.json. A receipt is evidence and a recommendation; your deployment's policy decides whether the agent acts. POST https://crawlcheck.io/api/v1/preflight/replay re-runs a receipt's decision from the archived answer and the inputs it carries.

Give your agent the check as a tool

agent_tools(cc) returns three tools: crawlcheck_check_site (the signed, verified decision for an action on a site, including the business facts the site publishes and which AI engines get them wrong), crawlcheck_verify_fact (one signed verdict on a phone number, address, hours or service area you hold) and crawlcheck_fetch (fetches a page only after its domain and every redirect hop are allowed). All fail closed.

from crawlcheck import CrawlCheck, agent_tools, tools_for
tools = agent_tools(CrawlCheck(), agent="MyAgentBot")
defs, call = tools_for("anthropic", tools)   # or "openai", "mcp"; on a tool call: call(name, args)
print(call("crawlcheck_verify_fact", {"site": "example.com", "field": "phone", "value": "(720) 555-0100"})["verdict"])

Preflight inside your MCP client

Wrap an MCP ClientSession so initialize and call_tool are decided before anything is sent (proposal):

from mcp import ClientSession
from mcp.client.streamable_http import streamable_http_client
from crawlcheck import CrawlCheck
from crawlcheck.mcp import McpPreflight, with_preflight, McpPreflightRefused

cc = CrawlCheck()
pf = McpPreflight(cc, confirm=ask_the_user)
await pf.before_connect(url)              # McpPreflightRefused before any transport opens: block / unsupported / no confirmation
async with streamable_http_client(url) as (read, write):
    async with with_preflight(ClientSession(read, write), cc, server_url=url, preflight=pf) as s:
        await s.initialize()
        await s.call_tool("search", {"q": "x"})

A refusal raised inside the transport's context arrives in an ExceptionGroup (except* McpPreflightRefused); deciding first, as above, keeps it a plain exception. Options: lock= (an approved lockfile: changed tools refuse, tools outside it refuse), confirm_destructive=True, allow_warn=True, local="refuse" for stdio servers. Other stacks use McpPreflight(cc).before_connect(url) / before_tool_call(name, annotations).

Tool-interface continuity (1.12.0)

The promise, and only this one: your agent does not silently proceed with a changed tool definition when your policy requires renewed approval (spec). Keep per-part hashes in the lockfile (lock["parts"] = lock_parts(tools)), then McpPreflight(cc, lock=lock, confirm=ask, continuity={"mode": "observe", "on_record": log.write}), moving to "review" and then "enforce". observe never blocks and records what the policy would do; review sends every change to a person (integrity failures still deny); enforce pauses a changed or new tool on its own while the rest stay usable, and denies on an invalid signature, an unpublished or retired key, or a different server. Every check leaves a crawlcheck-continuity-record in pf.records with the baseline, what was observed, each event, the policy's sha256, the outcome and its own record_sha256. continuity_check(lock, tools, mode=...) and continuity_failure(cls, mode=...) run the same logic without a session and match the JS SDK byte for byte. A record states interface continuity at the check point: not code attestation, not a judgement that a tool is harmless, not user authorisation.

Catch a split view

Every Resolve answer carries its place in CrawlCheck's transparency log. The client keeps every signed tree head it sees in client.log (plain JSON; persist it and pass it back as log_store=) and treats two heads for one batch, or a head that does not chain to the one before, as a fork with both signed heads as proof. guard() returns False while the client holds one.

import json, os
from crawlcheck import CrawlCheck, new_log_store
store = json.load(open("cc-log.json")) if os.path.exists("cc-log.json") else new_log_store()
client = CrawlCheck(log_store=store)
client.resolve("example.com")
r = client.log_check()          # current head, GitHub's witness co-signature (RS256, checked here), the chain down to your heads
print(r["summary"])
json.dump(client.log, open("cc-log.json", "w"))

Use the site's staple

v = client.resolve_for("shop.example.com", headers=response_headers)   # or staple="v=ccr1;..."
v["source"]                 # "staple" (no request made) or "network"
v["decisions"]["transact"]

An expired, foreign or tampered staple is ignored and the signed answer is fetched instead. staple_check(record, host, published_kids) does the offline check on its own. Spec: https://crawlcheck.io/spec/staple

Hold the revocation list offline

import json, os
from crawlcheck import CrawlCheck, new_revocation_store
store = json.load(open("cc-revocations.json")) if os.path.exists("cc-revocations.json") else new_revocation_store()
client = CrawlCheck(revocation_store=store)
client.sync_revocations()                       # signed, verified, a ?since= delta after the first sync
json.dump(client.revocations, open("cc-revocations.json", "w"))
ok, _ = client.guard("pay.example.com", "transact")
client.last_guard.get("revoked")                # set when the held list refused it - no request was made

Owner incidents, suspected impersonations and unconfirmed high-risk field changes are listed. revoked_for(store, domain, action) is the same check on its own; reading and citing are never revoked by this list.

Look up without naming the domain

CrawlCheck(private_lookups=True) makes resolve() and resolve_for() send only the first 2 hex characters of sha256(domain) and verify the matching answer locally. A domain with no public measurement raises CrawlCheckError (404) instead of a just-in-time answer. preflight() and guard() still send the domain: a receipt is bound to it.

Metadata

Release files for crawlcheck 1.12.0

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for crawlcheck 1.12.0
File Size Uploaded Provenance Actions
crawlcheck-1.12.0.tar.gz 60.2 kB With provenance Details for crawlcheck-1.12.0.tar.gz

Built distribution (wheel)

Table of built distributions (wheels) for crawlcheck 1.12.0
File Interpreter ABI Platform Actions
crawlcheck-1.12.0-py3-none-any.whl Python 3 none any Details for crawlcheck-1.12.0-py3-none-any.whl

Total release size: 111.9 kB

Release files / crawlcheck-1.12.0.tar.gz

Download URL crawlcheck-1.12.0.tar.gz
Size 60.2 kB
Tags Source
SHA-256 checksum
How to use checksums
474fd7771e489284d7f7289381647031b842aa3cff44473382bff82647a7ec61
BLAKE2b-256 checksum
How to use checksums
6e120befeb1c2e186633b81d284f6ae66deb5c41044c2f1f553950c7e8117b98
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
Yes
Uploaded via twine/7.0.0 CPython/3.13.16

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Nov 7, 2026.

Transparency log

Release files / crawlcheck-1.12.0-py3-none-any.whl

Download URL crawlcheck-1.12.0-py3-none-any.whl
Size 51.7 kB
Tags Python 3
SHA-256 checksum
How to use checksums
7ef87fd44cd326168df44e27bc64a04faa50efd26c29c3e213bd7528da7316da
BLAKE2b-256 checksum
How to use checksums
637271a01885e7a5a941f717ab2d97bd12eb28af4caeaf52d8464d0317484019
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
Yes
Uploaded via twine/7.0.0 CPython/3.13.16

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Nov 7, 2026.

Transparency log

Security for crawlcheck 1.12.0

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

Full provenance

Both files in this release have been uploaded with provenance metadata.

Select from 2 files

Attestation for crawlcheck-1.12.0.tar.gz

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Nov 7, 2026.

Transparency log

A note on package security

PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.

To learn more about protecting yourself from malicious actors, visit our package security documentation.

Release history Release notifications | RSS feed

This release

1.12.0 This release

2 release files Full provenance

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page