ForgeMesh Pay-per-Crawl for Python

Charge AI crawlers and bots per page with x402 USDC micropayments on Base, paid straight to your wallet. Humans and verified search engines read free. Middleware for FastAPI / Starlette (ASGI), Flask and any WSGI app, and Django, on one framework-free core.

This is the Python edition of the WordPress plugin; behaviour is identical.

• Humans see nothing different. Browsers get the page as always.
• Verified search engines stay free. Googlebot, Bingbot, DuckDuckBot, Applebot and others are checked by reverse DNS, so SEO is untouched. A crawler that only claims to be Googlebot is treated as a script.
• AI crawlers, SEO crawlers and scripts get 402 Payment Required with an x402 challenge (v2 PAYMENT-REQUIRED header + v1 JSON body) and an MPP WWW-Authenticate: Payment challenge. They pay USDC on Base and retry; the page is served only after the payment settles on-chain.
• You keep 100%. The payer's USDC goes directly to your wallet. No custody, no account, no API key, no revenue share.
• Discovery built in. Serves /.well-known/x402.json, /llms.txt and /agents.txt.

Install

pip install forgemesh-pay-per-crawl

Python 3.10+. One dependency (pycryptodome, for keccak256). httpx is used for facilitator calls if installed, otherwise the standard library. Start with mode="monitor" (records what bots would have paid, never blocks), then switch to mode="gate".

FastAPI / Starlette

from forgemesh_pay_per_crawl import Config
from forgemesh_pay_per_crawl.asgi import PayPerCrawlMiddleware

app.add_middleware(PayPerCrawlMiddleware,
    config=Config(pay_to="0xYourWalletOnBase", mode="gate", price_usd=0.01, base_url="https://example.com"))

Flask (and any WSGI app)

from forgemesh_pay_per_crawl import Config
from forgemesh_pay_per_crawl.wsgi import PayPerCrawlWSGI

app.wsgi_app = PayPerCrawlWSGI(app.wsgi_app,
    Config(pay_to="0xYourWalletOnBase", mode="gate", price_usd=0.01, base_url="https://example.com"))

Django

# settings.py
MIDDLEWARE = ["forgemesh_pay_per_crawl.django.PayPerCrawlMiddleware", *MIDDLEWARE]  # near the top
PAY_PER_CRAWL = {"pay_to": "0xYourWalletOnBase", "mode": "gate", "price_usd": 0.01, "base_url": "https://example.com"}

Settings

All are Config(...) keyword arguments (or keys of PAY_PER_CRAWL).

Bot classes: human, search, social, monitor, seo, ai, automation, bot. The catalog covers 85 crawlers and tools; see core/catalog.py.

Multiple worker processes: pass lock=FileNonceLock("/var/tmp/fmppc") to Gate(...) (and use PayPerCrawlMiddleware(app, gate=gate)) so a payment authorization cannot be replayed on another worker, and set a stable mpp_secret. The default lock is in-process memory.

Behind Cloudflare

If your site is proxied by Cloudflare, set AI Crawl Control (and any Bot Fight Mode / AI Labyrinth / "Block AI bots" rule) to Allow for the crawlers you want to charge. Cloudflare answers those requests at the edge before they ever reach your app, so the gate would never see them. Set trust_proxy_header="CF-Connecting-IP" and base_url to your public origin.

How a payment works

1. A bot requests a page and gets 402 with the price, asset (USDC on Base) and your wallet, in the PAYMENT-REQUIRED header (x402 v2), the JSON body (x402 v1, with invalidReason when a payment was rejected) and WWW-Authenticate: Payment (MPP).
2. The bot signs a USDC transfer authorization (EIP-3009) and retries with it in PAYMENT-SIGNATURE (v2), X-PAYMENT (v1) or Authorization: Payment (MPP).
3. The middleware checks recipient, amount and expiry locally, locks the authorization nonce against replay, then posts it to the facilitator (/verify, then /settle). The facilitator verifies and submits it on-chain; it never holds funds. v1 payments are forwarded in v1 shape.
4. The page is served only when a transaction hash comes back, with the receipt in PAYMENT-RESPONSE, X-PAYMENT-RESPONSE and (MPP) Payment-Receipt.

The package never contacts ForgeMesh. Its only outbound requests are to the facilitator you chose, and only when a payment is presented. (The optional Pro reporter below is the one exception, and is off without a key.)

Money flow

The payer's USDC moves wallet to wallet on Base, from the payer to the pay_to address you configured. There is no ForgeMesh account, no custody, no settlement service and no revenue share: the site owner keeps 100%. The facilitator (PayAI by default) charges the seller no fee at the time of writing; check your facilitator's terms.

Using the pieces directly

from forgemesh_pay_per_crawl import Classifier, Config, Gate, Request, X402, build_challenge

Classifier().classify("Mozilla/5.0 (compatible; GPTBot/1.2)").cls        # "ai"
gate = Gate(Config(pay_to="0x...", mode="gate"))
decision = gate.handle(Request(method="GET", path="/x", headers={"user-agent": "curl/8"}, host="example.com"))
decision.response   # a Response (status, headers, body) to send, or None to serve the page
decision.headers    # headers to add to the page you serve (payment receipt, Link)

Gate.handle is blocking (it can call the facilitator); in async code run it in a thread, as the ASGI middleware does. The facilitator transport is injectable (Gate(config, transport=fn) with fn(url, body, headers) -> dict).

Pro (optional)

With a ForgeMesh Pay-per-Crawl Pro license (pro_key="fmppc_...") the package reports daily bot rollups and settlements to the hosted dashboard at proxy.forgemesh.io and pulls the crawler-intel feed into the classifier. Nothing is sent without a key. The rollups are held in memory per process, so with several workers each reports its own counts.

Verified

• Unit and integration tests: pytest (keccak vectors incl. block-boundary cases, classifier with rDNS, 402 shapes, MPP challenge/credential round trip, facilitator flow, replay, HEAD, file lock, Pro sync, all three adapters, and an integration matrix against live uvicorn/FastAPI and Flask servers with a mock facilitator).
• Real settlements on Base mainnet through PayAI, October 2026, from the FastAPI example gated at $0.01: x402 v1 (tx) and x402 v2 (tx).

Development

python -m venv .venv && .venv/bin/pip install -e ".[test]"
.venv/bin/pytest
python -m build

MIT. Made by ForgeMesh Labs (hello@forgemesh.io).