scrapy-turnstile

scrapy-turnstile is a Scrapy downloader middleware that solves Cloudflare Turnstile automatically. When a request hits a Turnstile-protected page, it detects the challenge, solves it through Peak, and re-issues the request with the token so your spider gets the real page.

Why

Scrapy makes plain HTTP requests. It has no JS engine, so a Turnstile-protected site returns a 403 or the "Just a moment..." interstitial instead of the content you asked for. The old scrapy-cloudflare-middleware only handled the legacy IUAM (I'm Under Attack Mode) JS challenge and is unmaintained; it does nothing for Turnstile, which is what Cloudflare now serves. cloudscraper has the same limitation: it can't solve Turnstile.

This middleware routes the Turnstile challenge to Peak, which returns a valid token. When your setup gets blocked, drop in a Peak API key and it just works.

Powered by Peak

This package uses Peak to solve Turnstile.

• Solve Cloudflare Turnstile & the 5s challenge in about a second
• $0.90 per 1,000 successful Turnstile solves, down to $0.35 at volume
• 1,000 free solves to start, no card. → Get your free API key · Docs · Pricing

Install

pip install scrapy-turnstile

Quickstart

Add the middleware to your project's settings.py and set your Peak key:

# settings.py
DOWNLOADER_MIDDLEWARES = {
    "scrapy_turnstile.TurnstileMiddleware": 585,
}

PEAK_API_KEY = "pk_your_api_key"   # better: read from an env var
# PEAK_PROXY = "http://user:pass@ip:port"   # optional, routed to Peak

Or per-spider:

import os
import scrapy


class ProtectedSpider(scrapy.Spider):
    name = "protected"
    start_urls = ["https://protected.example.com/"]

    custom_settings = {
        "DOWNLOADER_MIDDLEWARES": {
            "scrapy_turnstile.TurnstileMiddleware": 585,
        },
        "PEAK_API_KEY": os.environ["PEAK_API_KEY"],
    }

    def parse(self, response):
        # The challenge, if any, was already solved upstream.
        yield {"title": response.css("title::text").get()}

export PEAK_API_KEY=pk_your_api_key
scrapy runspider examples/quotes_spider.py

A runnable spider is in examples/quotes_spider.py.

How it works

1. process_response inspects every response for Turnstile markers (cf-turnstile, the challenges.cloudflare.com/turnstile script, the "Just a moment" interstitial).
2. On a match it extracts the data-sitekey and the request URL.
3. It calls Peak: POST https://api.peak.fo/solve with {"task_type": "turnstiletask", "sitekey": ..., "url": ...} and header X-API-Key.
4. It re-issues the original request with the returned token injected as the cf-turnstile-response form field, and also on request.meta["cf_turnstile_response"] for custom submits.
5. A meta flag stops the retried request from being re-processed, so there are no loops.

Peak also supports the Cloudflare 5s interstitial via task_type: "cloudflare5stask".

Settings

Earn with your app ID

Set PEAK_APP_ID in your Scrapy settings to earn 5% of every solve your spider makes, paid as Peak solve credit. It is optional and does not change whether a solve succeeds or how fast it returns.

# settings.py
PEAK_APP_ID = "app_your_id"   # optional; earns solve credit on each solve

Create an app id at peak.fo/dashboard/developer; details at peak.fo/earn.

Legitimate use

For automation, QA, and scraping public data you are allowed to access. Respect each target's Terms of Service and robots.txt, and rate-limit yourself. Do not use this for credential stuffing or to access data you have no right to.

License

MIT — see LICENSE.