scrapy-calyprium

Anti-detection Scrapy middleware for web scraping — proxy routing and stealth browser rendering powered by Calyprium.

Install

pip install scrapy-calyprium

Quick Start

# settings.py
import scrapy_calyprium

scrapy_calyprium.configure(api_key="clp_your_key_here")

This auto-configures:

• VeilProxyMiddleware — routes requests through rotating proxies with TLS fingerprinting
• MimicBrowserMiddleware — renders JavaScript pages with stealth browser instances
• S3 feed storage — write spider output to Calyprium storage using Scrapy's built-in S3FeedStorage

Usage

Automatic Configuration (recommended)

# settings.py
import scrapy_calyprium

scrapy_calyprium.configure(
    api_key="clp_your_key_here",
    mimic_stealth_level="maximum",  # basic, moderate, maximum
)

Manual Configuration

# settings.py
DOWNLOADER_MIDDLEWARES = {
    "scrapy_calyprium.VeilProxyMiddleware": 100,
    "scrapy_calyprium.MimicBrowserMiddleware": 200,
}

CALYPRIUM_API_KEY = "clp_your_key_here"
VEIL_USER_ID = "your-user-id"

Request Tracing

CalypriumTraceMiddleware (added by configure()) gives every request one trace: the wait in Scrapy's queue, each download attempt, the challenge solve if there is one, and the callback that parses the page. The SDK's calls to Mimic and Tessera carry the trace (traceparent), so their steps join it. On the Calyprium platform the trace of every request of a run is shown with its run. To set it up by hand, register it as both a downloader and a spider middleware:

DOWNLOADER_MIDDLEWARES = {"scrapy_calyprium.CalypriumTraceMiddleware": 50}
SPIDER_MIDDLEWARES = {"scrapy_calyprium.CalypriumTraceMiddleware": 50}

Failure Diagnosis and Retries

Every download attempt gets a diagnosis from the fetch layer's shared failure classifier (the same one Mimic and Tessera use), in response.meta["calyprium_diagnosis"]: {"class", "number", "side", "move", "vendor"?, "status"?, "evidence"?, "confidence"?, "retry_after"?}. A transport error leaves it on failure.request.meta for an errback. When Mimic's own call failed and the request fell back to a plain download, meta["calyprium_fallback_diagnosis"] and meta["calyprium_fetch_error"] say why.

MimicBrowserMiddleware retries an attempt when its diagnosis's move is worth another try: retry_elsewhere and climb at once, retry_later after CALYPRIUM_RETRY_DELAY, slow_down after a doubling backoff (a Retry-After wins), fix_ours once, and never a stop (not found, sign-in / geo / consent walls, policy). A retry carries meta["calyprium_attempt_number"] and meta["calyprium_prior_class"], which Mimic receives to resume its ladder. meta["dont_retry"] and meta["max_retry_times"] work as with Scrapy's RetryMiddleware; with that middleware enabled, the statuses in RETRY_HTTP_CODES are left to it. Stats: calyprium/retry/<class> per retry, calyprium/class/<class> per final response.

Saving Output to Calyprium Storage

Spider output is saved to Calyprium's S3-compatible storage using Scrapy's built-in feed export:

# settings.py
import scrapy_calyprium

scrapy_calyprium.configure(api_key="clp_your_key_here")

FEEDS = {
    "s3://calyprium/my-spider/%(time)s.jl": {
        "format": "jsonlines",
    },
}

The S3 credentials are auto-configured by configure() — no additional setup needed.

Browser Rendering

Mark requests that need JavaScript rendering:

import scrapy

class MySpider(scrapy.Spider):
    name = "example"

    def start_requests(self):
        # Regular request (proxy only)
        yield scrapy.Request("https://example.com")

        # Browser-rendered request
        yield scrapy.Request(
            "https://example.com/spa",
            meta={"mimic": True},
        )

Browser-Made Form Submits

Some sites refuse a form POST that their own page did not make (F5 BIG-IP's bot defence adds per-request tokens in the page just before its search postback). Mark such a request and a real browser page submits it: Mimic loads the page on the run's exit, types the fields, clicks the control that submits, and the spider gets the page after the submit as a normal response. The page is kept for the run, so a follow-up postback (the next result page) acts on the results the last one left:

yield scrapy.FormRequest(url, formdata=fields, callback=self.parse_results,
                         meta={"mimic_browser_submit": {"submit": "ctl00$MainContent$cmdSearch"}})
yield scrapy.Request(url, callback=self.parse_results, dont_filter=True,
                     meta={"mimic_browser_submit": {"submit": "ctl00$MainContent$lbtnNextHeader",
                                                    "navigate": "never"}})

A captcha or challenge page comes back as it is (Mimic closes that browser and never solves it), a submit the page did not apply comes back empty, and a browser submit is never re-sent by the middleware. meta["mimic_submit"] says how it went. Options and the MIMIC_SUBMIT_* settings: scrapy_calyprium/_browser_submit.py.

Authentication

All middleware requires a valid API key. Set it via:

1. scrapy_calyprium.configure(api_key="clp_...")
2. CALYPRIUM_API_KEY environment variable

Settings Reference

Development

src/scrapy_calyprium/routing/diagnosis.py is vendored: it is Mimic's failure classifier, copied verbatim from the Calyprium monorepo (stealth/mimic/mimic/routing/diagnosis.py). Never edit it here; when Mimic changes it, copy the file again (its VERSION constant says which contract it is) and run the tests. SDK-side adaptations live in routing/block_detect.py.

License

MIT