scrapy-calyprium

Anti-detection Scrapy middleware for web scraping — proxy routing and stealth browser rendering powered by Calyprium.

Install

pip install scrapy-calyprium

Quick Start

# settings.py
import scrapy_calyprium

scrapy_calyprium.configure(api_key="clp_your_key_here")

This auto-configures:

• VeilProxyMiddleware — routes requests through rotating proxies with TLS fingerprinting
• MimicBrowserMiddleware — renders JavaScript pages with stealth browser instances
• PaceMiddleware / PoolBusyMiddleware — an optional pages-per-minute pace, and backpressure when the shared proxy pool is busy
• S3 feed storage — write spider output to Calyprium storage using Scrapy's built-in S3FeedStorage

Usage

Automatic Configuration (recommended)

# settings.py
import scrapy_calyprium

scrapy_calyprium.configure(
    api_key="clp_your_key_here",
    mimic_stealth_level="maximum",  # basic, moderate, maximum
)

Manual Configuration

# settings.py
DOWNLOADER_MIDDLEWARES = {
    "scrapy_calyprium.VeilProxyMiddleware": 100,
    "scrapy_calyprium.MimicBrowserMiddleware": 200,
}

CALYPRIUM_API_KEY = "clp_your_key_here"
VEIL_USER_ID = "your-user-id"

Request Tracing

CalypriumTraceMiddleware (added by configure()) gives every request one trace: the wait in Scrapy's queue, each download attempt, the challenge solve if there is one, and the callback that parses the page. The SDK's calls to Mimic and Tessera carry the trace (traceparent), so their steps join it. On the Calyprium platform the trace of every request of a run is shown with its run. To set it up by hand, register it as both a downloader and a spider middleware:

DOWNLOADER_MIDDLEWARES = {"scrapy_calyprium.CalypriumTraceMiddleware": 50}
SPIDER_MIDDLEWARES = {"scrapy_calyprium.CalypriumTraceMiddleware": 50}

Failure Diagnosis and Retries

Every download attempt gets a diagnosis from the fetch layer's shared failure classifier (the same one Mimic and Tessera use), in response.meta["calyprium_diagnosis"]: {"class", "number", "side", "move", "vendor"?, "status"?, "evidence"?, "confidence"?, "retry_after"?}. A transport error leaves it on failure.request.meta for an errback. When Mimic's own call failed and the request fell back to a plain download, meta["calyprium_fallback_diagnosis"] and meta["calyprium_fetch_error"] say why.

MimicBrowserMiddleware retries an attempt when its diagnosis's move is worth another try: retry_elsewhere and climb at once, retry_later after CALYPRIUM_RETRY_DELAY, slow_down after a doubling backoff (a Retry-After wins), fix_ours once, and never a stop (not found, sign-in / geo / consent walls, policy). A retry carries meta["calyprium_attempt_number"] and meta["calyprium_prior_class"], which Mimic receives to resume its ladder. meta["dont_retry"] and meta["max_retry_times"] work as with Scrapy's RetryMiddleware; with that middleware enabled, the statuses in RETRY_HTTP_CODES are left to it. Stats: calyprium/retry/<class> per retry, calyprium/class/<class> per final response.

Pace and a Shared Proxy Pool

configure() adds two downloader middlewares from scrapy_calyprium.middleware.portfolio:

• PaceMiddleware (120) holds the whole spider, all domains together, to CALYPRIUM_PACE_PPM pages a minute (a token bucket of CALYPRIUM_PACE_BURST tokens). It paces every attempt, including the ones MimicBrowserMiddleware answers itself on the local route, where DOWNLOAD_DELAY and the per-domain concurrency do not apply. Without the setting it is not installed. Stats: calyprium/pace/delayed, calyprium/pace/wait_seconds.
• PoolBusyMiddleware (950) handles the Calyprium gateway's pool_busy answer. The gateway sends it when the shared proxy pool has no free connection: 503 Service Unavailable (pool_busy) with X-Calyprium-Proxy-Error: pool_busy, which Scrapy reports as a TunnelError for HTTPS and the local route as PoolBusyError. It is not the site's answer, so the request goes back in the queue after a 2-10 s jittered backoff and keeps its retries (retry_times and the Calyprium retry's count). Nothing solves, falls back or blames a cookie slot, exit or domain, and the run's stats do not count the refusal as a failure. After CALYPRIUM_POOL_BUSY_MAX_REQUEUES refusals of one request, the usual error handling takes over. With CALYPRIUM_POOL_AIMD on, each refusal cuts CONCURRENT_REQUESTS to 3/4, at most once every 5 s and never below 1. Every 20 responses in a row without a refusal add 1 back, up to the configured value. Stats: calyprium/pool_busy, calyprium/pool_busy/requeued, calyprium/pool_busy/max_reached, calyprium/pool_busy/wait_seconds, calyprium/pool_busy/concurrency.

DOWNLOADER_MIDDLEWARES = {
    "scrapy_calyprium.PaceMiddleware": 120,
    "scrapy_calyprium.PoolBusyMiddleware": 950,
}
CALYPRIUM_PACE_PPM = 30  # optional: one page every 2 s

Reporting Progress

A spider that knows how much work it has can say so, and Calyprium shows how long its run has left (the work left divided by the recent pace of the work done). Call progress.report whenever the counts change; both are in one unit:

from scrapy_calyprium import progress

progress.report(self, done=pages_done, left=pages_left, unit="companies")

PrismSitemapSpider reports its place in Prism's URL list by itself. A spider that reports nothing gets an estimate from its earlier runs instead.

Saving Output to Calyprium Storage

Spider output is saved to Calyprium's S3-compatible storage using Scrapy's built-in feed export:

# settings.py
import scrapy_calyprium

scrapy_calyprium.configure(api_key="clp_your_key_here")

FEEDS = {
    "s3://calyprium/my-spider/%(time)s.jl": {
        "format": "jsonlines",
    },
}

The S3 credentials are auto-configured by configure() — no additional setup needed.

Browser Rendering

Mark requests that need JavaScript rendering:

import scrapy

class MySpider(scrapy.Spider):
    name = "example"

    def start_requests(self):
        # Regular request (proxy only)
        yield scrapy.Request("https://example.com")

        # Browser-rendered request
        yield scrapy.Request(
            "https://example.com/spa",
            meta={"mimic": True},
        )

Browser-Made Form Submits

Some sites refuse a form POST that their own page did not make (F5 BIG-IP's bot defence adds per-request tokens in the page just before its search postback). Mark such a request and a real browser page submits it: Mimic loads the page on the run's exit, types the fields, clicks the control that submits, and the spider gets the page after the submit as a normal response. The page is kept for the run, so a follow-up postback (the next result page) acts on the results the last one left:

yield scrapy.FormRequest(url, formdata=fields, callback=self.parse_results,
                         meta={"mimic_browser_submit": {"submit": "ctl00$MainContent$cmdSearch"}})
yield scrapy.Request(url, callback=self.parse_results, dont_filter=True,
                     meta={"mimic_browser_submit": {"submit": "ctl00$MainContent$lbtnNextHeader",
                                                    "navigate": "never"}})

A captcha or challenge page comes back as it is (Mimic closes that browser and never solves it), a submit the page did not apply comes back empty, and a browser submit is never re-sent by the middleware. meta["mimic_submit"] says how it went. Options and the MIMIC_SUBMIT_* settings: scrapy_calyprium/_browser_submit.py.

Authentication

All middleware requires a valid API key. Set it via:

1. scrapy_calyprium.configure(api_key="clp_...")
2. CALYPRIUM_API_KEY environment variable

Settings Reference

Development

src/scrapy_calyprium/routing/diagnosis.py is vendored: it is Mimic's failure classifier, copied verbatim from the Calyprium monorepo (stealth/mimic/mimic/routing/diagnosis.py). Never edit it here; when Mimic changes it, copy the file again (its VERSION constant says which contract it is) and run the tests. SDK-side adaptations live in routing/block_detect.py.

License

MIT