scrapy-calyprium

Anti-detection Scrapy middleware for web scraping — proxy routing and stealth browser rendering powered by Calyprium.

Install

pip install scrapy-calyprium

Quick Start

# settings.py
import scrapy_calyprium

scrapy_calyprium.configure(api_key="clp_your_key_here")

This auto-configures:

VeilProxyMiddleware — routes requests through rotating proxies with TLS fingerprinting
MimicBrowserMiddleware — renders JavaScript pages with stealth browser instances
S3 feed storage — write spider output to Calyprium storage using Scrapy's built-in S3FeedStorage

Usage

Automatic Configuration (recommended)

# settings.py
import scrapy_calyprium

scrapy_calyprium.configure(
    api_key="clp_your_key_here",
    mimic_stealth_level="maximum",  # basic, moderate, maximum
)

Manual Configuration

# settings.py
DOWNLOADER_MIDDLEWARES = {
    "scrapy_calyprium.VeilProxyMiddleware": 100,
    "scrapy_calyprium.MimicBrowserMiddleware": 200,
}

CALYPRIUM_API_KEY = "clp_your_key_here"
VEIL_USER_ID = "your-user-id"

Request Tracing

CalypriumTraceMiddleware (added by configure()) gives every request one trace: the wait in Scrapy's queue, each download attempt, the challenge solve if there is one, and the callback that parses the page. The SDK's calls to Mimic and Tessera carry the trace (traceparent), so their steps join it. On the Calyprium platform the trace of every request of a run is shown with its run. To set it up by hand, register it as both a downloader and a spider middleware:

DOWNLOADER_MIDDLEWARES = {"scrapy_calyprium.CalypriumTraceMiddleware": 50}
SPIDER_MIDDLEWARES = {"scrapy_calyprium.CalypriumTraceMiddleware": 50}

Failure Diagnosis and Retries

Every download attempt gets a diagnosis from the fetch layer's shared failure classifier (the same one Mimic and Tessera use), in response.meta["calyprium_diagnosis"]: {"class", "number", "side", "move", "vendor"?, "status"?, "evidence"?, "confidence"?, "retry_after"?}. A transport error leaves it on failure.request.meta for an errback. When Mimic's own call failed and the request fell back to a plain download, meta["calyprium_fallback_diagnosis"] and meta["calyprium_fetch_error"] say why.

MimicBrowserMiddleware retries an attempt when its diagnosis's move is worth another try: retry_elsewhere and climb at once, retry_later after CALYPRIUM_RETRY_DELAY, slow_down after a doubling backoff (a Retry-After wins), fix_ours once, and never a stop (not found, sign-in / geo / consent walls, policy). A retry carries meta["calyprium_attempt_number"] and meta["calyprium_prior_class"], which Mimic receives to resume its ladder. meta["dont_retry"] and meta["max_retry_times"] work as with Scrapy's RetryMiddleware; with that middleware enabled, the statuses in RETRY_HTTP_CODES are left to it. Stats: calyprium/retry/<class> per retry, calyprium/class/<class> per final response.

Saving Output to Calyprium Storage

Spider output is saved to Calyprium's S3-compatible storage using Scrapy's built-in feed export:

# settings.py
import scrapy_calyprium

scrapy_calyprium.configure(api_key="clp_your_key_here")

FEEDS = {
    "s3://calyprium/my-spider/%(time)s.jl": {
        "format": "jsonlines",
    },
}

The S3 credentials are auto-configured by configure() — no additional setup needed.

Browser Rendering

Mark requests that need JavaScript rendering:

import scrapy

class MySpider(scrapy.Spider):
    name = "example"

    def start_requests(self):
        # Regular request (proxy only)
        yield scrapy.Request("https://example.com")

        # Browser-rendered request
        yield scrapy.Request(
            "https://example.com/spa",
            meta={"mimic": True},
        )

Authentication

All middleware requires a valid API key. Set it via:

scrapy_calyprium.configure(api_key="clp_...")
CALYPRIUM_API_KEY environment variable

Settings Reference

Setting	Description	Default
`CALYPRIUM_API_KEY`	API key for all services	—
`VEIL_GATEWAY_URL`	Proxy gateway URL	`https://proxy.calyprium.com`
`VEIL_USER_ID`	User ID for proxy routing	—
`VEIL_PROFILE`	Proxy routing profile	—
`VEIL_PROXY_TYPE`	`datacenter`, `residential`, `residential_rotating`	—
`MIMIC_SERVICE_URL`	Mimic browser service URL	`https://mimic.calyprium.com`
`MIMIC_STEALTH_LEVEL`	`basic`, `moderate`, `maximum`	`moderate`
`MIMIC_BROWSER_ENGINE`	Specific browser engine	auto
`MIMIC_USE_PROXY`	Route browser through proxy	`False`
`MIMIC_ALL_REQUESTS`	Render all requests via browser	`False`
`MIMIC_USE_SPECTRE`	Use device fingerprints	`True`
`CALYPRIUM_RETRY_ENABLED`	Retry attempts whose diagnosis says so	`True`
`CALYPRIUM_RETRY_TIMES`	Retries per request	`2`
`CALYPRIUM_RETRY_DELAY`	Seconds before a `retry_later`; base of the `slow_down` backoff	`5`
`CALYPRIUM_RETRY_MAX_DELAY`	Cap on any retry delay (seconds)	`60`
`PRISM_WAIT_FOR_SCAN_S`	`PrismSitemapSpider`: how long to wait for Prism to index an unknown domain before closing as `prism_no_urls` (ends 30 s before `CLOSESPIDER_TIMEOUT`)	`900`

Development

src/scrapy_calyprium/routing/diagnosis.py is vendored: it is Mimic's failure classifier, copied verbatim from the Calyprium monorepo (stealth/mimic/mimic/routing/diagnosis.py). Never edit it here; when Mimic changes it, copy the file again (its VERSION constant says which contract it is) and run the tests. SDK-side adaptations live in routing/block_detect.py.

License

MIT

Metadata

Release files for scrapy-calyprium 1.35.0

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for scrapy-calyprium 1.35.0
File	Size	Uploaded	Provenance
scrapy_calyprium-1.35.0.tar.gz	200.1 kB	Oct 19, 2026	With provenance	Details

Built distribution (wheel)

Table of built distributions (wheels) for scrapy-calyprium 1.35.0
File	Interpreter	ABI	Platform	Reset
scrapy_calyprium-1.35.0-py3-none-any.whl 139.5 kB Oct 19, 2026 With provenance	Python 3	none	any	Details

Total release size: 339.6 kB

Release files / scrapy_calyprium-1.35.0.tar.gz

Download URL	scrapy_calyprium-1.35.0.tar.gz
Size	200.1 kB
Tags	Source
SHA-256 checksum How to use checksums	`4e188ac259da2e0a1542c035d9e85e7c845976583becd0d0f472bf4809b77de2`
BLAKE2b-256 checksum How to use checksums	`3c8a6bbb69fed208b1dc359a66515d4b78b5693eb6c6744d54b78d587f667437`
Upload date	Oct 19, 2026
Uploaded using Trusted Publishing? What is trusted publishing?	Yes
Uploaded via	`twine/7.0.0 CPython/3.13.14`

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

Identity

Publishing platform: GitHub Actions
Publishing repository: github.com/Aarkc/scrapy-calyprium
Publishing commit: github.com/Aarkc/scrapy-calyprium/tree/b050457caf562707164bef3d162e6ef575cd741a

Workflow

Publishing configuration: github.com/Aarkc/scrapy-calyprium/blob/b050457caf562707164bef3d162e6ef575cd741a/.github/workflows/publish.yml
Publishing logs: github.com/Aarkc/scrapy-calyprium/actions/runs/39262019265/attempts/1

Artifact

Subject: scrapy_calyprium-1.35.0.tar.gz
SHA-256 checksum: 4e188ac259da2e0a1542c035d9e85e7c845976583becd0d0f472bf4809b77de2 Verifying attestations

Release files / scrapy_calyprium-1.35.0-py3-none-any.whl

Download URL	scrapy_calyprium-1.35.0-py3-none-any.whl
Size	139.5 kB
Tags	Python 3
SHA-256 checksum How to use checksums	`6c23172bcae28226d341dea0bf3d51e45ccd83a63a5bbe461c8921492336c04e`
BLAKE2b-256 checksum How to use checksums	`d5d3067d5b24697202f608ab2d1df49b3686ee7b824ffc7fc757d3762ca7ae2f`
Upload date	Oct 19, 2026
Uploaded using Trusted Publishing? What is trusted publishing?	Yes
Uploaded via	`twine/7.0.0 CPython/3.13.14`

Provenance

Identity

Publishing platform: GitHub Actions
Publishing repository: github.com/Aarkc/scrapy-calyprium
Publishing commit: github.com/Aarkc/scrapy-calyprium/tree/b050457caf562707164bef3d162e6ef575cd741a

Workflow

Publishing configuration: github.com/Aarkc/scrapy-calyprium/blob/b050457caf562707164bef3d162e6ef575cd741a/.github/workflows/publish.yml
Publishing logs: github.com/Aarkc/scrapy-calyprium/actions/runs/39262019265/attempts/1

Artifact

Subject: scrapy_calyprium-1.35.0-py3-none-any.whl
SHA-256 checksum: 6c23172bcae28226d341dea0bf3d51e45ccd83a63a5bbe461c8921492336c04e Verifying attestations

Security for scrapy-calyprium 1.35.0

Report project as malware

Provenance

Full provenance

All 2 files in this release have been uploaded with provenance metadata.

Attestation for scrapy_calyprium-1.35.0.tar.gz

Identity

Publishing platform: GitHub Actions
Publishing repository: github.com/Aarkc/scrapy-calyprium
Publishing commit: github.com/Aarkc/scrapy-calyprium/tree/b050457caf562707164bef3d162e6ef575cd741a

Workflow

Publishing configuration: github.com/Aarkc/scrapy-calyprium/blob/b050457caf562707164bef3d162e6ef575cd741a/.github/workflows/publish.yml
Publishing logs: github.com/Aarkc/scrapy-calyprium/actions/runs/39262019265/attempts/1

Artifact

Subject: scrapy_calyprium-1.35.0.tar.gz
SHA-256 checksum: 4e188ac259da2e0a1542c035d9e85e7c845976583becd0d0f472bf4809b77de2 Verifying attestations

A note on package security

PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.