scrapy-calyprium
Anti-detection Scrapy middleware for web scraping — proxy routing and stealth browser rendering powered by Calyprium.
Install
pip install scrapy-calyprium
Quick Start
# settings.py
import scrapy_calyprium
scrapy_calyprium.configure(api_key="clp_your_key_here")
This auto-configures:
- VeilProxyMiddleware — routes requests through rotating proxies with TLS fingerprinting
- MimicBrowserMiddleware — renders JavaScript pages with stealth browser instances
- S3 feed storage — write spider output to Calyprium storage using Scrapy's built-in
S3FeedStorage
Usage
Automatic Configuration (recommended)
# settings.py
import scrapy_calyprium
scrapy_calyprium.configure(
api_key="clp_your_key_here",
mimic_stealth_level="maximum", # basic, moderate, maximum
)
Manual Configuration
# settings.py
DOWNLOADER_MIDDLEWARES = {
"scrapy_calyprium.VeilProxyMiddleware": 100,
"scrapy_calyprium.MimicBrowserMiddleware": 200,
}
CALYPRIUM_API_KEY = "clp_your_key_here"
VEIL_USER_ID = "your-user-id"
Request Tracing
CalypriumTraceMiddleware (added by configure()) gives every request one trace: the wait in Scrapy's queue,
each download attempt, the challenge solve if there is one, and the callback that parses the page. The SDK's calls
to Mimic and Tessera carry the trace (traceparent), so their steps join it. On the Calyprium platform the
trace of every request of a run is shown with its run. To set it up by hand, register it as both a downloader and
a spider middleware:
DOWNLOADER_MIDDLEWARES = {"scrapy_calyprium.CalypriumTraceMiddleware": 50}
SPIDER_MIDDLEWARES = {"scrapy_calyprium.CalypriumTraceMiddleware": 50}
Saving Output to Calyprium Storage
Spider output is saved to Calyprium's S3-compatible storage using Scrapy's built-in feed export:
# settings.py
import scrapy_calyprium
scrapy_calyprium.configure(api_key="clp_your_key_here")
FEEDS = {
"s3://calyprium/my-spider/%(time)s.jl": {
"format": "jsonlines",
},
}
The S3 credentials are auto-configured by configure() — no additional setup needed.
Browser Rendering
Mark requests that need JavaScript rendering:
import scrapy
class MySpider(scrapy.Spider):
name = "example"
def start_requests(self):
# Regular request (proxy only)
yield scrapy.Request("https://example.com")
# Browser-rendered request
yield scrapy.Request(
"https://example.com/spa",
meta={"mimic": True},
)
Authentication
All middleware requires a valid API key. Set it via:
scrapy_calyprium.configure(api_key="clp_...")CALYPRIUM_API_KEYenvironment variable
Settings Reference
| Setting | Description | Default |
|---|---|---|
CALYPRIUM_API_KEY |
API key for all services | — |
VEIL_GATEWAY_URL |
Proxy gateway URL | https://proxy.calyprium.com |
VEIL_USER_ID |
User ID for proxy routing | — |
VEIL_PROFILE |
Proxy routing profile | — |
VEIL_PROXY_TYPE |
datacenter, residential, residential_rotating |
— |
MIMIC_SERVICE_URL |
Mimic browser service URL | https://mimic.calyprium.com |
MIMIC_STEALTH_LEVEL |
basic, moderate, maximum |
moderate |
MIMIC_BROWSER_ENGINE |
Specific browser engine | auto |
MIMIC_USE_PROXY |
Route browser through proxy | False |
MIMIC_ALL_REQUESTS |
Render all requests via browser | False |
MIMIC_USE_SPECTRE |
Use device fingerprints | True |
License
MIT
1 maintainer
Metadata
1 maintainer
Release files for scrapy-calyprium 1.30.0
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | Provenance | |
|---|---|---|---|---|
| scrapy_calyprium-1.30.0.tar.gz | 152.8 kB | With provenance | Details |
Built distribution (wheel)
| File | Interpreter | ABI | Platform | Reset |
|---|---|---|---|---|
| scrapy_calyprium-1.30.0-py3-none-any.whl | Python 3 | none | any | Details |
Total release size: 265.7 kB
Release files / scrapy_calyprium-1.30.0.tar.gz
| Download URL | scrapy_calyprium-1.30.0.tar.gz |
|---|---|
| Size | 152.8 kB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
cd45260895bfeb02b871def151a3213b293a8d516ec76e07ca5991af1b74366d
|
|
BLAKE2b-256 checksum How to use checksums |
fc181faf1480e8f59ae97d9c0da43d12806241e9c6541b6f829b829b40007b8c
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
Yes |
| Uploaded via |
twine/7.0.0 CPython/3.13.14
|
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on Oct 17, 2026.
Transparency logRelease files / scrapy_calyprium-1.30.0-py3-none-any.whl
| Download URL | scrapy_calyprium-1.30.0-py3-none-any.whl |
|---|---|
| Size | 112.9 kB |
| Tags | Python 3 |
|
SHA-256 checksum How to use checksums |
be36fcbf8639aa33ba6de78439d41b8093a7464ca60bbb45576be7c2fdf775ed
|
|
BLAKE2b-256 checksum How to use checksums |
4c467c30e2b02b8c2db7747297524c5d76db6a5753b3b8d47045b50ceadc7218
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
Yes |
| Uploaded via |
twine/7.0.0 CPython/3.13.14
|
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on Oct 17, 2026.
Transparency logSecurity for scrapy-calyprium 1.30.0
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
Full provenance
All 2 files in this release have been uploaded with provenance metadata.
Attestation for scrapy_calyprium-1.30.0.tar.gz
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on Oct 17, 2026.
Transparency logA note on package security
PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.
