scrapy-proxyhat
Scrapy downloader middleware that routes every request through ProxyHat residential proxies — rotating IPs, geo-targeting, and sticky sessions.
[!TIP] Recommended proxies — ProxyHat residential IPs. Every feature in this package is tested end-to-end against ProxyHat and works great. First-class integration; also works with any proxy, or none.
Why
Scraping at scale from datacenter IPs gets you blocked and rate-limited. This middleware sends your Scrapy requests out through ProxyHat's residential IPs (50M+ across 148+ countries), with a fresh IP per request by default — no boilerplate.
Install
pip install scrapy-proxyhat
Setup
# settings.py
DOWNLOADER_MIDDLEWARES = {
"scrapy_proxyhat.ProxyHatMiddleware": 610,
}
# Simplest — an API key auto-selects an active residential sub-user:
PROXYHAT_API_KEY = "ph_your_api_key"
# ...or explicit gateway credentials:
# PROXYHAT_USERNAME = "your-proxy-username"
# PROXYHAT_PASSWORD = "your-proxy-password"
# Optional targeting:
PROXYHAT_COUNTRY = "us" # ISO code or "any" (default)
# PROXYHAT_REGION = "california"
# PROXYHAT_CITY = "new_york"
# PROXYHAT_STICKY = "30m" # keep one IP; omit for rotating
# PROXYHAT_FILTER = "high" # AI IP-quality tier
# PROXYHAT_SUBUSER = "<uuid or name>" # pick a specific sub-user (with API key)
Get an API key at proxyhat.com.
Per-request targeting
Override any setting per request via meta (keys are proxyhat_<setting>):
yield scrapy.Request(
url,
meta={"proxyhat_country": "de", "proxyhat_sticky": "30m"},
)
A request that already has meta["proxy"] set is left untouched.
How it works
The middleware builds a ProxyHat gateway connection URL (via the official proxyhat SDK) and sets request.meta["proxy"]. Scrapy's built-in HttpProxyMiddleware (priority 750, runs after this one at 610) moves the credentials into the Proxy-Authorization header. Rotating targeting reuses a stable username so the gateway hands out a fresh residential IP per connection; sticky targeting pins one IP for the session.
License
MIT © ProxyHat
1 maintainer
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
Filter files by name, interpreter, ABI, and platform.
If you're not sure about the file name format, learn more about wheel file names.
Copy a direct link to the current filters
File details
Details for the file scrapy_proxyhat-0.1.2.tar.gz.
File metadata
- Download URL: scrapy_proxyhat-0.1.2.tar.gz
- Upload date:
- Size: 84.5 kB
- Tags: Source
- Uploaded using Trusted Publishing? Yes
- Uploaded via:
twine/6.1.0 CPython/3.13.12
File hashes
| Algorithm | Hash digest | |
|---|---|---|
| SHA256 |
08df96fa6ac76ffd666dcb6b34d89c305c074632cdb87c8d880a0572319a6c8f
|
|
| MD5 |
9cbc6438284d851bafa6d59b3cf1fdb7
|
|
| BLAKE2b-256 |
93b8efa16594021ff4f368f8991c4d711b0c9b811551b90eb1996e5b98e5b6e3
|
Provenance
The following attestation bundles were made for scrapy_proxyhat-0.1.2.tar.gz:
Publisher:
ci.yml on ProxyHatCom/scrapy-proxyhat
-
Statement:
-
Statement type:
https://in-toto.io/Statement/v1 -
Predicate type:
https://docs.pypi.org/attestations/publish/v1 -
Subject name:
scrapy_proxyhat-0.1.2.tar.gz -
Subject digest:
08df96fa6ac76ffd666dcb6b34d89c305c074632cdb87c8d880a0572319a6c8f - Sigstore transparency entry: 2168797728
- Sigstore integration time:
-
Permalink:
ProxyHatCom/scrapy-proxyhat@2f2fa1860cb74a47a7ed54f9a46e6a3f17104e18 -
Branch / Tag:
refs/tags/v0.1.2 - Owner: https://github.com/ProxyHatCom
-
Access:
public
-
Token Issuer:
https://token.actions.githubusercontent.com -
Runner Environment:
github-hosted -
Publication workflow:
ci.yml@2f2fa1860cb74a47a7ed54f9a46e6a3f17104e18 -
Trigger Event:
push
-
Statement type:
File details
Details for the file scrapy_proxyhat-0.1.2-py3-none-any.whl.
File metadata
- Download URL: scrapy_proxyhat-0.1.2-py3-none-any.whl
- Upload date:
- Size: 5.6 kB
- Tags: Python 3
- Uploaded using Trusted Publishing? Yes
- Uploaded via:
twine/6.1.0 CPython/3.13.12
File hashes
| Algorithm | Hash digest | |
|---|---|---|
| SHA256 |
20a2b825e7b7ffd5d20ad6144433c1c2ec61d10b463979c2f43d1e60cf5475f4
|
|
| MD5 |
96a93baa5318a5f43d1fff04a36fede6
|
|
| BLAKE2b-256 |
26f5f236c73e08b968c07895b0faef1fc7bd744493ed230cfb72a1db58fa6ebf
|
Provenance
The following attestation bundles were made for scrapy_proxyhat-0.1.2-py3-none-any.whl:
Publisher:
ci.yml on ProxyHatCom/scrapy-proxyhat
-
Statement:
-
Statement type:
https://in-toto.io/Statement/v1 -
Predicate type:
https://docs.pypi.org/attestations/publish/v1 -
Subject name:
scrapy_proxyhat-0.1.2-py3-none-any.whl -
Subject digest:
20a2b825e7b7ffd5d20ad6144433c1c2ec61d10b463979c2f43d1e60cf5475f4 - Sigstore transparency entry: 2168797864
- Sigstore integration time:
-
Permalink:
ProxyHatCom/scrapy-proxyhat@2f2fa1860cb74a47a7ed54f9a46e6a3f17104e18 -
Branch / Tag:
refs/tags/v0.1.2 - Owner: https://github.com/ProxyHatCom
-
Access:
public
-
Token Issuer:
https://token.actions.githubusercontent.com -
Runner Environment:
github-hosted -
Publication workflow:
ci.yml@2f2fa1860cb74a47a7ed54f9a46e6a3f17104e18 -
Trigger Event:
push
-
Statement type:
