scrapy-proxyhat

Scrapy downloader middleware that routes every request through ProxyHat residential proxies — rotating IPs, geo-targeting, and sticky sessions.

Why

Scraping at scale from datacenter IPs gets you blocked and rate-limited. This middleware sends your Scrapy requests out through ProxyHat's residential IPs (50M+ across 148+ countries), with a fresh IP per request by default — no boilerplate.

Install

pip install scrapy-proxyhat

Setup

# settings.py
DOWNLOADER_MIDDLEWARES = {
    "scrapy_proxyhat.ProxyHatMiddleware": 610,
}

# Simplest — an API key auto-selects an active residential sub-user:
PROXYHAT_API_KEY = "ph_your_api_key"

# ...or explicit gateway credentials:
# PROXYHAT_USERNAME = "your-proxy-username"
# PROXYHAT_PASSWORD = "your-proxy-password"

# Optional targeting:
PROXYHAT_COUNTRY = "us"          # ISO code or "any" (default)
# PROXYHAT_REGION = "california"
# PROXYHAT_CITY = "new_york"
# PROXYHAT_STICKY = "30m"        # keep one IP; omit for rotating
# PROXYHAT_FILTER = "high"       # AI IP-quality tier
# PROXYHAT_SUBUSER = "<uuid or name>"   # pick a specific sub-user (with API key)

Get an API key at proxyhat.com.

Per-request targeting

Override any setting per request via meta (keys are proxyhat_<setting>):

yield scrapy.Request(
    url,
    meta={"proxyhat_country": "de", "proxyhat_sticky": "30m"},
)

A request that already has meta["proxy"] set is left untouched.

How it works

The middleware builds a ProxyHat gateway connection URL (via the official proxyhat SDK) and sets request.meta["proxy"]. Scrapy's built-in HttpProxyMiddleware (priority 750, runs after this one at 610) moves the credentials into the Proxy-Authorization header. Rotating targeting reuses a stable username so the gateway hands out a fresh residential IP per connection; sticky targeting pins one IP for the session.

License

MIT © ProxyHat