scrapegraphai-proxyhat

Route ScrapeGraphAI LLM scraping graphs through ProxyHat residential proxies — a geo-targeted, sticky-by-default residential IP dropped straight into your graph_config.

Why

ScrapeGraphAI drives a real browser (Playwright) to render pages before the LLM reads them. From a datacenter IP that browser gets flagged, CAPTCHA-walled, and blocked. This package plugs ProxyHat's residential IPs (50M+ across 148+ countries) into ScrapeGraphAI through the exact hook it already supports — graph_config["loader_kwargs"]["proxy"] — with one pinned residential IP for the whole scrape by default, so pagination and JS interactions stay on one consistent identity. No fork, no boilerplate.

Install

pip install scrapegraphai-proxyhat scrapegraphai

scrapegraphai is a heavy peer dependency (LLM + Playwright stack) — bring your own version. The config builders (proxyhat_graph_config(), proxyhat_proxy()) work on their own; you only need scrapegraphai installed to actually run a graph.

Quick start

from scrapegraphai.graphs import SmartScraperGraph
from scrapegraphai_proxyhat import proxyhat_graph_config

# An API key auto-selects an active residential sub-user:
graph_config = proxyhat_graph_config(
    {"llm": {"model": "openai/gpt-4o-mini"}},
    country="us",   # sticky US residential IP for the whole scrape
)

scraper = SmartScraperGraph(
    prompt="List every project with its title and description.",
    source="https://perinim.github.io/projects/",
    config=graph_config,
)
print(scraper.run())

Get an API key at proxyhat.com.

proxyhat_graph_config(base_config, ...) merges the proxy into loader_kwargs.proxy and preserves everything else in your config (llm, verbose, headless, any existing loader_kwargs). Your base_config is never mutated.

Just want the proxy dict for a config you build yourself?

from scrapegraphai_proxyhat import proxyhat_proxy

graph_config = {
    "llm": {"model": "openai/gpt-4o-mini"},
    "loader_kwargs": {"proxy": proxyhat_proxy(country="de", sticky="1h")},
}
# proxyhat_proxy(...) -> {"server": "http://gate.proxyhat.com:8080", "username": ..., "password": ...}

Or the raw URL string (proxyhat_proxy_url(...)) for any other tool:

from scrapegraphai_proxyhat import proxyhat_proxy_url

url = proxyhat_proxy_url(country="us")
# http://<user>-country-us-sid-<id>-ttl-30m:<pass>@gate.proxyhat.com:8080

Credentials

Pass them explicitly or via environment variables — options win over env:

Targeting

proxyhat_graph_config(
    base_config,
    protocol="http",   # or "socks5"
    country="us",      # ISO code or "any" (default)
    region="california",
    city="los_angeles",
    filter="high",     # AI IP-quality tier
    sticky="30m",      # session lifetime (default); sticky=False rotates every request
)

All three builders (proxyhat_graph_config, proxyhat_proxy, proxyhat_proxy_url) take the same targeting keyword arguments.

Sticky IP per scrape (default)

A single scrape often hits the same site many times — rendering, scrolling, following pagination, retrying. If the exit IP changed mid-scrape the site would see a visitor teleporting between cities and block it. So this package is sticky by default: one residential IP is pinned for the whole run (sticky="30m"), keeping the session coherent.

Want a fresh IP on every request instead (e.g. a SearchGraph fanning out across many independent pages)? Turn stickiness off:

proxyhat_graph_config(base_config, country="us", sticky=False)  # rotating residential IP per connection

Set a custom lifetime with sticky="2h".

How it works

ScrapeGraphAI reads a proxy at graph_config["loader_kwargs"]["proxy"] and hands it to its ChromiumLoader (and on to Playwright) as a Playwright-style {server, username, password} mapping. proxyhat_proxy(...) resolves your gateway credentials (via the official proxyhat SDK — an API key auto-picks an active sub-user, or pass username/password), then builds exactly that mapping: server is the ProxyHat gateway (http://gate.proxyhat.com:8080, or socks5://…:1080) and username carries ProxyHat's targeting grammar (<user>-country-us-sid-<id>-ttl-30m). proxyhat_graph_config(...) just merges that dict into loader_kwargs.proxy over your config. A sticky username pins one residential IP; a rotating one makes the gateway hand out a fresh IP per connection.

License

MIT © ProxyHat