llamaindex-proxyhat

Read the web in LlamaIndex through ProxyHat residential proxies — a web reader with rotating IPs, geo-targeting, and sticky sessions.

Why

RAG pipelines that read the open web from datacenter IPs get blocked, rate-limited, and served bot-detection pages instead of content. This package routes LlamaIndex's web reads through ProxyHat's residential IPs (50M+ across 148+ countries) — a fresh IP per page by default, or one pinned IP for a whole multi-page load.

• ProxyHatWebReader — a BaseReader whose load_data(urls) fetches each URL and returns Document objects for indexing, retrieval, or summarization.

Install

pip install llamaindex-proxyhat

llama-index-core is a peer — bring your own version (>=0.10), the same one your app and other LlamaIndex integrations already use.

Quick start

Load pages as Document objects:

from llamaindex_proxyhat import ProxyHatWebReader

# An API key auto-selects an active residential sub-user:
reader = ProxyHatWebReader(
    api_key="ph_your_api_key",
    country="us",
)

docs = reader.load_data(["https://example.com", "https://example.org"])

for doc in docs:
    print(doc.metadata["url"], doc.metadata["status_code"], len(doc.text))

Feed the documents straight into an index:

from llama_index.core import VectorStoreIndex

index = VectorStoreIndex.from_documents(docs)
answer = index.as_query_engine().query("What does the page say?")

Get an API key at proxyhat.com.

Credentials

Pass them explicitly or via environment variables — options win over env:

Targeting

ProxyHatWebReader(
    api_key="ph_your_api_key",
    country="us",        # ISO code or "any" (default)
    region="california",
    city="new_york",
    filter="high",       # AI IP-quality tier
    sticky="30m",        # keep one IP across the whole load; omit for rotating
    protocol="http",     # or "socks5"
)

Each Document carries text (the response body) and metadata with url, status_code, and content_type.

Rotating vs sticky

By default the reader is rotating: each page is fetched over a new connection with a stable gateway username, so ProxyHat hands out a fresh residential IP per page. Set sticky (e.g. sticky="30m" or sticky=True) to pin one residential IP across every page in the load_data call — the right choice for paginated sites, logged-in sessions, or anything that must look like a single user.

How it works

ProxyHatWebReader subclasses BaseReader (llama_index.core.readers.base) and returns Document objects (llama_index.core). It resolves your gateway credentials once (via the official proxyhat SDK — an API key auto-picks an active sub-user, or you supply username/password directly), then builds a ProxyHat gateway connection URL with the SDK's targeting grammar and fetches each URL through it with httpx. Rotating loads reuse a stable username so the gateway rotates the IP per connection; sticky loads reuse one session id (a single httpx client) so every request exits from the same IP.

Publishing to the LlamaHub / LlamaIndex integrations registry is a planned follow-up.

License

MIT © ProxyHat