proxmint-haystack

Haystack integration for Proxmint proxies: fetch web pages from residential, datacenter or 4G/5G mobile IPs and get them back as Documents. Pick the exit country, state, city, ZIP or ASN, and keep one IP for up to two hours when a site needs it.

• ProxmintFetcher: a component that takes a list of URLs and returns documents (readable page text, or the raw HTML) and failed_urls. Use it as the first step of an indexing pipeline, or wrap it with ComponentTool to give an agent a "read this page from Germany" tool.

Requests go straight from your machine to the Proxmint gateway (gw.proxmint.com). Nothing else sits in between, and bandwidth comes out of your own Proxmint account.

Install

pip install proxmint-haystack

Credentials

Create an API key under Settings in the Proxmint dashboard and export it:

export PROXMINT_API_KEY="pmk_..."

The key reads the proxy login for the product you choose (residential by default). You can use the proxy login instead, with PROXMINT_USERNAME and PROXMINT_PASSWORD.

Fetch pages

from haystack_integrations.components.fetchers.proxmint import ProxmintFetcher

fetcher = ProxmintFetcher(country="gb")
result = fetcher.run(urls=["https://example.com", "https://example.org"])

print(result["documents"][0].content)
print(result["documents"][0].meta)  # source, requested_url, status, title, content_type, country
print(result["failed_urls"])

Use it in a pipeline

from haystack import Pipeline
from haystack.components.preprocessors import DocumentSplitter
from haystack.components.writers import DocumentWriter
from haystack.document_stores.in_memory import InMemoryDocumentStore
from haystack_integrations.components.fetchers.proxmint import ProxmintFetcher

store = InMemoryDocumentStore()
pipe = Pipeline()
pipe.add_component("fetcher", ProxmintFetcher(country="us", city="New York", session_minutes=30))
pipe.add_component("splitter", DocumentSplitter(split_by="sentence", split_length=5))
pipe.add_component("writer", DocumentWriter(store))
pipe.connect("fetcher.documents", "splitter.documents")
pipe.connect("splitter.documents", "writer.documents")

pipe.run({"fetcher": {"urls": ["https://example.com/pricing", "https://example.org/pricing"]}})

Use it as an agent tool

from haystack.tools import ComponentTool
from haystack_integrations.components.fetchers.proxmint import ProxmintFetcher

tool = ComponentTool(
    component=ProxmintFetcher(),
    name="fetch_pages",
    description="Fetch web pages through a proxy in a chosen country and return their text.",
)

Pass the tool to an Agent or a chat generator as usual. The model can set urls and an optional country per call.

Options

A gateway refusal (no bandwidth left, no IP matching the targeting, a blocked site) names what to change.

Good to know

• Web traffic works on destination ports 80, 443, 8080, 8443, 8888 and 8090 (port access).
• A short list of sites, mostly banking and government, is refused by the network (blocked websites).
• City, state, ZIP and ASN targeting meter at 2x on residential, datacenter and mobile (targeting).
• Each product allows up to 2,000 concurrent connections.

Full guide: proxmint.com/docs/haystack. Pricing: proxmint.com/pricing. Bandwidth is billed per GB and never expires.

Development

uv run --group test pytest

The tests run offline against a stand-in for the gateway.

License

MIT