langchain-hydrafetch

LangChain integration for the Hydrafetch web data API.

Load pages as documents, search the web as a retriever, and give agents four tools that read, extract and enrich from the live web. Sync and async, fully typed.

Installation

pip install langchain-hydrafetch

Create a key at app.hydrafetch.com and set it:

export HYDRAFETCH_API_KEY="hf_..."

Everything below reads that variable when no api_key= is passed.

Quick start

from langchain_hydrafetch import HydrafetchLoader

docs = HydrafetchLoader("https://example.com/article").load()
print(docs[0].page_content)
print(docs[0].metadata["title"])

Document loader

Three modes. scrape loads one page, crawl walks a site and loads every page it finds, and map returns one document per discovered URL without fetching the bodies.

HydrafetchLoader("https://example.com/article").load()
HydrafetchLoader("https://example.com", mode="crawl", params={"limit": 50}).load()
HydrafetchLoader("https://example.com", mode="map").load()

content_format chooses what lands in page_content — markdown by default, or html, rawHtml, summary. params is forwarded to the API untouched, so anything the endpoint accepts works here:

HydrafetchLoader(
    "https://example.com/article",
    content_format="markdown",
    params={"onlyMainContent": True, "preferStructure": True, "blockAds": True},
).load()

Documents carry source and status, plus whatever page metadata was found: title, description, language, site_name, author, published_time, word_count, page_type, image. A redirect adds final_url; a crawled page adds depth; a mapped URL adds lastmod when the sitemap declares one.

Use lazy_load() to stream a large crawl instead of building the whole list in memory:

for doc in HydrafetchLoader("https://example.com", mode="crawl").lazy_load():
    index.add(doc)

Error pages are refused, not loaded

A URL that answers with an error status raises instead of returning a document, because the body of a 404 page is not the page you asked for and a retrieval index should not quietly absorb one.

HydrafetchLoader("https://example.com/gone").load()
# ValueError: https://example.com/gone returned HTTP 404. The body of an error
# page is not the page you asked for; pass raise_for_status=False to load it anyway.

In crawl mode a single dead page must not throw away the whole job, so error pages are dropped and the rest of the crawl is kept. Pass raise_for_status=False to load error pages in either mode.

Retriever

from langchain_hydrafetch import HydrafetchSearchRetriever

retriever = HydrafetchSearchRetriever(k=5)
docs = retriever.invoke("best open source vector databases")

Each document holds the result snippet, with source, title and rank in metadata. Pass scrape_content=True to fetch and return the full page body for every result instead:

HydrafetchSearchRetriever(k=3, scrape_content=True).invoke("...")

That costs one extra credit per result. k works as a constructor argument or per call, and search_params is forwarded to the search endpoint:

retriever.invoke("...", k=2)
HydrafetchSearchRetriever(search_params={"country": "us"})

Agent tools

from langchain_hydrafetch import (
    HydrafetchBrandTool,
    HydrafetchExtractTool,
    HydrafetchScrapeTool,
    HydrafetchSearchTool,
)

tools = [
    HydrafetchSearchTool(),
    HydrafetchScrapeTool(),
    HydrafetchExtractTool(),
    HydrafetchBrandTool(),
]

Search finds pages, scrape reads a page you already have. Giving an agent both is the usual setup.

Structured extraction takes either a schema or a plain-language description:

HydrafetchExtractTool().invoke({
    "urls": ["https://example.com/pricing"],
    "json_schema": {
        "type": "object",
        "properties": {"plans": {"type": "array", "items": {"type": "string"}}},
    },
})

HydrafetchExtractTool().invoke({
    "urls": ["https://example.com/about"],
    "prompt": "the founding year and the headquarters city",
})

Async

The retriever and every tool support ainvoke, and the loader supports alazy_load and aload:

docs = await HydrafetchSearchRetriever().ainvoke("...")
text = await HydrafetchScrapeTool().ainvoke({"url": "https://example.com"})

Error handling

Failures raise HydrafetchError from the underlying client, carrying the API's error code, HTTP status and request id.

from hydrafetch import HydrafetchError, HydrafetchTimeout

try:
    docs = HydrafetchLoader(url).load()
except HydrafetchTimeout:
    raise
except HydrafetchError as err:
    if err.is_out_of_credits:
        top_up()
    elif err.is_retryable:
        enqueue(url)
    else:
        raise

A page that loads but answers with an error status raises ValueError from the loader instead — that is a bad URL, not a failed request.

Configuration

Every class accepts the same connection options, all optional:

Credits

Failed requests are not billed. Pricing does not vary with page difficulty, so there is no render, stealth or proxy option to set.

Implementation notes

• Prefer map then a batch of scrape loads over a broad crawl. Fetching a whole site and discarding most of it is the most common source of wasted credits.
• params in crawl mode is forwarded to the crawl endpoint, so per-page options belong under scrapeOptions. At the top level they are ignored.
• preferStructure is off by default. Turn it on when headings, lists and tables matter; leave it off for raw article text.
• The loader constructs its client eagerly, so a missing API key fails at construction rather than at load().
• The retriever and tools construct their clients lazily on first use, which keeps them cheap to build and safe to define at import time.
• Metadata keys are snake_cased on the way out, so the API's siteName becomes site_name.
• Scraped content is untrusted input. Do not pass it to a model as instructions, and keep metadata["source"] with anything extracted from it.

Development

uv sync
uv run ruff check src tests
uv run ruff format --check src tests
uv run pytest -q

Unit tests run offline against fake clients. The integration tests are LangChain's own langchain-tests standard suite; they are skipped unless HYDRAFETCH_API_KEY is set, and they spend real credits.

Links

• Documentation
• OpenAPI specification
• MCP server and editor setup
• Python client — the client this integration wraps
• LlamaIndex reader
• Other clients: Node · Go · Ruby · Rust · PHP

License

MIT