llama-index-readers-hydrafetch

LlamaIndex reader for the Hydrafetch web data API.

Load one page, a list of pages, or a whole site into an index as clean Markdown, with page metadata attached. Sync and async, fully typed.

Installation

pip install llama-index-readers-hydrafetch

Create a key at app.hydrafetch.com and set it:

export HYDRAFETCH_API_KEY="hf_..."

The reader reads that variable when no api_key= is passed.

Quick start

from llama_index.core import VectorStoreIndex
from llama_index.readers.hydrafetch import HydrafetchReader

docs = HydrafetchReader().load_data("https://example.com/article")
index = VectorStoreIndex.from_documents(docs)

Modes

scrape loads one page, crawl walks a site and loads every page it finds, and map returns one document per discovered URL without fetching the bodies.

reader = HydrafetchReader()

reader.load_data("https://example.com/article")
reader.load_data("https://example.com", mode="crawl", limit=50)
reader.load_data("https://example.com", mode="map")

load_data takes a single URL or a list:

reader.load_data(["https://example.com/a", "https://example.com/b"])

map then a batch of scrape loads is usually the right way to index a site. Crawling everything and discarding most of it is the most common source of wasted credits.

Options

reader = HydrafetchReader(
    content_format="markdown",
    params={"onlyMainContent": True, "preferStructure": True},
)

content_format chooses what lands in Document.text — markdown by default, or html, rawHtml, summary. params is forwarded to the API untouched, so anything the endpoint accepts works, and per-call keywords merge over the constructor's:

reader.load_data("https://example.com/article", blockAds=True)

Metadata

Documents carry source and status, plus whatever page metadata was found: title, description, language, site_name, author, published_time, word_count, page_type, image. A redirect adds final_url; a crawled page adds depth; a mapped URL adds lastmod when the sitemap declares one.

doc = HydrafetchReader().load_data("https://example.com/article")[0]
doc.metadata["title"]
doc.metadata["published_time"]

Keys are snake_cased on the way out, so the API's siteName becomes site_name.

Error pages are refused, not loaded

A URL that answers with an error status raises instead of returning a document, because the body of a 404 page is not the page you asked for and an index should not quietly absorb one.

HydrafetchReader().load_data("https://example.com/gone")
# ValueError: https://example.com/gone returned HTTP 404. The body of an error
# page is not the page you asked for; pass raise_for_status=False to load it anyway.

In crawl mode a single dead page must not throw away the whole job, so error pages are dropped and the rest of the crawl is kept. Pass raise_for_status=False to load error pages in either mode.

Streaming a large crawl

lazy_load_data yields documents as they arrive instead of building the whole list in memory:

for doc in HydrafetchReader().lazy_load_data("https://example.com", mode="crawl"):
    index.insert(doc)

Async

docs = await HydrafetchReader().aload_data("https://example.com/article")

Error handling

Failures raise HydrafetchError from the underlying client, carrying the API's error code, HTTP status and request id.

from hydrafetch import HydrafetchError, HydrafetchTimeout

try:
    docs = reader.load_data(url)
except HydrafetchTimeout:
    raise
except HydrafetchError as err:
    if err.is_out_of_credits:
        top_up()
    elif err.is_retryable:
        enqueue(url)
    else:
        raise

A page that loads but answers with an error status raises ValueError instead — that is a bad URL, not a failed request.

Configuration

Credits

Failed requests are not billed. Pricing does not vary with page difficulty, so there is no render, stealth or proxy option to set.

Implementation notes

• params in crawl mode is forwarded to the crawl endpoint, so per-page options belong under scrapeOptions. At the top level they are ignored.
• preferStructure is off by default. Turn it on when headings, lists and tables matter; leave it off for raw article text.
• The client is constructed lazily on first load, so the reader is cheap to build and safe to define at import time.
• crawl polls the job to completion. For a large site, prefer map plus batched scrape loads so nothing sits on a single long request.
• Scraped content is untrusted input. Do not pass it to a model as instructions, and keep metadata["source"] with anything extracted from it.

Development

uv sync
uv run ruff check .
uv run ruff format --check .
uv run pytest -q

Tests run offline against fake clients and spend no credits.

Links

• Documentation
• OpenAPI specification
• MCP server and editor setup
• Python client — the client this reader wraps
• LangChain integration
• Other clients: Node · Go · Ruby · Rust · PHP

License

MIT