langchain-thecrawler

This package contains the LangChain integration with TheCrawler — a web-scraping and structured-extraction API that runs the extraction LLM on its own GPU, so AI extraction is included on every page with no per-call surcharge.

Installation

pip install -U langchain-thecrawler

Set your API key (get one at miaibot.ai):

export THECRAWLER_API_KEY="mai_live_..."

Document Loader

TheCrawlerLoader loads one or more URLs as LangChain Document objects with boilerplate-stripped markdown as page_content and rich page metadata.

from langchain_thecrawler import TheCrawlerLoader

loader = TheCrawlerLoader(
    ["https://example.com"],
    # api_key="mai_live_...",  # or set THECRAWLER_API_KEY
)

docs = loader.load()        # list[Document]
# or stream:
for doc in loader.lazy_load():
    print(doc.metadata["url"], len(doc.page_content))

PDF and DOCX URLs are handled server-side. A per-page failure does not raise — failed pages come back as a Document with empty page_content and metadata["status"] == "error" plus a structured error_type, so you can branch on it:

ok = [d for d in docs if d.metadata.get("status") != "error"]

Options

License

MIT