WebCrawlerAPI LangChain Integration

WebcrawlerAPI - is a website to LLM data API. It allows to convert websites and webpages markdown or cleaned content.

Pay as you go (No subscription required).

This package provides LangChain integration for WebCrawlerAPI, allowing you to easily use web crawling capabilities with LangChain document processing pipeline.

Installation

Get your API key first

pip install webcrawlerapi-langchain

Usage

Requires Python 3.10 or newer. api_key and base_url fall back to the WEBCRAWLERAPI_API_KEY and WEBCRAWLERAPI_BASE_URL environment variables when not passed.

Basic Loading

from webcrawlerapi_langchain import WebCrawlerAPILoader

# Initialize the loader
loader = WebCrawlerAPILoader(
    url="https://example.com",
    api_key="your-api-key",
    output_format="markdown",
    items_limit=10
)

# Load documents
documents = loader.load()

# Use documents in your LangChain pipeline
for doc in documents:
    print(doc.page_content[:100])
    print(doc.metadata)

Async Loading

# Async loading
documents = await loader.aload()

Lazy Loading

Yields each page as soon as it finishes crawling:

# Lazy loading
for doc in loader.lazy_load():
    print(doc.page_content[:100])

Async Lazy Loading

# Async lazy loading
async for doc in loader.alazy_load():
    print(doc.page_content[:100])

Error Handling

Crawling and API failures raise WebCrawlerAPILoaderError, which is importable from webcrawlerapi_langchain. Invalid parameters raise ValueError.

from webcrawlerapi_langchain import WebCrawlerAPILoaderError

try:
    documents = loader.load()
except WebCrawlerAPILoaderError as e:
    print(f"Crawl failed: {e}")

Configuration

The loader accepts the following parameters:

• url: The URL to crawl (keyword arguments follow it)
• api_key: Your WebCrawlerAPI API key. Defaults to WEBCRAWLERAPI_API_KEY
• base_url: API base URL. Defaults to WEBCRAWLERAPI_BASE_URL
• output_format: Content format of each Document: markdown (default), cleaned, or html
• scrape_type: Deprecated, use output_format instead
• items_limit: Maximum number of pages to crawl (default 10)
• whitelist_regexp: Regex pattern for URL whitelist
• blacklist_regexp: Regex pattern for URL blacklist
• main_content_only: Extract only the main content of each page (default False)
• max_depth: Maximum crawl depth. 0 is the seed URL only, 1 is the seed URL plus direct links
• max_age: Max age in seconds for cached content. 0 always fetches fresh content
• respect_robots_txt: Whether to respect robots.txt (default False)
• keep_query_params: Keep URL query params when deduplicating links
• max_polls: Maximum number of job status checks before giving up (default 100)

Document metadata keys: url, title, status_code, created_at, referred_url, depth, cost, job_id, item_id.

Links

• WebCrawlerAPI Python SDK
• WebCrawlerAPI Documentation
• WebCrawlerAPI API Reference

If you need help with integration feel free to contact us.

License

MIT License