langchain-scrapingpros

LangChain integration for Scraping Pros — web scraping tools for AI agents.

Installation

pip install langchain-scrapingpros

Quick Start

from langchain_scrapingpros import ScrapingProsScraper

# Use demo token (5,000 credits/month, no signup)
tool = ScrapingProsScraper(api_key="demo_6x595maoA6GdOdVb")
result = tool.invoke({"url": "https://example.com"})
print(result)

Tools

ScrapingProsScraper

Scrape a web page and return clean markdown text. Ideal for RAG pipelines and LLM consumption.

from langchain_scrapingpros import ScrapingProsScraper

tool = ScrapingProsScraper()  # uses SCRAPINGPROS_API_KEY env var

# Simple scrape
result = tool.invoke({"url": "https://example.com"})

# JavaScript-heavy site
result = tool.invoke({"url": "https://spa-site.com", "browser": True})

ScrapingProsExtractor

Extract structured data using CSS selectors. Returns JSON.

from langchain_scrapingpros import ScrapingProsExtractor

tool = ScrapingProsExtractor()
result = tool.invoke({
    "url": "https://quotes.toscrape.com/",
    "fields": {
        "quotes": "css:.text",
        "authors": "css:.author",
    },
})

Use with LangChain Agents

from langchain_scrapingpros import ScrapingProsScraper, ScrapingProsExtractor

tools = [ScrapingProsScraper(), ScrapingProsExtractor()]
# Pass tools to your LangChain agent

Authentication

Set SCRAPINGPROS_API_KEY environment variable, or pass api_key= explicitly.

Demo token for quick testing: demo_6x595maoA6GdOdVb (5,000 credits/month).

Get a dedicated token at scrapingpros.com.

Features

• Clean markdown output for AI/LLM consumption
• Automatic proxy rotation (950+ proxies, 200+ countries)
• Headless browser rendering for JavaScript sites
• CSS/XPath data extraction
• Anti-bot bypass (Camoufox anti-detection)
• CAPTCHA detection with clear error messages
• Sync and async support

Links

• Scraping Pros
• API Documentation
• Python SDK
• Full Reference

License

MIT