langchain-mrscraper 0.1.0

LangChain tools for the MrScraper web-scraping API

langchain-mrscraper

LangChain integration package for the MrScraper SDK.

This package exposes MrScraper capabilities as LangChain tools so agents can:

• Fetch rendered HTML from protected websites
• Create AI scrapers from natural-language prompts
• Rerun AI/manual scrapers (single and bulk)
• List and fetch scraping results

Installation

pip install -U langchain-mrscraper

or:

uv add langchain-mrscraper

mrscraper-sdk is installed automatically as a dependency, so users do not need to install it separately.

Quick start

import os
from langchain_mrscraper import MrScraperToolkit

os.environ["MRSCRAPER_API_KEY"] = "your-token"

tools = MrScraperToolkit().get_tools()

Use with an agent

from langgraph.prebuilt import create_react_agent
from langchain_openai import ChatOpenAI
from langchain_mrscraper import MrScraperToolkit

tools = MrScraperToolkit(token="your-token").get_tools()
agent = create_react_agent(ChatOpenAI(model="gpt-4o-mini"), tools)

Available tools

• mrscraper_fetch_html
• mrscraper_create_scraper
• mrscraper_rerun_scraper
• mrscraper_bulk_rerun_ai_scraper
• mrscraper_rerun_manual_scraper
• mrscraper_bulk_rerun_manual_scraper
• mrscraper_get_all_results
• mrscraper_get_result_by_id

API styles

You can initialize via:

• MrScraperToolkit(...).get_tools() (recommended)
• load_mrscraper_tools(...) convenience function
• per-tool constructors with token="..." or mrscraper_api_key="..."
• environment variables MRSCRAPER_API_KEY (preferred) or MRSCRAPER_API_TOKEN

Tools vs. loaders

This integration is intentionally tools-first. MrScraper endpoints are action-oriented (fetch, create, rerun, list, retrieve) and best represented as BaseTool methods that agents can call explicitly.

A document loader abstraction is usually better when the primary job is deterministic "URL -> documents" ingestion into vector stores. MrScraper can support that in a separate package later, but this package should remain focused on agent tools.

Testing

pytest tests/unit_tests -v

Integration smoke tests (real API):

MRSCRAPER_API_KEY=your-token pytest tests/integration_tests -m integration -v

Local release workflow

1. Update version in pyproject.toml
2. Build: python -m build
3. Upload to TestPyPI: twine upload --repository testpypi dist/*
4. Verify install from TestPyPI
5. Upload to PyPI: twine upload dist/*

Docs files for LangChain PR

• Provider page: docs/providers/mrscraper.mdx
• Tool pages: docs/tools/*.mdx (one page per tool)

These are prepared to submit to langchain-ai/docs.