langchain-scrapeunblocker

LangChain tools for ScrapeUnblocker: give your agent live web data. Read any public page in a real browser as clean Markdown or structured JSON, search Google, and run 100+ ready-made site scrapers (Amazon, Walmart, eBay, Google Maps, YouTube, Zillow, Indeed and more), including pages behind Cloudflare, DataDome, PerimeterX and Akamai.

Install

pip install -U langchain-scrapeunblocker
export SCRAPEUNBLOCKER_API_KEY="your-api-key"

Get a key at app.scrapeunblocker.com - a free account comes with 500 requests to start.

Use with an agent

from langchain.agents import create_agent
from langchain_scrapeunblocker import ScrapeUnblockerToolkit

agent = create_agent(
    "anthropic:claude-sonnet-5-5",
    tools=ScrapeUnblockerToolkit().get_tools(),
)
result = agent.invoke(
    {"messages": [{"role": "user", "content": "What does the Echo Dot cost on amazon.com today?"}]}
)
print(result["messages"][-1].content)

Tools

Use them one by one:

from langchain_scrapeunblocker import ScrapeUnblockerMarkdown, ScrapeUnblockerRunPlugin

ScrapeUnblockerMarkdown().invoke({"url": "https://example.com"})

ScrapeUnblockerRunPlugin().invoke(
    {"plugin": "youtube_transcript", "params": {"video": "dQw4w9WgXcQ"}}
)

ScrapeUnblockerGoogleSearch, ScrapeUnblockerParsedData and ScrapeUnblockerRunPlugin return content_and_artifact: the model gets a compact text or JSON, and the full API response is on the ToolMessage's artifact for your code. Called with a tool call, they return that message:

from langchain_scrapeunblocker import ScrapeUnblockerGoogleSearch

msg = ScrapeUnblockerGoogleSearch().invoke(
    {"name": "scrapeunblocker_google_search", "args": {"query": "langchain"}, "id": "1", "type": "tool_call"}
)
msg.content    # what the model reads
msg.artifact   # the whole results page: ads, knowledge panel, local, shopping, videos...

Every tool takes an optional proxy_country (ISO code such as "US" or "DE") to make the request from that country, and works with ainvoke too.

The plugin catalog comes from the installed scrapeunblocker SDK: pip install -U scrapeunblocker picks up new plugins without a new release of this package.

Configuration

ScrapeUnblockerToolkit(include_plugins=False) leaves out the two plugin tools.

Failed calls come back to the model as text it can act on (for example "the page itself answered 404, do not retry"), not as exceptions. Set handle_tool_error=False on a tool to raise instead.

Links

• ScrapeUnblocker docs
• Site plugins
• Python SDK

Development

pip install -e ".[test]"
pytest                                 # unit tests, no network
SCRAPEUNBLOCKER_API_KEY=... pytest tests/integration_tests   # real API calls

License

MIT