crewai-browserless

A CrewAI tool for scraping web pages using the Browserless /smart-scrape API. Handles anti-bot detection, captchas, and proxying automatically.

Installation

pip install crewai-browserless

Or with uv:

uv add crewai-browserless

Setup

Get a Browserless API Token at https://www.browserless.io/, then set the BROWSERLESS_API_TOKEN:

export BROWSERLESS_API_TOKEN="your-token"

# for private deplyments:
# export BROWSERLESS_API_URL="https://your-browserless-instance.com"

Usage

With a CrewAI agent

from crewai import Agent, Crew, Task
from crewai_browserless import BrowserlessSmartScrapeTool

agent = Agent(
    role="Web Researcher",
    goal="Scrape web pages and summarize their content.",
    backstory="An expert at extracting useful information from websites.",
    tools=[BrowserlessSmartScrapeTool()],
)

task = Task(
    description="Scrape https://www.browserless.io/blog/java-memory-leak and summarize what the page is about.",
    expected_output="A short summary of the page content.",
    agent=agent,
)

crew = Crew(agents=[agent], tasks=[task])
result = crew.kickoff()

print(result)

Standalone

from crewai_browserless import BrowserlessSmartScrapeTool

tool = BrowserlessSmartScrapeTool()

# Call the tool directly
result = tool.run(url="https://en.wikipedia.org/wiki/Headless_browser", formats=["markdown"])

print(result)

Parameters

Environment Variables

How It Works

The tool sends a POST request to the Browserless /smart-scrape endpoint, which uses a cascading strategy pipeline:

1. HTTP fetch — fast, direct request
2. HTTP fetch with proxy — retries through a residential proxy
3. Browser rendering — headless browser for JavaScript-heavy pages
4. Browser with captcha solving — handles captcha challenges automatically

The first strategy that succeeds returns the result. If screenshot or pdf formats are requested, browser strategies are used automatically.

License

SSPL-1.0