crewai-alterlab

CrewAI integration for the AlterLab web scraping API.

Installation

pip install crewai-alterlab

Setup

Set your AlterLab API key:

export ALTERLAB_API_KEY="your-api-key"

Get your API key at alterlab.io/dashboard.

Scrape Tool

Scrape individual web pages with AlterLabScrapeTool:

from crewai_alterlab import AlterLabScrapeTool

tool = AlterLabScrapeTool()
result = tool.run("https://example.com")
print(result)

With a CrewAI Agent

from crewai import Agent, Task, Crew
from crewai_alterlab import AlterLabScrapeTool

researcher = Agent(
    role="Web Researcher",
    goal="Research topics by scraping web pages",
    backstory="You are an expert web researcher.",
    tools=[AlterLabScrapeTool()],
)

task = Task(
    description="Scrape https://example.com and summarise the content",
    expected_output="A summary of the page content",
    agent=researcher,
)

crew = Crew(agents=[researcher], tasks=[task])
result = crew.kickoff()

Custom Parameters

tool = AlterLabScrapeTool(
    scrape_params={
        "formats": ["markdown"],
        "mode": "js",  # JavaScript rendering
    },
)

Crawl Tool

Crawl multiple pages from a website with AlterLabCrawlTool:

from crewai_alterlab import AlterLabCrawlTool

tool = AlterLabCrawlTool()
result = tool.run(url="https://docs.example.com", max_pages=5)
print(result)

Research Agent with Both Tools

from crewai import Agent
from crewai_alterlab import AlterLabScrapeTool, AlterLabCrawlTool

researcher = Agent(
    role="Deep Researcher",
    goal="Thoroughly research topics using web data",
    backstory="You gather comprehensive information from the web.",
    tools=[AlterLabScrapeTool(), AlterLabCrawlTool()],
)

API Reference

See docs.alterlab.io for the full AlterLab API documentation.

Search Tool

Run web searches with AlterLabSearchTool (numbered title, URL and snippet):

from crewai_alterlab import AlterLabSearchTool

tool = AlterLabSearchTool(num_results=5, search_params={"engine": "google"})
print(tool.run(query="web scraping api"))

Blocked-Fetch Fallback

BlockedFetchFallbackTool runs your own fetch first and retries through AlterLab only when the result is a 403/429, a bot-protection challenge page, a CAPTCHA, or an empty JavaScript shell. Normal pages, 404s and network errors are returned or raised untouched, so credits are spent only on blocked fetches:

import requests
from crewai_alterlab import BlockedFetchFallbackTool

tool = BlockedFetchFallbackTool(fetch=lambda url: requests.get(url, timeout=20))
print(tool.run(url="https://example.com"))