CrewAI Olostep Integration

Give your CrewAI agents the power to scrape, search, and research the web.

This package provides ready-to-use CrewAI tools that integrate with the Olostep API for powerful web data access capabilities.

Features

• 🌐 Web Scraping - Extract content from any URL in markdown, HTML, or text format
• 📦 Batch Processing - Scrape up to 10,000 URLs in parallel
• 🕷️ Website Crawling - Automatically discover and scrape entire websites
• 🗺️ Sitemap Extraction - Get all URLs from a website for site structure analysis
• 🔍 AI-Powered Search - Get answers to questions with web sources and structured output

Installation

pip install crewai-olostep

Quick Start

export OLOSTEP_API_KEY="your_api_key_here"

from crewai import Agent, Task, Crew
from crewai_olostep import olostep_scrape_tool, olostep_answer_tool

# Create an agent with Olostep tools
researcher = Agent(
    role="Web Researcher",
    goal="Find accurate information from the web",
    backstory="Expert researcher with web scraping skills.",
    tools=[olostep_scrape_tool, olostep_answer_tool],
    verbose=True
)

# Create a task
task = Task(
    description="Research the pricing of Stripe's payment processing",
    expected_output="A summary of Stripe's pricing tiers and fees",
    agent=researcher
)

# Run the crew
crew = Crew(agents=[researcher], tasks=[task])
result = crew.kickoff()
print(result)

Available Tools

Get All Tools

from crewai_olostep import get_all_tools

all_tools = get_all_tools()  # Returns all 5 tools

Example: Multi-Agent Research Crew

from crewai import Agent, Task, Crew, Process
from crewai_olostep import (
    olostep_scrape_tool,
    olostep_sitemap_tool,
    olostep_answer_tool,
)

# Specialized agents
explorer = Agent(
    role="Site Explorer",
    goal="Discover website structures",
    tools=[olostep_sitemap_tool],
)

scraper = Agent(
    role="Content Extractor", 
    goal="Extract web content",
    tools=[olostep_scrape_tool],
)

analyst = Agent(
    role="Research Analyst",
    goal="Analyze and synthesize",
    tools=[olostep_answer_tool],
)

# Chained tasks
task1 = Task(description="Find all product pages on https://example.com", agent=explorer)
task2 = Task(description="Scrape the top 3 pages", agent=scraper, context=[task1])
task3 = Task(description="Summarize findings", agent=analyst, context=[task2])

crew = Crew(
    agents=[explorer, scraper, analyst],
    tasks=[task1, task2, task3],
    process=Process.sequential,
)

result = crew.kickoff()

Documentation

Full documentation at docs.olostep.com/integrations/crewai

Support

• PyPI: pypi.org/project/crewai-olostep
• Docs: docs.olostep.com
• Email: info@olostep.com

License

MIT License