LangChain Olostep Integration

The most reliable and cost-effective web search, scraping and crawling API for AI.

Build intelligent agents that can search, scrape, analyze, and structure data from any website. Perfect for LangChain and LangGraph applications.

Features

This integration provides access to all 5 Olostep API endpoints:

1. Scrapes (/v1/scrapes)

Extract content from any single URL in multiple formats (Markdown, HTML, JSON, text). Handles JavaScript rendering, anti-scraping measures, and supports specialized parsers for specific websites.

2. Batches (/v1/batches)

Process up to 10,000 URLs in parallel. Batch jobs typically complete in 5-8 minutes regardless of batch size. Perfect for large-scale data extraction and competitor analysis.

3. Answers (/v1/answers)

AI-powered web search with natural language queries. Ask questions and get structured answers with sources. Ground your AI products on real-world data and facts. Perfect for data enrichment and research.

4. Maps (/v1/maps)

Extract all URLs from a website for site structure analysis and content discovery. Can discover up to ~100,000 URLs in a single call. Perfect for SEO audits and preparing URLs for batch processing.

5. Crawls (/v1/crawls)

Autonomously discover and scrape entire websites by following links. Perfect for documentation sites, blogs, and comprehensive website data extraction.

Core Capabilities

• Multiple Formats: Markdown, HTML, JSON, and plain text
• JavaScript Rendering: Automatic handling of dynamic content
• Specialized Parsers: Custom parsers for Amazon, Google Search, and more
• Location-Specific: Scrape with country-specific settings
• LangGraph Ready: Perfect for building complex AI agent workflows
• Cost-Effective: Pay only for what you use

Installation

pip install langchain-olostep

Setup

Set your Olostep API key:

export OLOSTEP_API_KEY="your_olostep_api_key_here"

Get your API key from https://olostep.com/dashboard

Quick Start

1. Basic Web Scraping

from langchain_olostep import scrape_website
import asyncio

# Scrape a website
content = asyncio.run(scrape_website.ainvoke({
    "url": "https://example.com",
    "format": "markdown"
}))

print(content)

2. Batch Scraping

from langchain_olostep import scrape_batch
import asyncio

# Scrape multiple URLs in parallel
result = asyncio.run(scrape_batch.ainvoke({
    "urls": [
        "https://example1.com",
        "https://example2.com",
        "https://example3.com"
    ],
    "format": "markdown"
}))

print(result)

3. AI-Powered Web Search & Q&A

from langchain_olostep import answer_question
import asyncio

# Ask a question and get structured answer
result = asyncio.run(answer_question.ainvoke({
    "task": "What is the latest book by J.K. Rowling?",
    "json_schema": {
        "book_title": "",
        "author": "",
        "release_date": ""
    }
}))

print(result)
# Returns: {"book_title": "The Hallmarked Man", "author": "J.K. Rowling", ...}
# Also includes sources used

4. Extract URLs from Website

from langchain_olostep import extract_urls
import asyncio

# Get all URLs from a website
result = asyncio.run(extract_urls.ainvoke({
    "url": "https://example.com",
    "include_urls": ["/blog/**"],  # Only blog posts
    "top_n": 100
}))

print(result)

5. Crawl Entire Website

from langchain_olostep import crawl_website
import asyncio

# Autonomously crawl and scrape a website
result = asyncio.run(crawl_website.ainvoke({
    "start_url": "https://docs.example.com",
    "max_pages": 100,
    "exclude_urls": ["/admin/**"]
}))

print(result)

Usage with LangChain Agent

pip install langchain langchain-openai langchain-olostep

from langchain.agents import create_agent
from langchain_openai import ChatOpenAI
from langchain_olostep import (
    scrape_website,
    answer_question,
    extract_urls
)

# Create agent with Olostep tools
tools = [scrape_website, answer_question, extract_urls]
agent = create_agent(
    model=ChatOpenAI(model="gpt-4o-mini"),
    tools=tools,
)

# Use the agent
result = agent.invoke({
    "messages": [{
        "role": "user",
        "content": """
Research the company at https://company.com:
1. Scrape their about page
2. Search for their latest funding round
3. Extract all their product pages
""",
    }]
})

print(result["messages"][-1].content)

Usage with LangGraph

import json
from typing import Any, TypedDict

from langgraph.graph import StateGraph, END
from langchain_olostep import (
    scrape_batch,
    answer_question,
    extract_urls
)


class ResearchState(TypedDict, total=False):
    target_url: str
    research_question: str
    desired_format: dict
    urls: list[str]
    batch_id: str
    answer: Any


# Build a research agent workflow
def create_research_agent():
    workflow = StateGraph(ResearchState)
    
    def discover_pages(state: ResearchState):
        # Extract all URLs from target site
        result = extract_urls.invoke({
            "url": state["target_url"],
            "include_urls": ["/product/**"],
            "top_n": 50
        })
        return {"urls": json.loads(result)["urls"]}
    
    def scrape_pages(state: ResearchState):
        # Scrape discovered pages in batch
        result = scrape_batch.invoke({
            "urls": state["urls"],
            "format": "markdown"
        })
        return {"batch_id": json.loads(result)["batch_id"]}
    
    def answer_questions(state: ResearchState):
        # Use AI to answer questions about the data
        result = answer_question.invoke({
            "task": state["research_question"],
            "json_schema": state["desired_format"]
        })
        return {"answer": json.loads(result)["answer"]}
    
    workflow.add_node("discover", discover_pages)
    workflow.add_node("scrape", scrape_pages)
    workflow.add_node("analyze", answer_questions)
    
    workflow.set_entry_point("discover")
    workflow.add_edge("discover", "scrape")
    workflow.add_edge("scrape", "analyze")
    workflow.add_edge("analyze", END)
    
    return workflow.compile()

# Use the agent
agent = create_research_agent()
result = agent.invoke({
    "target_url": "https://store.com",
    "research_question": "What are the top 5 most expensive products?",
    "desired_format": {"products": [{"name": "", "price": "", "url": ""}]}
})

Available Tools

scrape_website

Scrape content from a single URL. Supports markdown, HTML, JSON, and text formats.

Parameters:

• url (required): Website URL to scrape
• format: Output format (markdown, html, json, text). Default: "markdown"
• country: Country code for location-specific content (e.g., "US", "GB", "CA")
• wait_before_scraping: Wait time in milliseconds for JavaScript rendering
• parser: Optional parser ID for specialized extraction (e.g., "@olostep/amazon-product")

scrape_batch

Scrape multiple URLs in parallel (up to 10,000 at once).

Parameters:

• urls (required): List of URLs to scrape
• format: Output format for all URLs. Default: "markdown"
• country: Country code for location-specific content
• wait_before_scraping: Wait time in milliseconds for JavaScript rendering
• parser: Optional parser ID for specialized extraction

answer_question

Search the web and get AI-powered answers with sources.

Parameters:

• task (required): Question or task to search for
• json_schema: Optional JSON schema dict/string describing desired output format

Examples:

# Simple question
answer_question.invoke({"task": "What is the capital of France?"})

# With structured output
answer_question.invoke({
    "task": "Find the CEO of Stripe",
    "json_schema": {"ceo_name": "", "founded_year": ""}
})

extract_urls

Extract all URLs from a website for site structure analysis.

Parameters:

• url (required): Website URL to extract URLs from
• search_query: Optional search query to filter URLs
• top_n: Limit number of URLs returned
• include_urls: Glob patterns to include (e.g., ["/blog/**"])
• exclude_urls: Glob patterns to exclude (e.g., ["/admin/**"])

crawl_website

Autonomously discover and scrape entire websites.

Parameters:

• start_url (required): Starting URL for the crawl
• max_pages: Maximum number of pages to crawl. Default: 100
• include_urls: Glob patterns to include
• exclude_urls: Glob patterns to exclude
• max_depth: Maximum depth to crawl from start_url
• include_external: Include external URLs. Default: False

Advanced Examples

Data Enrichment

from langchain_olostep import answer_question

# Enrich company data from spreadsheet
companies = ["Stripe", "Shopify", "Square"]

for company in companies:
    result = answer_question.invoke({
        "task": f"Find information about {company}",
        "json_schema": {
            "ceo": "",
            "headquarters": "",
            "employee_count": "",
            "latest_funding": ""
        }
    })
    print(f"{company}: {result}")

E-commerce Product Scraping

from langchain_olostep import scrape_website

# Scrape Amazon product with specialized parser
result = scrape_website.invoke({
    "url": "https://www.amazon.com/dp/PRODUCT_ID",
    "parser": "@olostep/amazon-product",
    "format": "json"
})
# Returns structured product data: price, title, rating, etc.

SEO Audit

from langchain_olostep import extract_urls, scrape_batch

# 1. Discover all pages
urls_result = extract_urls.invoke({
    "url": "https://yoursite.com",
    "top_n": 1000
})

# 2. Scrape all pages to analyze
urls = json.loads(urls_result)["urls"]
batch_result = scrape_batch.invoke({
    "urls": urls,
    "format": "html"
})

Documentation Scraping

from langchain_olostep import crawl_website

# Crawl entire docs site
result = crawl_website.invoke({
    "start_url": "https://docs.example.com",
    "max_pages": 500,
    "include_urls": ["/docs/**"],
    "exclude_urls": ["/api/**", "/v1/**"]  # Exclude old versions
})

Error Handling

from langchain_core.exceptions import LangChainException

try:
    result = await scrape_website.ainvoke({
        "url": "https://example.com"
    })
except LangChainException as e:
    print(f"Scraping failed: {e}")

Environment Variables

• OLOSTEP_API_KEY: Your Olostep API key (required)

Specialized Parsers

Olostep provides pre-built parsers for popular websites:

• @olostep/amazon-product: Amazon product pages
• @olostep/google-search: Google search results
• @olostep/google-maps: Google Maps data

Use them with the parser parameter:

scrape_website.invoke({
    "url": "https://www.amazon.com/dp/PRODUCT_ID",
    "parser": "@olostep/amazon-product"
})

Pricing

Olostep offers competitive pricing:

• Pay only for what you use
• No hidden fees
• Volume discounts available

Visit https://olostep.com/pricing for details.

Support

• Documentation: https://docs.olostep.com
• Issues: https://github.com/olostep-api/langchain-olostep/issues
• Email: info@olostep.com

Links

• PyPI: https://pypi.org/project/langchain-olostep/
• Documentation: https://docs.olostep.com/integrations/langchain
• Dashboard: https://olostep.com/dashboard
• Website: https://olostep.com

License

MIT