webscout-mcp

AI-powered web intelligence platform for AI agents. Search, fetch, crawl, extract, understand, and monitor the web — with built-in AI, vector search, browser automation, and alerting. Everything stays on your machine.

🎯 Project Positioning: webscout-mcp is primarily a Web Search / Fetch MCP server with extensive extension modules. The core MCP server exposes 6 stable tools (search, fetch, crawl, extract, cache stats, cache clear). Additional modules (AI, RAG, browser, monitoring, SEO, etc.) are available as Python libraries and are planned for MCP integration. See Module Status for detailed stability and integration status.

中文版本简介 | 快速了解项目，适合中文用户阅读

🎯 What's Included in MCP (Right Now)

The MCP server currently exposes these 6 core tools:

Tool	Description	Stability
web_search	Multi-backend web search with result merging	✅ Stable
web_fetch	Fetch and parse web pages with content extraction	✅ Stable
web_crawl	Concurrent website crawling with depth limits	🔶 Beta
web_extract	Structured content extraction with CSS selectors	✅ Stable
cache_stats	View cache statistics and hit rates	✅ Stable
cache_clear	Clear the search/fetch cache	✅ Stable

Available as Python libraries (not yet MCP tools): AI content understanding, vector search & RAG, headless browser automation, web monitoring & alerting, SEO analysis, OCR, PDF processing, knowledge graphs, and more. See Module Status for the full list.

✨ Features

🔍 Core Web Tools

• Multi-backend search — Bing, DuckDuckGo, Google, Brave HTML with automatic failover and result merging
• Smart content extraction — trafilatura + readability-lxml + html2text fallback, clean article content
• Concurrent crawler — BFS crawl with depth/page limits, robots.txt compliance, retry on failures
• Structured data extraction — CSS selectors, attributes, regex extraction
• Metadata extraction — JSON-LD, OpenGraph, Twitter Cards, article metadata, images, links
• RSS/Atom support — parse feeds and feed indexes

🤖 AI Content Understanding

• Text summarization — automatic article and page summarization
• Question answering — ask questions about fetched content
• Key points extraction — extract main ideas and takeaways
• Content classification — categorize content into custom categories
• Tag generation — auto-generate relevant tags
• Sentiment analysis — analyze text sentiment
• Document comparison — compare two documents side by side
• Entity extraction — extract people, places, organizations, dates
• Multiple LLM backends — Ollama (local/free), OpenAI, Doubao, custom OpenAI-compatible

🧠 Vector Search & RAG

• Semantic search — search by meaning, not just keywords
• RAG (Retrieval-Augmented Generation) — answer questions based on your crawled content
• Local vector database — ChromaDB persistent storage
• Multiple embedding backends — local sentence-transformers (free), OpenAI, custom
• Document chunking — automatic text splitting with overlap
• Similarity threshold — configurable relevance filtering

🌐 Headless Browser Automation

• JavaScript rendering — fetch modern SPAs and dynamic content
• User interaction simulation — scroll, click, fill forms
• Screenshot capture — full-page screenshots
• PDF export — convert web pages to PDF
• Login state management — cookie persistence across sessions
• Anti-detection stealth mode — navigator.webdriver, plugins, languages spoofing
• Resource blocking — block images, media, CSS, fonts for faster loading
• Proxy support — HTTP/HTTPS proxy configuration
• Multiple browsers — Chromium, Firefox, WebKit

📡 Web Monitoring & Alerting

• Scheduled monitoring — configurable check intervals
• Content change detection — text, HTML, specific element changes
• Keyword monitoring — appearance, disappearance, count changes
• Price monitoring — track price changes with threshold alerts
• Change history — persistent history with diff generation
• Multi-channel alerts — Webhook, Email (SMTP), DingTalk, WeCom
• Configurable thresholds — minimum change size, similarity thresholds

⚡ Performance & Security

• Smart caching — SQLite cache with TTL, size limits, automatic eviction
• Rate limiting — per-domain token-bucket rate limiting
• SSRF protection — blocks localhost, sensitive ports, invalid schemes
• Browser fingerprint rotation — random User-Agents + realistic headers
• TLS fingerprint simulation — realistic TLS ClientHello fingerprints
• Connection pooling — persistent HTTP connections
• Cookie management — automatic cookie handling and persistence

🚀 Easy Setup & Deployment

• One-click setup — webscout-mcp setup auto-installs all dependencies
• System detection — auto-detects OS, CPU, memory, GPU
• Smart recommendations — suggests optimal configuration based on hardware
• Docker support — pre-built images for amd64 and arm64
• Docker Compose — one-command deployment
• systemd service — Linux service file for production
• Kubernetes — deployment manifests for container orchestration
• Configuration hot-reload — reload config without restart

🔍 Website Analysis & Optimization

• SEO analyzer — comprehensive SEO audit: meta tags, headings, images, links, URL structure, content length, Open Graph, Twitter Cards, Schema markup, with multi-dimensional scoring and actionable recommendations
• Broken link checker — detect broken links, redirect chains, invalid URLs, mixed content; classify internal/external/mailto/tel/javascript links; detailed reporting with statistics
• Performance analyzer — page performance audit: HTML size, DOM size, resource counts, render-blocking resources, inline CSS/JS, optimization techniques (lazy loading, preconnect, preload), compression/cache detection, performance scoring
• Content quality assessor — readability scores (Flesch-Kincaid, Gunning Fog, SMOG), keyword density, content structure analysis, duplicate content detection, quality scoring

📊 Export & Integration

• Multiple export formats — JSON, CSV, Excel, Parquet, SQLite, Markdown, HTML
• Field selection & ordering — export only specified fields with custom column order
• Append mode — incremental exports for CSV and SQLite
• MCP server — native Model Context Protocol support
• CLI interface — command-line tools for search, fetch, crawl
• Python API — full programmatic access to all features
• Sitemap support — parse sitemap.xml and sitemap indexes
• Incremental crawling — only re-fetch changed pages via ETag/Last-Modified

⚠️ Search Backend Stability Notice

webscout-mcp uses direct HTML scraping for search backends (Bing, DuckDuckGo, Google, Brave) by default — no API keys required. This makes it free to use, but please be aware of the stability trade-offs:

What can go wrong

• DOM changes: Search engines frequently update their HTML structure, which can break scrapers
• CAPTCHAs: Automated requests may trigger CAPTCHAs (especially Google and Brave)
• Bot detection: Advanced bot detection may block or rate-limit requests
• IP blocking: Sustained automated requests can lead to IP bans
• Parameter changes: Search engines may change request parameters or headers

Mitigations built in

• ✅ Multi-backend failover: If one backend fails, automatically try the next one
• ✅ Realistic browser headers: Random User-Agents and realistic request headers
• ✅ Rate limiting: Per-domain rate limiting to avoid overwhelming search engines
• ✅ Caching: SQLite cache reduces repeated requests to the same queries
• ✅ Retry with backoff: Exponential backoff on transient failures

For production use

For production workloads requiring higher reliability, consider:

1. Using official search APIs (Bing Search API, SerpAPI, etc.) — planned for future releases
2. Deploying with rotating proxies
3. Increasing cache TTL to reduce request frequency
4. Monitoring search backend health and adjusting backends accordingly

Bottom line: webscout-mcp's default search is great for development, personal use, and low-volume workloads. For high-volume production use, plan for additional reliability measures.

📦 Installation

Quick Install

pip install webscout-mcp

Requires Python 3.10+.

One-Click Full Setup (Recommended)

# Install core package
pip install webscout-mcp

# Run setup to install all optional dependencies
webscout-mcp setup --playwright --ollama --vector-store

The setup command will:

• Detect your system configuration (OS, CPU, memory, GPU)
• Install Playwright and Chromium browser
• Install Ollama and download a local LLM (optional)
• Install ChromaDB and sentence-transformers for vector search (optional)
• Generate a configuration file
• Run a health check to verify everything works

Optional Dependencies

# Browser automation (Playwright)
pip install webscout-mcp[browser]
playwright install chromium

# Vector search & RAG (ChromaDB + sentence-transformers)
pip install webscout-mcp[vector]

# AI content understanding (OpenAI client)
pip install webscout-mcp[ai]

# All features
pip install webscout-mcp[all]

Docker

docker pull wxslang/webscout-mcp:latest
docker run -p 8000:8000 wxslang/webscout-mcp:latest

🚀 Quick Start

MCP Client Configuration

Add to your MCP client config:

Claude Desktop (~/Library/Application Support/Claude/claude_desktop_config.json on macOS, %APPDATA%\Claude\claude_desktop_config.json on Windows):

{
  "mcpServers": {
    "webscout": {
      "command": "webscout-mcp",
      "args": []
    }
  }
}

Cursor (Settings → MCP → Add new MCP server):

{
  "mcpServers": {
    "webscout": {
      "command": "webscout-mcp",
      "args": []
    }
  }
}

CLI Usage

# Search the web
webscout-mcp search "python web scraping" --max-results 10

# Fetch a page
webscout-mcp fetch https://example.com --extract --format markdown

# Crawl a site
webscout-mcp crawl https://example.com --depth 2 --pages 50

# Run setup
webscout-mcp setup --playwright

# Start MCP server
webscout-mcp serve

Python API

from webscout_mcp import WebScout

# Initialize
scout = WebScout()

# Search
results = scout.search("AI agents", max_results=5)
for result in results:
    print(result.title, result.url)

# Fetch and extract
page = scout.fetch("https://example.com/article")
print(page.title)
print(page.content)  # Clean article text

# Crawl
pages = scout.crawl("https://example.com", depth=2, max_pages=20)

🤖 AI Content Understanding

from webscout_mcp.ai_processor import AIProcessor, AIConfig

# Use local Ollama (free, no API key needed)
config = AIConfig(backend="ollama", model="qwen2.5:7b")
ai = AIProcessor(config=config)

# Summarize
summary = ai.summarize(page.content, max_length=500)
print(summary.content)

# Ask questions
answer = ai.answer_question(page.content, "What are the main points?")
print(answer.content)

# Extract key points
points = ai.extract_key_points(page.content, num_points=5)
print(points.content)

# Analyze sentiment
sentiment = ai.analyze_sentiment(page.content)
print(sentiment.content)

Using OpenAI API:

config = AIConfig(
    backend="openai",
    model="gpt-4o",
    api_key="your-api-key",
)

Using Doubao (豆包):

config = AIConfig(
    backend="doubao",
    model="ep-20240101",
    api_key="your-api-key",
)

🧠 Vector Search & RAG

from webscout_mcp.vector_store import VectorStore, RAGEngine, Document

# Initialize vector store (local, free)
store = VectorStore()

# Add documents
doc = Document.from_text(page.content, source=page.url)
store.add_document(doc)

# Semantic search
results = store.search("how to build AI agents", n_results=5)
for result in results:
    print(f"[{result.score:.2f}] {result.document.content[:100]}")

# RAG: Ask questions based on your documents
rag = RAGEngine(vector_store=store)
answer = rag.query("What is the best approach for web scraping?")
print(answer["answer"])
print("Sources:", answer["sources"])

🌐 Headless Browser Automation

from webscout_mcp.browser_fetcher import BrowserFetcher, BrowserConfig

# Initialize
config = BrowserConfig(headless=True, block_media=True)
browser = BrowserFetcher(config=config)

# Fetch JS-rendered page
result = browser.fetch(
    "https://example.com/spa",
    wait_for_selector=".content",
    scroll_to_bottom=True,
)
print(result.title)
print(result.content)

# Take screenshot
browser.take_screenshot("https://example.com", "screenshot.png", full_page=True)

# Export to PDF
browser.export_pdf("https://example.com", "page.pdf")

# Click element
result = browser.click_element("https://example.com", "button.load-more")

# Fill form
result = browser.fill_form(
    "https://example.com/login",
    {"#username": "user", "#password": "pass"},
    submit_selector="button[type=submit]",
)

browser.close()

📡 Web Monitoring & Alerting

from webscout_mcp.monitor import WebMonitor, MonitorConfig, WebhookAlert, EmailAlert

# Initialize
config = MonitorConfig(check_interval=300, min_change_size=10)
monitor = WebMonitor(config=config)

# Add alert channels
monitor.add_alert_channel(WebhookAlert("https://hooks.example.com/alert"))
monitor.add_alert_channel(EmailAlert(
    smtp_server="smtp.gmail.com",
    smtp_port=587,
    username="you@gmail.com",
    password="app-password",
    from_addr="you@gmail.com",
    to_addrs=["recipient@example.com"],
))

# Check for changes
changes = monitor.check_url("https://example.com/pricing")
for change in changes:
    print(f"{change.change_type}: {change.old_value} -> {change.new_value}")

# Get history
history = monitor.get_history("https://example.com/pricing")

🔍 Website Analysis & Optimization

SEO Analysis

from webscout_mcp.seo_analyzer import SEOAnalyzer

# Initialize
analyzer = SEOAnalyzer()

# Analyze a page
html = "<html>...</html>"
metrics = analyzer.analyze(html, url="https://example.com")

# Check scores
print(f"Overall SEO Score: {metrics.overall_score}/100")
print(f"Meta Score: {metrics.meta_score}")
print(f"Heading Score: {metrics.heading_score}")
print(f"Image Score: {metrics.image_score}")

# Check issues and recommendations
print("Issues:", metrics.issues)
print("Recommendations:", metrics.recommendations)

Broken Link Checking

from webscout_mcp.broken_link_checker import BrokenLinkChecker

# Initialize
checker = BrokenLinkChecker(timeout=10.0, max_redirects=5)

# Check all links on a page
html = "<html>...</html>"
report = checker.check_page(html, base_url="https://example.com")

# Check statistics
print(f"Total links: {report.total_links}")
print(f"OK: {report.ok_links}")
print(f"Broken: {report.broken_links}")
print(f"Redirects: {report.redirect_links}")
print(f"Broken percentage: {report.broken_link_percentage}%")

# Get only broken links
broken = checker.get_broken_links(report)
for link in broken:
    print(f"[{link.status}] {link.url} - {link.error_message}")

# Generate human-readable summary
print(checker.generate_summary(report))

Performance Analysis

from webscout_mcp.performance_analyzer import PerformanceAnalyzer

# Initialize
analyzer = PerformanceAnalyzer()

# Analyze page performance
html = "<html>...</html>"
headers = {"Content-Encoding": "gzip", "Cache-Control": "max-age=3600"}
metrics = analyzer.analyze(html, url="https://example.com", response_headers=headers)

# Check scores
print(f"Overall Performance Score: {metrics.overall_score}/100")
print(f"HTML Size: {metrics.html_size_kb}KB (score: {metrics.html_size_score})")
print(f"DOM Nodes: {metrics.dom_node_count} (score: {metrics.dom_size_score})")
print(f"Requests: {metrics.request_count} (score: {metrics.request_count_score})")

# Check optimization techniques
print(f"Has gzip: {metrics.has_gzip}")
print(f"Has brotli: {metrics.has_brotli}")
print(f"Has lazy loading: {metrics.has_lazy_loading}")
print(f"Has preconnect: {metrics.has_preconnect}")

# Check issues and recommendations
print("Issues:", metrics.issues)
print("Warnings:", metrics.warnings)
print("Recommendations:", metrics.recommendations)

Enhanced Data Export

from webscout_mcp.data_exporter import DataExporter, ExportConfig

# Sample data
data = [
    {"title": "Result 1", "url": "https://example.com/1", "score": 0.95},
    {"title": "Result 2", "url": "https://example.com/2", "score": 0.85},
]

# Export to JSON
config = ExportConfig(format="json", output_path="results.json", pretty_json=True)
exporter = DataExporter(config=config)
result = exporter.export(data)
print(f"Exported {result.record_count} records to {result.output_path}")

# Export to CSV with field selection
config = ExportConfig(
    format="csv",
    output_path="results.csv",
    fields=["title", "url"],  # Only export these fields
    csv_delimiter=",",
)
exporter = DataExporter(config=config)
result = exporter.export(data)

# Export to Excel
config = ExportConfig(format="excel", output_path="results.xlsx", excel_sheet_name="Results")
exporter = DataExporter(config=config)
result = exporter.export(data)

# Export to SQLite
config = ExportConfig(format="sqlite", output_path="results.db", sqlite_table_name="search_results")
exporter = DataExporter(config=config)
result = exporter.export(data)

# Export to Parquet (columnar storage)
config = ExportConfig(format="parquet", output_path="results.parquet")
exporter = DataExporter(config=config)
result = exporter.export(data)

# Export to Markdown
config = ExportConfig(format="markdown", output_path="results.md")
exporter = DataExporter(config=config)
result = exporter.export(data)

# Export to HTML
config = ExportConfig(format="html", output_path="results.html")
exporter = DataExporter(config=config)
result = exporter.export(data)

# Using convenience function
from webscout_mcp.data_exporter import export_data
result = export_data(data, "results.json", export_format="json", fields=["title", "url"])

⚙️ Configuration

Environment Variables

# Core
WEBSCOUT_CACHE_ENABLED=true
WEBSCOUT_CACHE_TTL=3600
WEBSCOUT_RATE_LIMIT_ENABLED=true

# Search
WEBSCOUT_SEARCH_DEFAULT_BACKEND=bing
WEBSCOUT_SEARCH_MAX_RESULTS=10

# AI
WEBSCOUT_AI_BACKEND=ollama
WEBSCOUT_AI_MODEL=qwen2.5:7b
WEBSCOUT_AI_API_KEY=your-key

# Vector Store
WEBSCOUT_VECTOR_DB=chroma
WEBSCOUT_EMBEDDING_BACKEND=local
WEBSCOUT_EMBEDDING_MODEL=BAAI/bge-small-zh-v1.5

# Browser
WEBSCOUT_BROWSER_TYPE=chromium
WEBSCOUT_BROWSER_HEADLESS=true
WEBSCOUT_BROWSER_BLOCK_MEDIA=true

# Monitor
WEBSCOUT_MONITOR_INTERVAL=300
WEBSCOUT_MONITOR_MIN_CHANGE=10

# Setup
WEBSCOUT_SETUP_PLAYWRIGHT=true
WEBSCOUT_SETUP_OLLAMA=false
WEBSCOUT_SETUP_CHROMADB=false

Config File

Create ~/.config/webscout/config.toml:

[server]
host = "127.0.0.1"
port = 8000

[cache]
enabled = true
ttl = 3600

[search]
default_backend = "bing"
max_results = 10

[ai]
backend = "ollama"
model = "qwen2.5:7b"

[vector_store]
enabled = true
vector_db = "chroma"
embedding_backend = "local"

[browser]
enabled = true
headless = true

[monitor]
check_interval = 300

📚 Documentation

• README — This file
• Module Status — Module stability levels and MCP integration status
• Project Introduction — Detailed project overview and architecture
• Deployment Guide — Docker, systemd, Kubernetes deployment
• Examples — Usage examples and sample code
• CHANGELOG — Version history
• CONTRIBUTING — Contributing guidelines
• CODE OF CONDUCT — Community code of conduct
• SECURITY — Security policy and vulnerability reporting

🧪 Testing

# Install dev dependencies
pip install webscout-mcp[dev]

# Run all tests
pytest tests/

# Run with coverage
pytest tests/ --cov=webscout_mcp --cov-report=html

Test coverage: 395+ tests covering all modules.

🤝 Contributing

Contributions are welcome! Please see CONTRIBUTING.md for guidelines.

1. Fork the repository
2. Create a feature branch
3. Make your changes
4. Add tests
5. Submit a pull request

📄 License

MIT License — see LICENSE for details.

🙏 Acknowledgments

• trafilatura — content extraction
• readability-lxml — readability fallback
• Playwright — browser automation
• ChromaDB — vector database
• sentence-transformers — text embeddings
• Ollama — local LLM runtime
• httpx — HTTP client
• BeautifulSoup — HTML parsing

---

Built with ❤️ for the AI agent community.