AI Web Scraper

A robust web scraper that extracts webpage content as clean Markdown, served via a FastAPI backend with a real-time streaming web UI.

Features

• MCP Server — Exposes scraping tools via the Model Context Protocol so AI agents (Claude Desktop, Cursor, Antigravity, etc.) can scrape web pages directly
• Real-time Web UI — Custom HTML/CSS/JS dashboard powered by FastAPI and Server-Sent Events (SSE) for live progress updates
• Smart content extraction — Trafilatura-first with BeautifulSoup pre-processing for maximum coverage
• Anti-bot bypass — Uses curl_cffi to impersonate real browsers (Chrome TLS fingerprint)
• Retry with backoff — Automatic retries with exponential backoff on failed requests
• Markdown output — Clean Markdown with title, source URL, and extracted content
• Rich metadata — Returns author, date, description, site name, language, word count, and Open Graph image alongside content
• Batch scraping — Scrape multiple URLs from a text file or the web UI
• Sitemap support — Extract URLs from XML sitemaps (including sitemap indexes) with recursive resolution
• URL validation — Validates all URLs before attempting to fetch
• Markdown cleanup — Fixes header levels, merges broken definition lists, and removes empty bullets from Trafilatura output

Installation

Requires Python 3.12+ and uv.

# Clone the repository
git clone https://github.com/SouravKAgarwal/web-scraper.git
cd web-scraper

# Install dependencies
uv sync

Usage

Web UI (Recommended)

Launch the FastAPI server to use the web interface:

uvicorn server:app --host 0.0.0.0 --port 8000 --reload

This starts the server at http://localhost:8000. Open it in your browser to:

• Enter one or more URLs (one per line) or a sitemap URL
• Set an optional limit on the number of pages to scrape
• Watch real-time progress as each page is scraped
• View rendered Markdown results with syntax highlighting

MCP Server (for AI Agents)

Run MDScrape as an MCP server so AI coding assistants can scrape web pages directly:

uv run mcp_server.py

Configure in Claude Desktop / Cursor / Antigravity:

{
  "mcpServers": {
    "mdscrape": {
      "command": "uv",
      "args": ["run", "--directory", "/path/to/web-scraper", "mcp_server.py"]
    }
  }
}

Available tools:

Tool	Description
scrape_url	Scrape a single page → Markdown + metadata
scrape_urls	Scrape multiple pages in batch
scrape_sitemap	Crawl a sitemap and scrape all pages
extract_sitemap_urls	List all URLs in a sitemap without scraping

CLI Usage

mdscrape is globally installed on your system. You can run it directly from any directory:

# Scrape a single URL
mdscrape https://example.com/article

# Scrape multiple URLs from a file
mdscrape -f urls.txt

# Scrape with limit
mdscrape -f urls.txt -n 10

# Directly crawl and scrape pages from a sitemap
mdscrape -s https://example.com/sitemap.xml -n 20

# Extract URLs from a sitemap without scraping (print to stdout)
mdscrape -s https://example.com/sitemap.xml --urls-only

Project Structure

web-scraper/
├── cli.py               # Main CLI interface (URLs, files, sitemaps)
├── mcp_server.py        # MCP server exposing scraping tools for AI agents
├── server.py            # FastAPI server serving Web UI and /scrape API
├── helpers/             # Utility functions
│   ├── http.py          # HTTP fetch with retries, URL validation
│   └── markdown.py      # Markdown post-processing and cleanup
├── services/            # Core business logic
│   ├── scraper.py       # Page fetching, BS4 + Trafilatura extraction
│   └── sitemap.py       # XML sitemap parsing & recursive traversal
├── static/              # Web UI assets
│   ├── index.html       # UI layout
│   ├── script.js        # SSE stream listener & renderer
│   └── style.css        # UI styling
├── pyproject.toml       # Project metadata & dependencies
├── uv.lock              # Dependency lockfile
└── .python-version      # Python version (3.12)

How It Works

flowchart LR
    A["URL Input"] --> B["curl_cffi Fetch"]
    B --> C["BS4 Pre-process"]
    C --> D{"Trafilatura\nExtract"}
    D -- "Success" --> E["Clean Markdown"]
    D -- "Fail" --> F["Error"]
    E --> G["Return JSON\nwith Metadata"]

1. Fetch — Downloads the page HTML using curl_cffi with Chrome browser impersonation to bypass anti-bot measures. Retries up to 3 times with exponential backoff.
2. Pre-process — Unwraps nested tags inside <pre> blocks with BeautifulSoup to prevent code block fragmentation in Trafilatura's output.
3. Extract — Passes the cleaned HTML to Trafilatura for intelligent content extraction as Markdown (with links and images). Also extracts metadata (title, author, date, description, etc.).
4. Clean — Post-processes the Markdown: downgrades header levels, merges broken definition lists, and removes empty bullets.
5. Return — Returns structured JSON containing the Markdown content and all extracted metadata.

API Reference

POST /scrape

Streams scrape results via Server-Sent Events (SSE).

Request body:

{
  "urls": ["https://example.com/page1", "https://example.com/page2"],
  "limit": 0
}

Request Example:

import json

import requests

result = {}


def scrape_result(urls: list, limit: int = 0) -> dict:
    resp = requests.post(
        url="http://localhost:8000/scrape",
        json={"urls": urls, "limit": limit},
        stream=True,
    )

    for line in resp.iter_lines(decode_unicode=True):
        if line.startswith("data:"):
            event = json.loads(line[len("data:") :])
            if event["type"] == "done":
                del event["type"]
                result.update(event)

    return result


if __name__ == "__main__":
    urls =  ["https://example.com/page1", "https://example.com/page2"]
    limit = 2

    result = scrape_result(urls, limit)
    print(json.dumps(result, indent=4))

Field	Type	Description
urls	list[str]	URLs to scrape, or a single sitemap URL if sitemap is true
sitemap	bool	Treat the first URL as a sitemap and extract URLs from it
limit	int	Max URLs to scrape (0 = unlimited)

SSE event types:

type	Description
progress	Sent before scraping each URL (current, total, url)
result	Scraped data for one URL (or error)
info	Informational messages (e.g., sitemap extraction count)
done	Final summary (successful, failed, total, response_time)
error	Fatal error (e.g., no valid URLs)

CLI Reference

main.py

Flag	Description	Default
url	A single URL to scrape (positional)	—
-f, --file	Text file with URLs (one per line)	—
-s, --sitemap	Sitemap XML URL to crawl	—
-n, --limit	Max number of URLs to process	Unlimited
--urls-only	Print sitemap URLs without scraping	False

If no URL, file, or sitemap is provided, prompts for manual input.

Output is printed as JSON to stdout (or URLs list with --urls-only).

---

License

MIT