website2md 0.1.8

Convert websites to markdown format using crawl4ai framework

Website2MD - Convert Websites to Markdown

A sophisticated web crawler that converts websites to markdown format using crawl4ai framework. Specializes in creating LLM-ready content from documentation sites, full websites, or URL lists with advanced JavaScript rendering and intelligent content selection.

📋 项目概述

Website2MD 是一个强大的网站内容抓取和转换工具，专门用于将网站内容转换为高质量的 Markdown 格式。特别适合：

• 🤖 AI/LLM 训练数据准备：将文档网站转换为训练数据集
• 📚 知识库构建：从网站批量提取内容构建知识库
• 🔍 内容迁移：将旧网站内容迁移到新平台
• 📖 离线文档：创建文档的离线副本

Features

• 🚀 Auto-detection: Automatically detects input type (site/docs/list)
• 📝 Markdown Output: Converts web content to clean markdown format
• 🤖 LLM-Ready: Optimized for use as LLM context and training data
• 🌐 JavaScript Support: Handles modern SPA websites and dynamic content
• 📚 Documentation Sites: Specialized crawler for docs with menu expansion
• 📋 Batch Processing: Process multiple URLs from files or lists
• ⚡ High Performance: Async processing with smart concurrency
• 🎯 Content Selection: Advanced CSS selectors and exclude patterns
• 🔒 Precise Domain Filtering: Only crawl exact same subdomain by default, with flexible domain control
• 🚫 Content Filtering: Exclude unwanted elements using CSS selectors (ads, popups, navigation, etc.)

Installation

Using pip (PyPI)

pip install website2md

Using uv (Recommended for faster installation)

# Install from PyPI with Chinese mirror for faster speed
uv pip install website2md --default-index https://mirrors.aliyun.com/pypi/simple

# Or use Tsinghua mirror
uv pip install website2md --default-index https://pypi.tuna.tsinghua.edu.cn/simple

From Source (Development)

git clone https://github.com/fengyunzaidushi/website2md.git
cd website2md

# Using uv (recommended)
uv venv
source .venv/Scripts/activate  # Windows
source .venv/bin/activate      # Linux/Mac
uv sync --default-index https://mirrors.aliyun.com/pypi/simple

# Or using pip
pip install -e .

Browser Setup (Required for JavaScript sites)

After installation, install Playwright browsers:

playwright install

Quick Start

Command Line Usage

# Auto-detect and convert website to markdown
website2md https://docs.example.com --output ./docs

# Convert documentation site (auto-detected as 'docs' type)
website2md https://docs.cursor.com --output ./cursor-docs --verbose

# Convert full website (auto-detected as 'site' type)  
website2md https://example.com --output ./website-content

# Process URL list from file
website2md urls.txt --type list --output ./batch-content

# Process URL list directly
website2md "url1,url2,url3" --type list --output ./multi-content

# Specify type explicitly with custom settings
website2md https://example.com --type site --max-pages 50 --output ./results

# Domain filtering: Only crawl exact same subdomain (default behavior)
website2md https://docs.anthropic.com/zh-CN/docs --output ./docs

# Allow additional specific domains
website2md https://docs.anthropic.com/zh-CN/docs \
  --allowed-domains "console.anthropic.com,api.anthropic.com" \
  --output ./docs

# Allow all external domains (use with caution)
website2md https://docs.anthropic.com/zh-CN/docs \
  --allow-external \
  --output ./docs

# Content filtering: Exclude unwanted elements using CSS selectors
website2md https://example.com \
  --exclude-selectors ".advertisement,.popup,.cookie-banner" \
  --output ./clean-content

# Combine multiple options for precise control
website2md https://docs.example.com \
  --type docs \
  --exclude-selectors "nav,.sidebar,.toc" \
  --max-pages 20 \
  --output ./docs

# Windows users: Use UTF-8 encoding to avoid codec errors
PYTHONIOENCODING=utf-8 website2md https://docs.example.com --output ./docs

Python API Usage

from website2md.doc_crawler import DocSiteCrawler
from website2md.url_list_crawler import URLListCrawler
from website2md.config import CrawlConfig

# Crawl documentation site with content filtering
config = CrawlConfig(
    max_pages=100, 
    wait_for_content=True,
    exclude_selectors=["#navigation-items", ".sidebar", ".advertisement"]
)
crawler = DocSiteCrawler(config)
results = await crawler.crawl_documentation_site("https://docs.example.com", "./output")

# Process URL list
url_crawler = URLListCrawler(config)
results = await url_crawler.crawl_url_list("url1,url2,url3", "./output")

📝 Python Examples

Check out the examples/ directory for comprehensive Python usage examples:

• crawl_cursor_docs.py - Crawl Cursor docs excluding #navigation-items
• crawl_anthropic_docs.py - Crawl Anthropic docs with default excludes
• basic_site_crawling.py - News, blog, e-commerce filtering
• advanced_filtering.py - Advanced CSS selector techniques

See examples/README.md for detailed usage instructions and best practices.

💡 Best Practices & Troubleshooting

URL Discovery Optimization

For complex documentation sites or sites with dynamic content, you may get better results by explicitly specifying the crawler type:

# If auto-detection gives poor results, try 'site' type for better URL discovery
website2md https://docs.cursor.com/en/welcome --type site --output ./docs

# The 'site' type uses recursive link discovery, often finding more pages
# than the 'docs' type which relies on menu expansion

Common Issues and Solutions

Issue: Only 1 page crawled instead of full site

Symptom: Expected many pages but only got the starting page.

Solutions:

1. Use --type site for better URL discovery:

   website2md https://docs.example.com --type site --output ./docs
   

2. Check if JavaScript is required: Some sites need JS rendering

   website2md https://spa-site.com --type site --output ./docs
   

Issue: Content contains too much navigation/clutter

Symptom: Generated markdown files contain sidebars, navigation menus, ads.

Solutions:

1. Use exclude selectors to filter unwanted content:

   website2md https://example.com --exclude-selectors "nav,aside,header,footer,.ads" --output ./clean
   

2. For documentation sites, try these common selectors:

   website2md https://docs.site.com --exclude-selectors ".sidebar,.navigation,.toc,.breadcrumb" --output ./docs
   

Issue: Windows encoding errors (GBK codec)

Symptom: 'gbk' codec can't encode character errors.

Solution: Set UTF-8 encoding:

# Windows Command Prompt
set PYTHONIOENCODING=utf-8 && website2md https://example.com --output ./docs

# PowerShell
$env:PYTHONIOENCODING="utf-8"; website2md https://example.com --output ./docs

Issue: Empty or missing content

Symptom: Generated files are empty or missing expected content.

Solutions:

1. Increase timeout for slow sites:

   website2md https://slow-site.com --type site --output ./docs
   

2. Check if playwright browsers are installed:

   playwright install
   

Performance Tips

• Start small: Use --max-pages to test before full crawls
• Use appropriate delays: Add --delay for rate limiting
• Monitor output: Use --verbose to see what's happening
• Choose the right type: site for comprehensive, docs for structured

Type Selection Guide

Site Type	Use --type site when	Use --type docs when
Documentation	Complex navigation, SPA-style docs	Traditional docs with clear menu structure
Corporate sites	Marketing sites, blogs	API docs, help centers
E-commerce	Product catalogs	Knowledge bases
News sites	Article listing pages	FAQ sections

Input Types

Website2MD automatically detects input types based on patterns:

• 📄 Site: Full website crawling (https://example.com)
• 📚 Docs: Documentation sites (https://docs.example.com, /docs/ URLs)
• 📋 List: URL files (.txt files) or comma-separated URL strings

Domain Filtering (New in v0.1.5)

Website2MD now provides precise domain control to ensure you only crawl relevant content:

Default Behavior: Exact Subdomain Matching

• Input: https://docs.anthropic.com/zh-CN/docs
• Only crawls: docs.anthropic.com domain
• Skips: console.anthropic.com, www.anthropic.com, etc.

Flexible Domain Control

# Include specific additional domains
--allowed-domains "console.anthropic.com,api.anthropic.com"

# Allow all external domains (use with caution)
--allow-external

Benefits

• Focused Content: Only get relevant documentation pages
• No Pollution: Avoid unrelated external links
• Better Performance: Fewer unnecessary requests
• Quality Control: Cleaner, more relevant output

Content Filtering (New in v0.1.7)

Website2MD supports powerful content filtering using CSS selectors to exclude unwanted elements from the crawled content.

Basic CSS Selector Syntax

ID Selectors

Use # to target elements by ID:

# Exclude div with id="navigation-items"
website2md https://example.com --exclude-selectors "#navigation-items"

# Exclude header with id="main-header" 
website2md https://example.com --exclude-selectors "#main-header"

Class Selectors

Use . to target elements by class:

# Exclude elements with class="sidebar"
website2md https://example.com --exclude-selectors ".sidebar"

# Exclude multiple classes
website2md https://example.com --exclude-selectors ".advertisement,.popup,.cookie-banner"

Tag Selectors

Target HTML tags directly:

# Exclude all nav elements
website2md https://example.com --exclude-selectors "nav"

# Exclude headers and footers
website2md https://example.com --exclude-selectors "header,footer"

Advanced Selector Examples

Attribute Selectors

# Exclude elements with specific attributes
website2md https://example.com --exclude-selectors "[data-testid='navigation']"

# Exclude elements with specific attribute values
website2md https://example.com --exclude-selectors "[role='banner']"

Descendant Selectors

# Exclude navigation menus inside header
website2md https://example.com --exclude-selectors "#header .navigation-menu"

# Exclude all divs inside sidebar
website2md https://example.com --exclude-selectors ".sidebar div"

Complex Combinations

# Comprehensive content filtering
website2md https://example.com \
  --exclude-selectors "#navigation-items,.sidebar,nav,.advertisement,[data-testid='footer']" \
  --output ./clean-content

Common Use Cases

Remove Website Navigation

website2md https://example.com \
  --exclude-selectors "#navigation,nav,.navbar,.menu" \
  --output ./content-only

Clean Documentation Sites

website2md https://docs.example.com \
  --type docs \
  --exclude-selectors ".toc,.breadcrumb,#sidebar,.docs-navigation" \
  --output ./clean-docs

Filter Marketing Content

website2md https://blog.example.com \
  --exclude-selectors ".advertisement,.cta-banner,.newsletter-signup,.social-share" \
  --output ./articles-only

E-commerce Content Extraction

website2md https://shop.example.com \
  --exclude-selectors ".price,.buy-button,.cart,.recommendations" \
  --output ./product-info

Smart Documentation Filtering

For --type docs, user-specified selectors are automatically combined with default documentation site filters:

Default Documentation Excludes:

• .sidebar, .nav, .navigation, #sidebar
• #starlight__sidebar, .docs-sidebar, .theme-doc-sidebar-container
• .header, .footer, .breadcrumb, .toc
• .border-r-border, .md\\:w-64, .xl\\:w-72

Example with Smart Merging:

# Your selectors are added to the defaults
website2md https://docs.example.com \
  --type docs \
  --exclude-selectors "#custom-banner,.advertisement" \
  --output ./clean-docs

Tips for Effective Filtering

1. Inspect Element: Use browser developer tools to identify element selectors
2. Test Selectors: Start with simple selectors and add complexity gradually
3. Multiple Passes: Use --verbose to see what's being excluded
4. Verify Results: Check the output files to ensure desired content remains

Advanced Configuration

Documentation Sites

from website2md.doc_crawler import DocSiteCrawler
from website2md.config import CrawlConfig

config = CrawlConfig(
    max_pages=200,
    wait_for_content=True,    # Enable JavaScript rendering
    js_wait_time=3.0,         # Wait time for JS execution
    expand_menus=True,        # Auto-click expandable menus
    scroll_for_content=True,  # Scroll to trigger lazy loading
    exclude_selectors=[       # Remove navigation elements
        '.sidebar', '.nav', '.breadcrumb', '.toc'
    ],
    timeout=60
)

crawler = DocSiteCrawler(config, "./docs-output")

Batch URL Processing

from website2md.url_file_crawler import URLFileCrawler

# Process URLs from file
config = CrawlConfig(max_pages=100, headless=True)
crawler = URLFileCrawler(config, "./batch-output")

# From file
results = await crawler.crawl_from_file("urls.txt")

# From list
results = await crawler.crawl_urls(["url1", "url2", "url3"])

Output Structure

All content is saved as individual markdown files in the specified output directory:

output/
├── page1.md
├── page2.md
├── subdir/
│   ├── page3.md
│   └── page4.md
└── crawl_summary.json

Each markdown file contains:

• Clean, LLM-ready content
• Preserved formatting and structure
• Metadata headers (title, URL, timestamp)

Use Cases

• 🤖 LLM Training Data: Convert documentation sites to training datasets
• 📚 Knowledge Bases: Build markdown knowledge bases from websites
• 🔍 Content Migration: Migrate content from old sites to new platforms
• 📖 Offline Documentation: Create offline copies of documentation
• 🎯 Content Analysis: Extract and analyze website content at scale

⚙️ Technical Requirements

System Requirements

• Python: 3.10+ (recommended: 3.11 or 3.12)
• Operating System: Windows, macOS, Linux
• Memory: 2GB+ RAM (4GB+ for large sites)
• Browser: Chromium/Firefox (auto-installed via Playwright)

Core Dependencies

• crawl4ai >= 0.6.0 - Web crawling framework with browser automation
• aiohttp - Async HTTP client for concurrent requests
• beautifulsoup4 - HTML parsing and content extraction
• click - Command-line interface framework
• playwright - Browser automation for JavaScript rendering

Development Tools (Optional)

• uv - Fast Python package manager (recommended)
• black - Code formatting
• flake8 - Code linting
• mypy - Type checking

🛠️ Development & Contributing

Development Setup

# Clone repository
git clone https://github.com/fengyunzaidushi/website2md.git
cd website2md

# Setup development environment with uv (recommended)
uv venv
source .venv/Scripts/activate  # Windows
source .venv/bin/activate      # Linux/Mac

# Install with development dependencies
uv sync --default-index https://mirrors.aliyun.com/pypi/simple

# Install Playwright browsers
playwright install

# Run development checks
black website2md/          # Format code
flake8 website2md/         # Lint code  
mypy website2md/           # Type check

Contributing Guidelines

1. Fork the repository
2. Create a feature branch (git checkout -b feature/amazing-feature)
3. Make your changes following code style guidelines
4. Test your changes with real websites
5. Submit a pull request with clear description

Testing Your Changes

# Test CLI with various site types
PYTHONIOENCODING=utf-8 website2md https://docs.cursor.com --output ./test-output --verbose
website2md https://example.com --output ./test-site --max-pages 5

License

MIT License - see LICENSE file for details.

Support

• 📦 PyPI: website2md
• 📖 Documentation: GitHub Wiki
• 🐛 Issues: GitHub Issues
• 💬 Discussions: GitHub Discussions