feishu-wiki-scrape 0.1.0

A Python library to scrape Feishu wiki pages and convert them to Markdown

Feishu Wiki Scraper

A Python library to scrape Feishu (飞书) wiki pages and convert them to Markdown format, similar to Firecrawl. This tool can scrape entire wiki sites by following sidebar links and extracting all content.

Features

• 🚀 Scrape single Feishu wiki pages or entire wiki sites
• 📝 Convert HTML content to clean Markdown format
• 🔗 Automatically follow sidebar links to scrape related pages
• 🍪 Support for authentication via cookies and custom headers
• ⚙️ Configurable scraping options (delays, max pages, etc.)
• 💾 Export to Markdown files or JSON format
• 📂 Directory output mode — save each page as a separate .md file preserving wiki tree structure
• 🎯 Command-line interface for easy usage
• 🔥 Firecrawl-compatible JSON output with metadata

Installation

From source

git clone https://github.com/rwifeng/feishu-wiki-scrape.git
cd feishu-wiki-scrape
pip install -e .

Dependencies

pip install -r requirements.txt

Usage

Command Line Interface

Basic usage to scrape a Feishu wiki:

# Save to a single Markdown file
feishu-wiki-scrape https://zcn3fx96oxg4.feishu.cn/wiki/H5V5wMczPif5A5khSG3cWx65nbc -o output.md

# Save as a directory tree (one .md file per page, preserving wiki structure)
feishu-wiki-scrape https://zcn3fx96oxg4.feishu.cn/wiki/H5V5wMczPif5A5khSG3cWx65nbc -o ./wiki-docs/

Options

• -o, --output: Output path (default: output.md). If the path ends with /, is an existing directory, or has no file extension, each page is saved as a separate .md file in a nested directory tree matching the wiki structure
• --max-pages: Maximum number of pages to scrape (default: unlimited)
• --no-sidebar: Don't follow sidebar links (scrape only the given URL)
• --delay: Delay between requests in seconds (default: 1.0)
• --cookies: Cookies as JSON string for authentication
• --headers: Custom headers as JSON string
• --json-output: Output as JSON instead of Markdown file
• --firecrawl-format: Output in Firecrawl-compatible JSON format with metadata
• -v, --verbose: Enable verbose logging

Examples

Scrape a single page without following links:

feishu-wiki-scrape https://example.feishu.cn/wiki/page --no-sidebar -o single_page.md

Scrape with authentication cookies:

feishu-wiki-scrape https://example.feishu.cn/wiki/page \
  --cookies '{"session_id": "your-session-id"}' \
  -o authenticated_output.md

Limit to 10 pages with custom delay:

feishu-wiki-scrape https://example.feishu.cn/wiki/page \
  --max-pages 10 \
  --delay 2.0 \
  -o limited_output.md

Output as JSON:

feishu-wiki-scrape https://example.feishu.cn/wiki/page --json-output > output.json

Save as directory tree preserving wiki structure:

feishu-wiki-scrape https://example.feishu.cn/wiki/page -o ./docs/

This produces a directory tree like:

docs/
  🚀 Introduction/
    index.md          # parent page with children
    Getting Started.md
  FAQ/
    index.md
    Common Errors.md
  Claude.md           # leaf page (no children)

Python API

from feishu_wiki_scrape import FeishuWikiScraper

# Create scraper instance
scraper = FeishuWikiScraper(
    cookies={"session_id": "your-session-id"},  # Optional
    headers={"Custom-Header": "value"},          # Optional
    delay=1.0                                    # Delay between requests
)

# Scrape a single page
page = scraper.scrape_page("https://example.feishu.cn/wiki/page")
print(page["title"])
print(page["markdown"])

# Scrape entire wiki (follows sidebar links)
results = scraper.scrape_wiki(
    start_url="https://example.feishu.cn/wiki/page",
    max_pages=50,           # Optional: limit number of pages
    include_sidebar=True    # Follow sidebar links
)

for page in results:
    print(f"Title: {page['title']}")
    print(f"URL: {page['url']}")
    print(f"Content:\n{page['markdown']}\n")

# Save to file
scraper.scrape_to_file(
    start_url="https://example.feishu.cn/wiki/page",
    output_file="output.md",
    max_pages=None,         # No limit
    include_sidebar=True
)

# Save to directory tree (preserves wiki sidebar structure)
count = scraper.scrape_wiki_to_directory(
    start_url="https://example.feishu.cn/wiki/page",
    output_dir="./docs/",
    max_pages=None          # No limit
)
print(f"Saved {count} pages")

Firecrawl-Compatible Output

This library supports Firecrawl-compatible JSON output with rich metadata, making it easy to build API-compatible tools.

Using CLI

feishu-wiki-scrape https://example.feishu.cn/wiki/page \
  --firecrawl-format \
  --max-pages 10 > output.json

Output format:

{
  "success": true,
  "status": "completed",
  "completed": 10,
  "total": 10,
  "data": [
    {
      "markdown": "# Page Title\n\nPage content...",
      "metadata": {
        "url": "https://example.feishu.cn/wiki/page",
        "title": "Page Title",
        "keywords": "keyword1, keyword2",
        "language": "zh-CN",
        "sourceURL": "https://example.feishu.cn/wiki/page",
        "statusCode": 200,
        "contentType": "text/html; charset=utf-8",
        "description": "Page description"
      }
    }
  ]
}

Using Python API

from feishu_wiki_scrape import FeishuWikiScraper

scraper = FeishuWikiScraper()

# Scrape with metadata
results = scraper.scrape_wiki_with_metadata(
    start_url="https://example.feishu.cn/wiki/page",
    max_pages=10,
    include_sidebar=True
)

# Format as Firecrawl response (automatically handles metadata format)
firecrawl_response = scraper.format_as_firecrawl(results, start_url)

print(firecrawl_response)

For a complete example of building a Firecrawl-compatible API, see example_firecrawl.py.

How It Works

1. Page Fetching: Uses requests to fetch wiki pages with configurable headers and cookies
2. Content Extraction: Parses HTML with BeautifulSoup to extract main content area
3. Link Discovery: Finds all wiki links in sidebars and navigation elements
4. Markdown Conversion: Converts HTML to clean Markdown using html2text
5. Crawling: Follows links breadth-first to scrape entire wiki sites
6. Rate Limiting: Respects configurable delays between requests

Authentication

Feishu wikis may require authentication. You can provide cookies or headers:

Getting Cookies

1. Open the Feishu wiki in your browser
2. Open Developer Tools (F12)
3. Go to Application/Storage > Cookies
4. Copy the relevant cookie values
5. Pass them using --cookies option or in Python code

Example:

feishu-wiki-scrape https://example.feishu.cn/wiki/page \
  --cookies '{"session_id": "abc123", "other_cookie": "value"}'

Output Format

Single Markdown File (-o output.md)

Pages are separated by horizontal rules (---) with each page containing:

• Page title as H1 heading
• Source URL
• Markdown content

Directory Tree (-o dir/)

Each wiki page is saved as a separate .md file. The directory structure mirrors the wiki's sidebar tree:

• Pages with children become a directory containing index.md (the page content) plus child pages
• Leaf pages (no children) are saved as {title}.md in the parent directory
• The wiki space root container is skipped so the output directory maps directly to the top-level pages

JSON Format

[
  {
    "url": "https://example.feishu.cn/wiki/page1",
    "title": "Page Title",
    "markdown": "# Content\n\nPage content in markdown..."
  },
  {
    "url": "https://example.feishu.cn/wiki/page2",
    "title": "Another Page",
    "markdown": "# Content\n\nMore content..."
  }
]

Troubleshooting

Pages not loading

• Check if authentication is required (try with cookies)
• Verify the URL is accessible in a browser
• Increase delay between requests

Missing content

• Some content may be loaded dynamically with JavaScript
• Try using cookies from an authenticated session
• Check verbose output with -v flag

License

MIT License

Contributing

Contributions are welcome! Please feel free to submit a Pull Request.