site_to_markdown

This tool is a web scraper built using Scrapy that extracts and consolidates content from a documentation website into a single Markdown file. It is flexible, allowing you to specify a starting URL, restrict crawling to specific domains, and define the output file name.

Features

• Crawls a documentation website starting from a given URL.
• Automatically extracts the main content using Readability while skipping non-relevant elements (headers, sidebars, etc.).
• Consolidates all pages into a single Markdown file with structured headings.
• Allows optional restriction to specific domains.
• Skips non-English pages to keep the output consistent.

Installation

This project is now available as a pip package. Install it using:

pip install site-to-markdown

Usage

The scraper can now be run directly as a command-line tool.

Command Syntax

site-to-markdown \
  -u <STARTING_URL> \
  [-d <DOMAIN1,DOMAIN2>] \
  [-o <OUTPUT_FILENAME>] \
  [-c <PATH_TO_COOKIE_JSON_FILE>]
  [-e <EXCLUDED_FILETYPE_1,EXCLUDED_FILETYPE_2>]

Arguments

1. -u (required): The starting URL for the crawler. The scraper will begin its crawl from this URL.

   • Example: https://example-docs-site.com

2. -d (optional): A comma-separated list of domains to restrict the crawl. If not provided, the scraper will infer the domain from the start_url.

   • Example: example-docs-site.com,docs.example.com

3. -o (optional): The name of the output Markdown file. Default is documentation.md.

   • Example: output.md

4. -c (optional): The path to a JSON file of cookies to use for requests. Default is None.

   • Example: ./cookies.json

5. -e (optional): A comma-separated list of file types to exclude. This filtering is done based on the URL path, not the Content-Type.

   • Example: ./cookies.json

Example Usage

Basic Crawling

To crawl a single domain:

site-to-markdown -u [https://example-docs-site.com](https://example-docs-site.com)

Multiple Domains

To allow crawling across multiple domains:

site-to-markdown -u [https://example-docs-site.com](https://example-docs-site.com) -d example-docs-site.com,docs.example.com

Custom Output File

To specify a custom output file:

site-to-markdown -u [https://example-docs-site.com](https://example-docs-site.com) -o my_documentation.md

Exclude Filetypes

To exclude particular file types based on the HTTP Path.

site-to-markdown -u [https://example-docs-site.com](https://example-docs-site.com) -e rst.txt,md 

Output Format

The scraper generates a single Markdown file with the following structure:

# Documentation

## Page 1 Title

Content for Page 1...

## Page 2 Title

Content for Page 2...

...

Notes

• Non-Text Content: The scraper skips non-HTML pages (e.g., images, PDFs).
• Non-English Pages: Only English pages are processed.
• URL Validation: Ensures only valid URLs are crawled (ignores javascript:, mailto:, etc.).
• File Overwriting: If the output file already exists, it will be overwritten.

Contributing

Contributions are welcome! Feel free to submit pull requests or report issues on GitHub.