DocHarvest

Turn Any Documentation Site into LLM-Ready Markdown, Vector Context & Offline Books

Zero-Config CLI · React Desktop GUI · Native FastMCP Server · Pure-Python PDF Studio

⚡ Overview

Your coding agent doesn't read documentation — it reads web pages. Navbars, cookie banners, search modals, and footer scripts can make up 80–85% of a raw page's bytes before a single API fact arrives. Chunks captured without source URLs make hallucinations unfalsifiable, and per-page cloud API bills spike the moment you index a real docs portal.

DocHarvest (package: gitbook-downloader) fixes that locally, in one command. It detects the documentation platform, bounds the crawl strictly to documentation subpaths, extracts clean markdown via direct .md endpoint probing and AST-based DOM cleaning, and compiles a deterministic, noise-free knowledge corpus — measured at ~83% token reduction on a real portal (full-suite reference capture: 673 pages in 18.2 seconds).

Whether you are feeding 500-page API manuals to Cursor / Claude Code, building vector RAG pipelines with LangChain & LlamaIndex, reading offline on an airplane, or archiving technical libraries — every capture ends in the same verifiable shape: clean markdown with SHA-256 provenance, ready for your agent or your bookshelf.

⏱️ 30-Second Start

pip install gitbook-downloader
docharvest capture https://docs.openalgo.in/ --rag --pdf

No API key. No account. No telemetry. When the command finishes you own a book.md, an llms.txt, a RAG JSONL dataset, and a printable PDF — all local, all MIT. Full install paths (standalone .exe, uvx, optional extras) are in Quick Start.

🧩 Supported Documentation Platforms (8 Real Providers)

DocHarvest features dedicated, priority-ordered parsers that extract clean article content and strip headers, footers, sidebars, anchor hashes, and cookie banners:

🌟 Key Capabilities

• 🤖 Zero-Noise LLM Context: Auto-detects 8 documentation frameworks, probes native .md endpoints, and cleans DOM trees — measured at ~83% token reduction vs raw pages.
• 📦 Four-Part Output Contract: Every capture yields a modular pages/ tree with SHA-256 YAML frontmatter, a consolidated book.md with TOC, a standardized llms.txt manifest, and search index records.
• 🚀 Export Studio & Local Search: RAG JSONL for vector databases, pure-Python PDF handbooks (fpdf2, zero C-dependencies), and AST markdown chunks — all indexed into embedded SQLite FTS5 BM25 search.
• 🔌 Native FastMCP v2 Server: 12 MCP tools plus resources and prompts over stdio, with ready-made configs for 14 AI clients (Cursor, Claude Code/Desktop, Windsurf, VS Code & more). Crash-safe atomic storage and semver snapshot diffing included.

What DocHarvest Is Not For

A tool that claims to do everything has earned none of your trust. Honest scope:

• Not for login-walled, paywalled, or CAPTCHA-protected content. DocHarvest is built for public technical documentation and will not bypass access controls.
• Not for internet-scale crawling. Millions of arbitrary URLs is Common Crawl / Scrapy territory; this is a documentation compiler, not a search-engine crawler.
• Not for e-commerce or social feeds. Product catalogs and social streams are out of scope by design.
• Not a cloud service. No dashboard, no subscription, no telemetry — because nothing of yours ever leaves your machine.

🖥️ Desktop GUI: Document Library & Management

The desktop application includes a dedicated Document Library for managing all harvested technical docs:

• Instant Search: Real-time filtering across downloaded portals.
• In-App Renaming: Organize project labels without breaking file paths.
• 1-Click Export Studio: Export selected docsets to Markdown, Vector JSONL, or PDF handbooks directly from the GUI.
• Direct Finder/Explorer Integration: Open raw markdown source folders with a single click.

📋 The Four-Part Output Contract

What a generic crawler hands your LLM (every page, every time):

<nav class="sidebar">…47 links…</nav>
<div class="cookie-banner">We value your privacy…</div>
<main>
  <h1>OAuth 2.0<a class="anchor" href="#oauth2">¶</a></h1>
  <pre><code><span class="token-keyword">import</span> <span class="token-variable">requests</span>…</code></pre>
</main>

What DocHarvest delivers — the same page, with cryptographic provenance:

---
source_url: https://docs.openalgo.in/v/v2.0/api-reference/oauth
title: "OAuth 2.0 Authentication"
content_hash: "sha256-2fa9ca2a57c4e974f1725657f88f757e25b90adee3e18ef809f65932d283746c"
---

# OAuth 2.0 Authentication

## Request Signature

```python
import requests

response = requests.post(
    "https://api.openalgo.in/oauth/token",
    json={"client_id": "pk_live_..."},
)
```

The content_hash is the real SHA-256 of the markdown body shown above — paste it into any SHA-256 tool and it verifies.

Every crawl produces the same standardized, deterministic directory structure:

~/.gitbook-downloader/docs/
└── docs.openalgo.in/
    ├── pages/                     # Modular individual markdown files
    │   ├── 001_quickstart.md
    │   └── 002_api_reference.md
    ├── book.md                    # Consolidated single handbook with hierarchical TOC
    ├── llms.txt                   # Standardized AI discovery manifest
    ├── exports/
    │   ├── openalgo_rag.jsonl     # Tokenized vector chunks + metadata
    │   └── openalgo_handbook.pdf  # Publication-grade printable PDF (pure Python)
    └── .manifest.json             # Crawl metadata, engine version & cryptographic hashes

🚀 Quick Start

Option 1: Standalone Executable (Zero Setup)

Download docharvest-windows-latest.exe from the latest release:

• Double-click to launch the Desktop GUI Application.
• Or execute directly in your terminal:

  .\docharvest-windows-latest.exe crawl https://docs.openalgo.in/ --rag --pdf
  

Option 2: Install via pip / PyPI

# Standard installation (100% local, zero C-dependencies)
pip install gitbook-downloader

# Optional headless browser rendering for dynamic JavaScript SPAs
pip install "gitbook-downloader[render]"
playwright install chromium

# MCP server: the FastMCP SDK ships in the base install — nothing extra needed.
# (The [mcp] extra is a backward-compatibility no-op; this line still resolves.)
pip install "gitbook-downloader[mcp]"

# Launch desktop GUI:
docharvest --gui

# Or run a CLI crawl:
docharvest capture https://docs.openalgo.in/ --rag --pdf

Option 3: Ultra-Fast One-Liner via uv / uvx

# Launch GUI instantly without permanent installation:
uvx gitbook-downloader --gui

# Or install as a global CLI tool:
uv tool install gitbook-downloader

💻 CLI Command Reference

# Basic Documentation Crawl (aliases: capture, dl, crawl)
# `docharvest capture` is the canonical verb; `crawl` remains as a documented alias.
docharvest capture https://docs.openalgo.in/

# Full Compilation (Markdown + RAG JSONL + llms.txt + PDF Handbook)
docharvest capture https://docs.openalgo.in/ --rag --pdf

# Crawl Dynamic Client-Rendered SPAs (Playwright Headless Browser)
docharvest capture https://omp.sh/docs --render

# Restrict Crawl to Specific Path Prefix & Limit Depth
docharvest capture https://docs.example.com/ --scope /api/ --max-pages 50

# Capture a Local Dev Docs Server (host:port domains are fully supported)
docharvest capture http://localhost:3000/docs --output library

# Full-Text BM25 Search across Harvested Docs
docharvest search "OAuth 2.0 authentication token"

# List Harvested Document Domains in Local Library
docharvest ls

# Show Snapshot History & Diff Versions
docharvest history docs.example.com
docharvest diff docs.example.com v1.0.0 v1.0.1

# Start FastMCP Server over Stdio for AI IDEs
docharvest --mcp

# List / Install the Bundled Agent Skill
docharvest skill list
docharvest skill install docharvest -o .claude/skills

# Launch Desktop GUI Application
docharvest --gui

🧠 Agent Skills: Teaching Harnesses to Use DocHarvest

The MCP server gives an agent tools; a skill tells it when and how to use them. DocHarvest ships one skill, docharvest, describing the download_docs → search_docs/read_doc → export_docs workflow, the library layout, and the CLI fallback for harnesses without MCP.

The skill is bundled inside the installed package, so it is versioned with the code and available to any harness without committing a copy per tool:

# See what is bundled and where the canonical file lives
docharvest skill list

# Install into the directory a harness actually reads
docharvest skill install docharvest -o .agents/skills     # default
docharvest skill install docharvest -o .claude/skills
docharvest skill install docharvest -o .cursor/skills
docharvest skill install docharvest -o .gemini/skills
docharvest skill install docharvest -o .github/skills
docharvest skill install docharvest -o .omp/skills

Layout written is <target>/docharvest/SKILL.md — skills are discovered one level under a skills/ root, so nested paths are not picked up. Existing files are left untouched unless you pass --force.

Out of the box: this repository also commits the skill at .omp/skills/docharvest/SKILL.md together with .omp/mcp.json, so opening the checkout in OMP registers both the server and the skill with no setup. Invoke it explicitly with /skill:docharvest <doc-url | search query>.

🔌 AI Agent Integration: Native FastMCP v2 Server

DocHarvest includes a native FastMCP (Model Context Protocol v2) server that exposes 12 high-level tools, MCP Resources, and MCP Prompts over standard input/output (stdio). It is compatible with both mcp<2 and mcp>=2.1.

The mcp SDK ships in the base install. pip install gitbook-downloader (or uvx gitbook-downloader mcp) is enough — no extras required. The gitbook-downloader[mcp] extra is still accepted for backward compatibility but is now a no-op.

All 12 Native MCP Tools

1. download_docs(url, max_pages=None, workers=8, path_scope=[], exclude_paths=[], site_versions=None, output_mode="both") Captures any documentation URL into Markdown, book.md, and llms.txt.
2. search_docs(query, domain=None, limit=10) Full-text search across downloaded documentation via SQLite FTS5 BM25.
3. find_docs(query, limit=10) Resolves library/framework names ("react", "nextjs") to indexed domains in the local library.
4. read_doc(domain, path=None, topic=None, max_tokens=4000, version=None) Reads a specific page or topic section with AST-safe token bounding — code blocks and tables are never split.
5. get_doc(domain, version=None) Retrieves the compiled documentation content or preview for a domain.
6. list_domains() Returns metadata for all harvested documentation portals in local storage.
7. query_doc_graph(domain, query, limit=10) Queries the semantic entity & concept graph to discover connected API endpoints and sections without reading full files.
8. get_related_concepts(domain, concept) Returns 1-hop and 2-hop connected concepts and prerequisite sections.
9. diff_versions(domain, v1, v2) Computes unified diffs and line change statistics between two snapshots.
10. list_versions(domain) Lists available captured snapshots and timestamps for a domain.
11. export_docs(domain, format="markdown") Exports documentation into "markdown", "jsonl", or "rag" metadata formats.
12. get_changelog(domain) Auto-generates version changelogs across captured snapshot iterations.

MCP v2 Resources & Prompts

• Resources: docs://{domain}/book (full handbook), docs://{domain}/manifest (llms.txt index).
• Prompts: prompt://search-docset (guided docset synthesis), prompt://summarize-library (library overview).

IDE & Agent Configuration Matrix (14 Clients)

The three most common clients are shown inline — expand the list for all 14.

1. Claude Code

claude mcp add docharvest docharvest mcp

Or in ~/.claude.json:

{
  "mcpServers": {
    "docharvest": {
      "command": "docharvest",
      "args": ["mcp"]
    }
  }
}

2. Claude Desktop (claude_desktop_config.json)

{
  "mcpServers": {
    "docharvest": {
      "command": "uvx",
      "args": ["gitbook-downloader", "mcp"]
    }
  }
}

3. Cursor (.cursor/mcp.json)

{
  "mcpServers": {
    "docharvest": {
      "command": "python",
      "args": ["-m", "gitbook_downloader.mcp"]
    }
  }
}

{
  "mcpServers": {
    "docharvest": {
      "command": "docharvest",
      "args": ["mcp"]
    }
  }
}

{
  "servers": {
    "docharvest": {
      "type": "stdio",
      "command": "docharvest",
      "args": ["mcp"]
    }
  }
}

{
  "context_servers": {
    "docharvest": {
      "command": "docharvest",
      "args": ["mcp"]
    }
  }
}

{
  "mcpServers": {
    "docharvest": {
      "command": "docharvest",
      "args": ["mcp"],
      "disabled": false,
      "autoApprove": ["search_docs", "get_doc", "list_domains", "query_doc_graph"]
    }
  }
}

{
  "experimental": {
    "modelContextProtocolServers": [
      {
        "transport": {
          "type": "stdio",
          "command": "docharvest",
          "args": ["mcp"]
        }
      }
    ]
  }
}

{
  "mcp": {
    "servers": {
      "docharvest": {
        "command": "docharvest",
        "args": ["mcp"]
      }
    }
  }
}

{
  "mcp": {
    "docharvest": {
      "command": "docharvest",
      "args": ["mcp"]
    }
  }
}

{
  "$schema": "https://raw.githubusercontent.com/can1357/oh-my-pi/main/packages/coding-agent/src/config/mcp-schema.json",
  "mcpServers": {
    "docharvest": {
      "type": "stdio",
      "command": "uvx",
      "args": ["gitbook-downloader", "mcp"],
      "timeout": 0
    }
  }
}

{
  "name": "docharvest",
  "command": "docharvest",
  "args": ["mcp"]
}

{
  "mcp_servers": {
    "docharvest": {
      "command": "docharvest",
      "args": ["mcp"]
    }
  }
}

🐍 Python SDK Example

from gitbook_downloader.api import capture, CaptureOptions

# Configure capture options
options = CaptureOptions(
    workers=8,
    output_mode="both",   # "both", "library", or "local"
    render=False,         # set True for dynamic JavaScript SPAs
)

# Execute deterministic capture
result = capture("https://docs.openalgo.in/", options=options)

print(f"Captured {result.pages_captured} pages using provider: {result.provider}")
print(f"Book file: {result.book_file}")
print(f"Manifest:  {result.manifest_file}")

🧪 Test Suite & Quality Benchmarks

DocHarvest is continuously tested across Windows, Linux, and macOS:

• 686 Automated Tests: 100% pass rate across engine discovery, BFS crawling, provider extraction, storage safety, DocGraph semantic search, and MCP v2 tools (verified on this release).
• 73%+ Statement Coverage: Rigorous test suites covering error recovery, invalid signatures, domain locks, and AST link normalization.
• Windows CRLF Safe: All link and boilerplate stripping routines are cross-platform normalized against Windows CRLF and Unix LF linebreaks.

To run the test suite locally:

uv run pytest --cov=gitbook_downloader

👨‍💻 Author & Connect

Created with ❤️ by Rohan Shetty.

• 🌐 Website: rohannshetty.github.io/gitbook-downloader
• 🐙 GitHub: @RohannShetty
• 💼 LinkedIn: linkedin.com/in/rohan-shettyy
• 🐦 X (Twitter): @rohan__shetty
• 📧 Email: shettyrohan2@gmail.com

📄 License

This project is licensed under the MIT License.