🔌 RAGify Docs: Recursive Documentation Scraper MCP Server

A production-ready Model Context Protocol (MCP) server that empowers AI agents to recursively scrape entire software documentation sites, compile them into an in-memory vector store, and provide accurate, context-grounded answers to developer questions.

Built using the FastMCP Framework, LangChain, and Ollama, this server lets tools like Claude Desktop, Cursor, or Zed read documentation pages in real-time to resolve coding problems without leaving the chat interface.

🔥 Key Features

• Recursive Deep Scraping: Crawls documentation sites up to two levels deep out of the box using custom beautifulsoup extraction.
• Dynamic Context Assembly: Automatically splits raw website texts into clean, code-aware semantic blocks.
• High-Diversity MMR Search: Uses Maximal Marginal Relevance to fetch contrasting context points rather than duplicating search matches from single sections.
• Local-First Architecture: Leverages a lightweight in-memory vector index alongside local llama3.2 models for privacy and cost efficiency.
• Safe Data Pipeline: Explicitly channels all background terminal metrics into stderr to avoid protocol communication corruption over standard I/O channels.

🛠️ Prerequisites

Before installing the server, ensure your local environment contains the following applications:

• Python: version 3.10 or higher.
• Ollama: Installed and running locally with the target model pulled:

  ollama pull llama3.2
  

📦 Installation & Setup

Follow these steps to download and configure the project directory on your local machine.

1. Clone and Navigate to the Project

git clone https://github.com
cd ragify_docs_2.0

2. Set Up a Virtual Environment

Create and boot up an isolated Python runtime container to avoid package conflicts:

# Windows PowerShell
python -m venv .venv
.venv\Scripts\Activate.ps1

# macOS / Linux
python3 -m venv .venv
source .venv/bin/activate

3. Install Dependencies

Install the required framing libraries directly via pip:

pip install fastmcp langchain-community langchain-text-splitters langchain-huggingface langchain-core langchain-ollama tqdm bs4 lxml sentence-transformers python-dotenv

4. Create Environment Configuration

Create a configuration file named .env in the root of your workspace:

# .env Configuration File
# Note: You can optionally add service platform API credentials here if swapping components

🚀 Local Testing & Debugging

The platform provides simple verification mechanisms to make sure everything links up fine before giving it over to real apps.

Running via FastMCP

Execute the framework inspection validator to ensure the internal tool parsing definitions register correctly:

fastmcp inspect .\src\ragify_docs\tools.py

To boot up the live server instance locally via your custom python orchestration file, run:

python .\src\ragify_docs\main.py

(You will see informational logging indicate the server is successfully listening for external connections over the standard I/O communication channel).

💻 Client Integration (Claude Desktop)

To expose this capability directly to Claude Desktop, add its execution routing coordinates to your application parameters profile.

1. Press Win + R (Windows) and paste the following directory location to open your config file:

   %APPDATA%\Claude\claude_desktop_config.json
   

2. Inject the following block into the structure (Ensure you escape all absolute disk location paths with double-backslashes \\):

{
  "mcpServers": {
    "ragify-docs-server": {
      "command": "C:\\Users\\yasho\\OneDrive\\Desktop\\ragify_docs_2.0\\.venv\\Scripts\\python.exe",
      "args": [
        "C:\\Users\\yasho\\OneDrive\\Desktop\\ragify_docs_2.0\\src\\ragify_docs\\main.py"
      ],
      "env": {}
    }
  }
}

3. Completely Restart Claude Desktop. If correctly parsed, a 🔌 Hammer Icon will materialize inside your message submission console tray!

📖 Usage Guide & Example Prompts

Once integrated, you do not need to format your query manually. Simply prompt the AI agent with a URL and a specific engineering problem:

• "Scrape https://yashodeep.me and tell me what the Anuvadak project is."
• "Look up the framework guidelines at https://example.com and show me how to initialize its basic auth route."