Crawlemoon MCP Server

A free, open-source MCP server that gives any agent (Claude Code, Cursor, Windsurf, …) 55 production-grade tools for the full web-crawling stack: deep analysis, stealth, API discovery, session recording → runnable crawler, smart extraction. No proprietary API. No per-request fee.

Quick start

The recommended path needs no install — uvx runs straight from PyPI:

{
  "mcpServers": {
    "crawlemoon": {
      "command": "uvx",
      "args": ["crawlemoon"]
    }
  }
}

Requires uv. Install once: curl -LsSf https://astral.sh/uv/install.sh | sh. Or use pipx run crawlemoon / pip install crawlemoon instead.

Where to put that JSON: Cursor → Settings → MCP. Claude Code → ~/.config/claude/mcp_settings.json. Windsurf → Settings → MCP Servers.

How it works

Your agent talks to Crawlemoon over the Model Context Protocol. Crawlemoon owns a hardened browser pool, an HTTP stack with TLS fingerprinting, and a rotating proxy pool. While it fetches pages, it captures network traffic, reads scripts, and introspects schemas — so the agent gets clean structured data, not raw HTML.

What's in the box

A short list — see the source for the full set of 55 tools.

Smart extraction — bring any LLM, including free ones

smart_extract works without any API key using pattern matching. Plug in any OpenAI-compatible endpoint for higher accuracy — including FREE tiers:

# OpenRouter (free models exist)
CRAWLEMOON_LLM_PROVIDER=openrouter
CRAWLEMOON_LLM_API_KEY=sk-or-v1-xxx
CRAWLEMOON_LLM_MODEL=meta-llama/llama-3.2-3b-instruct:free

# Groq (free, very fast)
CRAWLEMOON_LLM_PROVIDER=groq
CRAWLEMOON_LLM_API_KEY=gsk_xxx

# Local Ollama (no key needed)
CRAWLEMOON_LLM_PROVIDER=ollama
CRAWLEMOON_LLM_MODEL=llama3.2

Together, DeepSeek, Mistral, Fireworks, and standard OpenAI also work via CRAWLEMOON_LLM_BASE_URL.

Configuration

Proxy, V2ray & connection control

Crawlemoon uses one rotating proxy pool for browser contexts, stealth_request, and Xray/V2ray local exits. Credentials are stored separately from normalized URLs and are masked in stats/log-style responses.

Supported proxy entry formats:

http://user:pass@31.59.20.176:6754
socks5://user:pass@127.0.0.1:1080
31.59.20.176:6754:user:pass
31.59.20.176:6754

Start the MCP server with a proxy file:

{
  "mcpServers": {
    "crawlemoon": {
      "command": "uvx",
      "args": ["crawlemoon"],
      "env": {
        "CRAWLEMOON_PROXIES_FILE": "/secure/path/proxies.txt",
        "CRAWLEMOON_PROXY_SCHEME": "http",
        "CRAWLEMOON_PROXY_ROTATION": "sticky"
      }
    }
  }
}

Proxy files can use Webshare-style lines and comments:

# host:port:username:password
31.59.20.176:6754:uusmdewb:en3w097syrxh
31.56.127.193:7684:uusmdewb:en3w097syrxh

Configure or replace proxies at runtime with the configure_proxies MCP tool:

{
  "proxies_text": "31.59.20.176:6754:user:pass\n31.56.127.193:7684:user:pass",
  "default_scheme": "http",
  "rotation_strategy": "sticky",
  "health_check_interval": 300,
  "replace_existing": true
}

Use add_proxy, remove_proxy, test_proxy, and get_proxy_stats for incremental control. get_proxy_stats returns masked proxy URLs, for example http://user:***@31.59.20.176:6754.

For V2ray/Xray, call configure_xray_subscription with either subscription_url or raw_links. Crawlemoon starts local SOCKS5 exits such as socks5://127.0.0.1:10801, registers them in the same proxy pool, and can rotate or benchmark nodes with rotate_xray_node and test_xray_nodes.

Keep proxy files out of git. They contain live credentials.

Security

execute_js, execute_cdp, and deobfuscate_js are disabled by default — they execute or operate on arbitrary code in a real browser. Enable on trusted networks with CRAWLEMOON_ALLOW_DANGEROUS_JS=true. Even then, payloads are length-capped, time-bounded, and a denylist rejects eval, new Function, dynamic import(), document.write, importScripts, and WebAssembly.{compile,instantiate}. Set CRAWLEMOON_API_KEY so MCP clients must present a matching _api_key.

These are mitigations, not a sandbox: do not expose this server to untrusted clients.

Develop

git clone https://github.com/razavioo/crawlemoon.git
cd crawlemoon
make dev-install      # editable install + dev/captcha/ocr extras + pre-commit
make test             # pytest
make lint             # ruff + mypy

Releases

This project uses Trusted Publishing (OIDC) via GitHub Actions to automate publishing releases directly to PyPI.

To release a new version:

1. Bump the version number in pyproject.toml.
2. Commit the change and create a git tag matching the version (e.g. v1.1.8):

   git add pyproject.toml
   git commit -m "chore: bump version to 1.1.8"
   git tag v1.1.8
   

3. Push your branch and the tag to GitHub:

   git push origin main --tags
   

GitHub Actions will automatically run tests, build the package, and publish it securely to PyPI under the crawlemoon package space.

PRs welcome. Particularly interested in: distributed mode (Redis queue), result sinks (Postgres / S3), Prometheus metrics. See MIT License.

_{Made by emad.dev}