crawlnest-mcp

MCP server for web scraping, Twitter, and Reddit — 17 tools for any MCP-compatible LLM client.

Works with Claude Code, Claude Desktop, Cursor, Windsurf, and any other MCP client.

Quick Start

pip install crawlnest-mcp

Then configure your client with two env vars:

That's it. No SSH, no scripts, no cloning repos.

Client Configuration

Claude Code

claude mcp add crawlnest \
  -e CRAWLNEST_API_URL=https://api.crawlnest.com \
  -e CRAWLNEST_API_KEY=cn_your_key \
  -- crawlnest-mcp

Or add to ~/.claude.json manually:

{
  "mcpServers": {
    "crawlnest": {
      "command": "crawlnest-mcp",
      "env": {
        "CRAWLNEST_API_URL": "https://api.crawlnest.com",
        "CRAWLNEST_API_KEY": "cn_your_key"
      }
    }
  }
}

Claude Desktop

Add to your config file:

{
  "mcpServers": {
    "crawlnest": {
      "command": "crawlnest-mcp",
      "env": {
        "CRAWLNEST_API_URL": "https://api.crawlnest.com",
        "CRAWLNEST_API_KEY": "cn_your_key"
      }
    }
  }
}

Restart Claude Desktop after saving.

Cursor

Add to ~/.cursor/mcp.json:

{
  "mcpServers": {
    "crawlnest": {
      "command": "crawlnest-mcp",
      "env": {
        "CRAWLNEST_API_URL": "https://api.crawlnest.com",
        "CRAWLNEST_API_KEY": "cn_your_key"
      }
    }
  }
}

Windsurf

Add to ~/.codeium/windsurf/mcp_config.json:

{
  "mcpServers": {
    "crawlnest": {
      "command": "crawlnest-mcp",
      "env": {
        "CRAWLNEST_API_URL": "https://api.crawlnest.com",
        "CRAWLNEST_API_KEY": "cn_your_key"
      }
    }
  }
}

Self-Hosted / Tailscale / Proxmox

Point CRAWLNEST_API_URL at your instance:

{
  "mcpServers": {
    "crawlnest": {
      "command": "crawlnest-mcp",
      "env": {
        "CRAWLNEST_API_URL": "http://100.64.0.5:8000",
        "CRAWLNEST_API_KEY": "cn_your_key"
      }
    }
  }
}

Replace 100.64.0.5 with your Tailscale IP (tailscale ip on the server).

Tools (17)

Web Scraping

Twitter

Reddit

Social Media Credentials

Twitter/Reddit tools that need authentication use server-side credentials — not passed through MCP. Set them once via the API.

Twitter (for search, trending, timelines)

Get auth_token and ct0 from x.com cookies (DevTools → Application → Cookies):

curl -X POST $CRAWLNEST_API_URL/api/twitter/credentials \
  -H "X-API-Key: cn_your_key" \
  -H "Content-Type: application/json" \
  -d '{"auth_token": "YOUR_AUTH_TOKEN", "ct0": "YOUR_CT0"}'

Reddit (for personalized feeds only)

Get reddit_session from reddit.com cookies:

curl -X POST $CRAWLNEST_API_URL/api/subreddit/credentials \
  -H "X-API-Key: cn_your_key" \
  -H "Content-Type: application/json" \
  -d '{"reddit_session": "YOUR_SESSION"}'

Usage Examples

> Scrape https://news.ycombinator.com and show the top stories

> Extract product name, price, and image from https://example.com/products as CSV

> Crawl https://anthropic.com/news with max 5 pages

> Get the latest tweet from https://x.com/anthropic/status/123456

> What's trending on Twitter right now?

> Show me the top posts from r/technology this week

> Search Reddit for "machine learning" in r/programming

Troubleshooting

"Command not found: crawlnest-mcp"

pip install --force-reinstall crawlnest-mcp
which crawlnest-mcp

"CrawlNest API not reachable"

curl -H "X-API-Key: cn_your_key" $CRAWLNEST_API_URL/health

"Twitter credentials missing"

Save credentials first (see Social Media Credentials).

Tools not showing

1. Restart your client
2. Check config JSON syntax: cat config.json | python3 -m json.tool
3. Check logs: tail -f /tmp/mcp_server.log

Self-Hosting

git clone https://github.com/WonderCrafts/CrawlNest
cd CrawlNest
make setup && make infra && make db-apply && make dev

Then use http://localhost:8000 as your CRAWLNEST_API_URL.

License

MIT

Links

• GitHub
• Issues
• Full MCP Setup Guide