crawlnest-mcp
MCP server for web scraping, Twitter, and Reddit — 17 tools for any MCP-compatible LLM client.
Works with Claude Code, Claude Desktop, Cursor, Windsurf, and any other MCP client.
Quick Start
pip install crawlnest-mcp
Then configure your client with two env vars:
| Variable | Description |
|---|---|
CRAWLNEST_API_URL |
Your CrawlNest API endpoint (e.g., https://api.crawlnest.com or http://100.x.x.x:8000 for Tailscale) |
CRAWLNEST_API_KEY |
Your API key (starts with cn_...) |
That's it. No SSH, no scripts, no cloning repos.
Client Configuration
Claude Code
claude mcp add crawlnest \
-e CRAWLNEST_API_URL=https://api.crawlnest.com \
-e CRAWLNEST_API_KEY=cn_your_key \
-- crawlnest-mcp
Or add to ~/.claude.json manually:
{
"mcpServers": {
"crawlnest": {
"command": "crawlnest-mcp",
"env": {
"CRAWLNEST_API_URL": "https://api.crawlnest.com",
"CRAWLNEST_API_KEY": "cn_your_key"
}
}
}
}
Claude Desktop
Add to your config file:
| Platform | Config Path |
|---|---|
| macOS | ~/Library/Application Support/Claude/claude_desktop_config.json |
| Windows | %APPDATA%\Claude\claude_desktop_config.json |
| Linux | ~/.config/Claude/claude_desktop_config.json |
{
"mcpServers": {
"crawlnest": {
"command": "crawlnest-mcp",
"env": {
"CRAWLNEST_API_URL": "https://api.crawlnest.com",
"CRAWLNEST_API_KEY": "cn_your_key"
}
}
}
}
Restart Claude Desktop after saving.
Cursor
Add to ~/.cursor/mcp.json:
{
"mcpServers": {
"crawlnest": {
"command": "crawlnest-mcp",
"env": {
"CRAWLNEST_API_URL": "https://api.crawlnest.com",
"CRAWLNEST_API_KEY": "cn_your_key"
}
}
}
}
Windsurf
Add to ~/.codeium/windsurf/mcp_config.json:
{
"mcpServers": {
"crawlnest": {
"command": "crawlnest-mcp",
"env": {
"CRAWLNEST_API_URL": "https://api.crawlnest.com",
"CRAWLNEST_API_KEY": "cn_your_key"
}
}
}
}
Self-Hosted / Tailscale / Proxmox
Point CRAWLNEST_API_URL at your instance:
{
"mcpServers": {
"crawlnest": {
"command": "crawlnest-mcp",
"env": {
"CRAWLNEST_API_URL": "http://100.64.0.5:8000",
"CRAWLNEST_API_KEY": "cn_your_key"
}
}
}
}
Replace 100.64.0.5 with your Tailscale IP (tailscale ip on the server).
Tools (17)
Web Scraping
| Tool | Description | Key Parameters |
|---|---|---|
scrape_url |
Scrape a single page with anti-bot bypass | url |
crawl_website |
Crawl up to 50 pages with sitemap discovery | url, max_pages |
extract_structured |
Extract structured data with custom schema | url, schema, output_format |
| Tool | Description | Credentials |
|---|---|---|
fetch_tweet |
Fetch tweet by URL (3-layer cascade) | No |
twitter_user_info |
User profile info | No |
twitter_search |
Search tweets by query | Required |
twitter_trending |
Trending topics (local/global) | Required |
twitter_news |
News timeline | Required |
twitter_for_you |
Explore timeline | Required |
twitter_home_timeline |
Home feed (for_you/following) | Required |
| Tool | Description | Credentials |
|---|---|---|
reddit_post |
Fetch post + comment tree | No |
reddit_community |
Subreddit post listings | No |
reddit_about |
Subreddit metadata | No |
reddit_search |
Search across Reddit | No |
reddit_explore |
Discover subreddits | No |
reddit_popular |
Trending posts (with geo filter) | No |
reddit_feed |
Personalized home/news feed | Only with use_cookies=true |
Social Media Credentials
Twitter/Reddit tools that need authentication use server-side credentials — not passed through MCP. Set them once via the API.
Twitter (for search, trending, timelines)
Get auth_token and ct0 from x.com cookies (DevTools → Application → Cookies):
curl -X POST $CRAWLNEST_API_URL/api/twitter/credentials \
-H "X-API-Key: cn_your_key" \
-H "Content-Type: application/json" \
-d '{"auth_token": "YOUR_AUTH_TOKEN", "ct0": "YOUR_CT0"}'
Reddit (for personalized feeds only)
Get reddit_session from reddit.com cookies:
curl -X POST $CRAWLNEST_API_URL/api/subreddit/credentials \
-H "X-API-Key: cn_your_key" \
-H "Content-Type: application/json" \
-d '{"reddit_session": "YOUR_SESSION"}'
Usage Examples
> Scrape https://news.ycombinator.com and show the top stories
> Extract product name, price, and image from https://example.com/products as CSV
> Crawl https://anthropic.com/news with max 5 pages
> Get the latest tweet from https://x.com/anthropic/status/123456
> What's trending on Twitter right now?
> Show me the top posts from r/technology this week
> Search Reddit for "machine learning" in r/programming
Troubleshooting
"Command not found: crawlnest-mcp"
pip install --force-reinstall crawlnest-mcp
which crawlnest-mcp
"CrawlNest API not reachable"
curl -H "X-API-Key: cn_your_key" $CRAWLNEST_API_URL/health
"Twitter credentials missing"
Save credentials first (see Social Media Credentials).
Tools not showing
- Restart your client
- Check config JSON syntax:
cat config.json | python3 -m json.tool - Check logs:
tail -f /tmp/mcp_server.log
Self-Hosting
git clone https://github.com/WonderCrafts/CrawlNest
cd CrawlNest
make setup && make infra && make db-apply && make dev
Then use http://localhost:8000 as your CRAWLNEST_API_URL.
License
MIT
Links
Metadata
Release files for crawlnest-mcp 0.9.2
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | Provenance | Actions |
|---|---|---|---|---|
| crawlnest_mcp-0.9.2.tar.gz | 122.2 kB | Without provenance | Details for crawlnest_mcp-0.9.2.tar.gz |
Built distribution (wheel)
| File | Interpreter | ABI | Platform | Actions Reset |
|---|---|---|---|---|
| crawlnest_mcp-0.9.2-py3-none-any.whl | Python 3 | none | any | Details for crawlnest_mcp-0.9.2-py3-none-any.whl |
Total release size: 272.2 kB
Release files / crawlnest_mcp-0.9.2.tar.gz
| Download URL | crawlnest_mcp-0.9.2.tar.gz |
|---|---|
| Size | 122.2 kB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
1f6d3ce719edb47d52f8f64d35d187850df31005fe26555d66dafb8bb651a00b
|
|
BLAKE2b-256 checksum How to use checksums |
843149caab422c3b46de7aacbf443352dbff91b072e30260152899d8fa08a465
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/6.2.0 CPython/3.11.9
|
Release files / crawlnest_mcp-0.9.2-py3-none-any.whl
| Download URL | crawlnest_mcp-0.9.2-py3-none-any.whl |
|---|---|
| Size | 149.9 kB |
| Tags | Python 3 |
|
SHA-256 checksum How to use checksums |
4236bf2fec6522d7598f5650e1d9e75e50bf88ec903082a0f9b8e1ceb3d8d1a3
|
|
BLAKE2b-256 checksum How to use checksums |
2715549bf39c0ad013c36dbb8369507447dc9db4831b695d57b0b1225e51bae9
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/6.2.0 CPython/3.11.9
|
Security for crawlnest-mcp 0.9.2
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
No provenance found
None of the files in this release were uploaded with provenance metadata.
A note on package security
PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.
To learn more about protecting yourself from malicious actors, visit our package security documentation.