oncrawl-mcp-server 0.3.0

MCP server that exposes OnCrawl's API for use with Claude Code and Claude Desktop

OnCrawl MCP Server

MCP server that exposes OnCrawl's API for use with Claude Code and Claude Desktop. Enables Claude to perform deep technical SEO analysis by querying crawl data, Google Search Console metrics, and crawl-over-crawl comparisons.

Features

• 12 MCP Tools for comprehensive SEO analysis
• Raw data access: Query pages, links, clusters, structured data with flexible OQL
• Schema discovery: Claude learns available fields before querying
• Aggregations: Group/count by any dimension for pattern detection
• Full exports: No 10k limit for complete datasets
• Crawl-over-crawl analysis: Track changes between crawls (new pages, status changes, etc.)
• Google Search Console integration: 600+ GSC fields including clicks, impressions, CTR, position by device, brand/non-brand, and more
• Google Analytics 4 integration: Session, user, and engagement metrics

What Makes This Powerful

OnCrawl combines crawl data with GSC/GA4 traffic data, enabling analyses like:

• Orphan pages with traffic: Pages getting clicks from Google but not linked internally
• Underlinked high-performers: Popular pages with weak internal linking
• Low CTR opportunities: Pages with high impressions but poor click-through rates
• 404s still in Google: Broken pages still appearing in search results
• Deep pages with traffic: Buried content that Google values
• Mobile vs desktop performance: Traffic breakdowns by device

Prerequisites

• Python 3.11+
• OnCrawl account with API access
• OnCrawl API token (from your OnCrawl settings)

Installation

Option 1: Install from PyPI (Recommended)

pip install oncrawl-mcp-server

Then configure in Claude Desktop/Code (see Configuration).

Option 2: Install from Source

# Clone the repository
git clone https://github.com/Amaculus/oncrawl-mcp-server.git
cd oncrawl-mcp-server

# Create virtual environment
python -m venv .venv
source .venv/bin/activate  # Mac/Linux
.venv\Scripts\activate     # Windows

# Install dependencies
pip install -e .

Getting Your API Token

1. Log into OnCrawl
2. Go to Settings → API
3. Create a new token with projects:read scope
4. Copy the token

Getting Your Workspace ID

Your workspace ID is required to list projects. To find it:

1. Log into OnCrawl
2. Look at the URL in your browser - it will be in this format:

   https://app.oncrawl.com/workspace/5c015889451c956baf7ab7a9/projects
                                   ^^^^^^^^^^^^^^^^^^^^^^^^^
                                   This is your workspace ID
   

3. Copy the 24-character ID from the URL

You can optionally add it to your config for convenience, or pass it directly when calling tools.

Configuration

For Claude Desktop

Windows: Edit %APPDATA%\Claude\claude_desktop_config.json Mac: Edit ~/Library/Application Support/Claude/claude_desktop_config.json

{
  "mcpServers": {
    "oncrawl": {
      "command": "python",
      "args": ["-m", "oncrawl_mcp_server.server"],
      "env": {
        "ONCRAWL_API_TOKEN": "your-api-token-here",
        "ONCRAWL_WORKSPACE_ID": "your-workspace-id-here"
      }
    }
  }
}

Note: ONCRAWL_WORKSPACE_ID is optional but recommended for convenience. If not set, you'll need to provide it when calling oncrawl_list_projects.

For Claude Code (CLI)

# Add to your MCP config
claude mcp add oncrawl

# Or manually edit your config with:
{
  "mcpServers": {
    "oncrawl": {
      "command": "python",
      "args": ["-m", "oncrawl_mcp_server.server"],
      "env": {
        "ONCRAWL_API_TOKEN": "your-api-token-here",
        "ONCRAWL_WORKSPACE_ID": "your-workspace-id-here"
      }
    }
  }
}

Note: ONCRAWL_WORKSPACE_ID is optional but recommended for convenience.

Important: Restart Claude Desktop/Code after changing the config.

Available Tools

Tool	Purpose
oncrawl_list_projects	List all projects in a workspace
oncrawl_get_project	Get project details including crawl IDs and COC IDs
oncrawl_get_schema	Call first - discover available fields for a crawl
oncrawl_search_pages	Query pages with OQL filtering, sorting, pagination (max 10k)
oncrawl_search_links	Query internal link graph (max 10k)
oncrawl_search_all_pages	Auto-paginating page search - bypasses 10k limit
oncrawl_search_all_links	Auto-paginating link search - bypasses 10k limit
oncrawl_aggregate	Group and count by any dimension with range support
oncrawl_export_pages	Full export without 10k limit
oncrawl_search_clusters	Find duplicate content clusters
oncrawl_search_structured_data	Audit schema markup
oncrawl_get_coc_schema	Discover fields for crawl-over-crawl comparison
oncrawl_search_coc	Find what changed between two crawls
oncrawl_aggregate_coc	Aggregate change patterns at scale

Handling Large Result Sets

The OnCrawl API limits search results to 10,000 per request. For larger datasets:

• oncrawl_search_all_pages / oncrawl_search_all_links: Automatically paginate through all results
• Use max_results parameter to cap the total (e.g., max_results: 50000)
• For very large exports (100k+), consider using filters to reduce the dataset

Usage Examples

Getting Started

"List my OnCrawl projects"
# If ONCRAWL_WORKSPACE_ID is not set in config, specify it:
"List my OnCrawl projects in workspace 5c015889451c956baf7ab7a9"

"Get the schema for crawl xyz789 - what fields are available?"

"Show me the first 10 pages from this crawl"

Technical SEO Analysis

"Find all pages at depth > 5 with fewer than 3 inlinks"

"Show me 404 pages that still have internal links pointing to them"

"What's the status code distribution for this crawl?"

"Find pages with missing meta descriptions"

GSC Integration

"Find orphan pages (0 internal links) that are getting clicks from Google"

"Show me pages with high impressions but low CTR (<2%)"

"Which pages buried deep in the site are getting significant traffic?"

"Compare mobile vs desktop traffic for the top landing pages"

Crawl-Over-Crawl Analysis

"Show me pages that changed status code between the last two crawls"

"Find pages that were added in the latest crawl"

"Which pages increased in depth between crawls?"

"Show me the status code distribution changes over time"

Detective Work

"I want you to act as a senior SEO analyst. Investigate crawl xyz789 for issues:
- Site structure problems
- Orphan page clusters
- Broken internal links
- Pages that should be linked better
- Anything else that looks problematic"

"Analyze this site for low-hanging SEO opportunities using both crawl and GSC data"

OQL Query Language

OnCrawl uses OQL (OnCrawl Query Language) for filtering. Here are the key operators:

Basic Operators

// Equals
{"field": ["status_code", "equals", 200]}

// Greater than / Less than
{"field": ["depth", "gt", "3"]}
{"field": ["follow_inlinks", "lt", "5"]}

// Contains
{"field": ["url", "contains", "/blog/"]}

// Starts with
{"field": ["urlpath", "startswith", "/products/"]}

// Has value / No value
{"field": ["canonical", "has_value", ""]}
{"field": ["canonical", "has_no_value", ""]}

Combining Filters

// AND
{
  "and": [
    {"field": ["status_code", "equals", 200]},
    {"field": ["depth", "gt", "3"]},
    {"field": ["follow_inlinks", "lt", "5"]}
  ]
}

// OR
{
  "or": [
    {"field": ["status_code", "equals", 301]},
    {"field": ["status_code", "equals", 404]}
  ]
}

// Nested combinations
{
  "and": [
    {"field": ["status_code", "equals", 200]},
    {
      "or": [
        {"field": ["depth", "gt", "5"]},
        {"field": ["follow_inlinks", "equals", 0]}
      ]
    }
  ]
}

Regex Support

{"field": ["urlpath", "startswith", "/blog/[0-9]{4}/", {"regex": true}]}

Common Field Names

Crawl Fields

• url, urlpath, depth, status_code
• follow_inlinks, follow_outlinks
• title, description, h1, canonical
• content_length, load_time
• indexability, is_compliant

GSC Fields (600+ available)

• gsc_clicks, gsc_impressions, gsc_ctr, gsc_position
• gsc_clicks_device_mobile, gsc_clicks_device_desktop
• gsc_clicks_brand, gsc_clicks_nonbrand
• gsc_impressions_device_mobile, etc.

Google Analytics Fields

• google_analytics_users_seo
• google_analytics_sessions_seo
• google_analytics_engaged_sessions_seo
• google_analytics_engagement_rate_seo

Pro tip: Always call oncrawl_get_schema first to see exactly which fields are available for your specific crawl.

GSC Integration

OnCrawl automatically integrates with Google Search Console when connected to your account. The GSC fields will appear in the schema if integration is active.

How it works:

• If GSC is connected: 600+ GSC fields are available in the schema
• If GSC is not connected: GSC fields won't appear in the schema
• Querying GSC fields without integration returns a 400 error
• Fields with no data return 0 or null (not an error)

Detection: Check the schema first with oncrawl_get_schema to see if GSC fields are present.

Troubleshooting

"ONCRAWL_API_TOKEN environment variable required"

• Make sure the token is set in the env block of your MCP config
• Restart Claude Desktop/Code after changing the config

"Unknown field" errors

• Call oncrawl_get_schema first to see available fields
• Field names are case-sensitive
• GSC fields only appear if GSC integration is active

API rate limits

• OnCrawl API has rate limits
• If you hit 429 errors, slow down requests
• Use exports for large datasets instead of pagination

Tool not appearing in Claude

• Verify Python path in config is correct
• Check that oncrawl-mcp-server is installed
• Look at Claude's logs for MCP connection errors
• Restart Claude after config changes

Permission errors

• Verify API token has projects:read scope
• Check workspace/project/crawl IDs are correct

Development

Running Tests

# Set your API token
export ONCRAWL_API_TOKEN="your-token"

# Run the server directly
python -m oncrawl_mcp_server.server

# Test with a specific project
python test_full_mcp.py

Building from Source

# Install build tools
pip install build twine

# Build the package
python -m build

# Install locally
pip install -e .

Version History

0.2.0 (Latest)

• Added 3 crawl-over-crawl (COC) tools
• Total of 12 MCP tools
• Full GSC integration documentation

0.1.0

• Initial release with 9 core tools
• Basic OnCrawl API integration

Contributing

Contributions welcome! Please feel free to submit a Pull Request.

License

MIT License - see LICENSE file for details.

Links

• GitHub Repository
• Report Issues
• OnCrawl Documentation
• Model Context Protocol

Author

Antonio - GitHub