screamingfrog-audit-mcp

Drive the Screaming Frog SEO Spider from Claude, Cursor, or any other MCP client. Crawl a site, get a ranked issue register back, ask questions of the crawl data, and render a shareable report — without opening the GUI or writing a single command.

It works on the free, unlicensed SEO Spider

That is the point of this server, and it is unusual: the other MCP servers for Screaming Frog build on its saved-crawl database, which is a licensed feature, so they need a paid install. This one drives the crawl directly and never touches that database.

The free tier caps you at 500 URLs per invocation — not per site. So full=true reads robots.txt and the sitemaps, splits the URLs into batches under the cap, runs each through list mode, and merges the exports. A 3,000-page site audits completely on a free install.

A licence removes the cap and unlocks config= for JavaScript rendering and custom extraction. Both tiers are supported and the server adapts to whichever you have.

You:  Crawl example.com and tell me what's actually broken.

→ start_crawl(url="https://example.com")
→ crawl_status()                     # 248 URLs, 51s
→ get_issues(priority="High")

Claude: Three high-priority problems. The big one: robots.txt disallows
/_next/, which hides 85 JS and CSS bundles from Google...

Install

Step 1 — the two things you need

Screaming Frog SEO Spider, installed on the same computer (download). The free version is fine. Python 3.10 or newer — macOS and most Linux systems already have it; on Windows get it from python.org and tick "Add python.exe to PATH" on the first screen.

Then install uv, which runs the server for you. Open a terminal and paste one line:

macOS / Linux — open Terminal:

curl -LsSf https://astral.sh/uv/install.sh | sh

Windows — open PowerShell (press Start, type powershell):

powershell -ExecutionPolicy ByPass -c "irm https://astral.sh/uv/install.ps1 | iex"

Close the terminal and open a new one so it picks up the change.

Step 2 — one command sets up your apps

uvx screamingfrog-audit-mcp --install

That finds Claude Desktop, Cursor and VS Code on your machine and configures each one. You never open a JSON file. It:

• keeps everything already in your config — other MCP servers and settings are untouched
• writes the full path to the program, because Claude Desktop and other desktop apps cannot find commands the way a terminal can
• saves a timestamped backup before changing anything
• re-reads the file afterwards to prove it is still valid, and restores your backup if it is not
• refuses to touch a config that is already broken, and tells you instead

Want to see what it would do first? Add --dry-run. Want one app only? --install claude-desktop, --install cursor or --install vscode.

Then fully quit and reopen the app — on macOS that is Cmd+Q, not just closing the window. Ask it: "check my screaming frog install".

Command-line assistants

These have their own one-liners, so there is nothing to edit:

claude mcp add screaming-frog -- uvx screamingfrog-audit-mcp
codex mcp add screaming-frog -- uvx screamingfrog-audit-mcp
hermes mcp add screaming-frog --command uvx --args screamingfrog-audit-mcp

If something is not working

uvx screamingfrog-audit-mcp --doctor

It checks Python, the MCP SDK, whether the Spider is found and actually runs, which options your Screaming Frog version supports, your licence tier, and whether the audit folder is writable — then prints the exact config for your machine.

  [PASS] Python: Python 3.12.7 on Darwin
  [PASS] MCP SDK: MCP SDK 2.1.1, using MCPServer (mcp 2.x)
  [FAIL] SEO Spider: Screaming Frog SEO Spider not found
    Install it from https://www.screamingfrog.co.uk/seo-spider/ ,
    or set SCREAMING_FROG_PATH to the executable.

An MCP server talks over a pipe, so when it fails to start the app just shows a dead server with no reason. The doctor is how you see the reason.

{
  "mcpServers": {
    "screaming-frog": {
      "command": "/Users/you/.local/bin/uvx",
      "args": ["screamingfrog-audit-mcp"]
    }
  }
}

ChatGPT, and what cannot work

The ChatGPT desktop app bundles Codex and reads the same ~/.codex/config.toml, so the codex mcp add line above configures it too.

ChatGPT on the web or on your phone cannot use this, and no config changes that. Those connectors call a remote web address, while this is a program that runs on your computer and drives the Screaming Frog installed there. It has to live where the Spider lives. The same is true of any assistant that only accepts a remote URL.

Optional settings

Add these to the env block of any config (a [mcp_servers.screaming-frog.env] table in Codex):

{
  "mcpServers": {
    "screaming-frog": {
      "command": "uvx",
      "args": ["screamingfrog-audit-mcp"],
      "env": {
        "SF_MCP_AUDIT_DIR": "/Users/you/audits",
        "SF_ALLOWED_DOMAINS": "example.com,acme.co.uk"
      }
    }
  }
}

The free-tier situation

The received wisdom is that the Screaming Frog CLI needs a licence. It does not. Verified against a build reporting Licence Status: Missing:

Because the cap is per invocation, start_crawl(full=true) discovers URLs from robots.txt and the sitemaps, batches them under the cap through list mode, and merges the exports back into one set. That crawls a site of any size on the free tier.

A licence removes the cap and makes full unnecessary. Everything else works the same.

Tools

Two design decisions worth knowing

Crawls are background jobs. A crawl takes minutes; an MCP call should answer in seconds. start_crawl forks a detached child and hands back a job_id. Nothing blocks unless you pass wait_seconds. The crawl survives the MCP server restarting.

Reads are capped, on purpose. A finished crawl folder is tens of megabytes of CSV. Feeding that to a model is both useless and expensive. Every read tool caps at 500 rows, lets you pick columns, and truncates long cells. Ask get_issues first — it's the whole site in about 60 lines — and aggregate_export when the question is "how many" or "broken down by", since counting rows by hand through a model is the expensive way to get a number. Reach for read_export only when you actually need the rows.

The deliverable

build_report writes four files into the crawl folder:

Every sheet has a frozen header, autofilter, banded rows, sized columns and a coloured tab. Cells are highlighted where the value is the finding — issue priority, 4xx/5xx status codes, non-indexable URLs, thin content, slow responses — so the problems are visible before you read a cell.

Reports carry a credit line naming the tool and its author.

One workbook instead of seventy CSVs

A finished crawl leaves around seventy exports in the folder, and the workbook already carries every one of them. build_report(consolidate=True) folds them in and deletes them, so the folder holds the workbook, the report files, and a consolidated.json manifest naming which sheet holds which table.

The delete is earned. Each sheet is written, the saved workbook is reopened from disk, and every sheet is checked against the row count it should hold. Only then is a file removed. If the save fails, the reopen fails, or a sheet comes back short, nothing is deleted and the reason is reported. An export too large to carry in full is kept on disk and named in the manifest, because a sampled sheet is not a substitute for the file.

It is off by default: read_export and aggregate_export read those CSVs. Turn it on when the folder is a finished deliverable rather than a crawl you are still asking questions about. Those tools then explain the consolidation and point at the workbook instead of reporting the folder as empty.

Where crawls are stored

~/.screamingfrog-audit-mcp/audits/<label>/ by default. Each folder holds the raw Screaming Frog CSV exports, audit-summary.json, and whatever build_report wrote — or, after a consolidated report, the workbook and manifest in place of those exports.

Override with SF_MCP_AUDIT_DIR:

{
  "mcpServers": {
    "screaming-frog": {
      "command": "uvx",
      "args": ["screamingfrog-audit-mcp"],
      "env": { "SF_MCP_AUDIT_DIR": "/Users/you/audits" }
    }
  }
}

Set SCREAMING_FROG_PATH if the Spider is installed somewhere non-standard.

Restricting what it may crawl

By default this server will crawl any host it is asked to. That is fine on your own machine, and a liability when an agent runs unattended: a confused or prompt-injected one can point a crawler at internal addresses or at third parties who did not ask to be crawled.

Set SF_ALLOWED_DOMAINS and start_crawl refuses anything else:

"env": { "SF_ALLOWED_DOMAINS": "example.com,acme.co.uk" }

Subdomains of a listed domain are allowed; look-alikes are not, so shop.example.com passes and example.com.evil.com does not. --doctor reports whether an allowlist is active.

Environment variables

All three are listed with examples under Optional settings in the install section.

Use it without MCP

The crawl pipeline is a plain module:

python -m screamingfrog_audit_mcp.runner --url https://example.com --output ./audit
python -m screamingfrog_audit_mcp.runner --url https://example.com --output ./audit --full

Gotchas found the hard way

• An unknown command-line flag aborts the whole crawl, it is not ignored: FATAL - SeoSpider failed to start ... UnrecognizedOptionException. The option set differs between versions — --skip-empty does not exist in 19.8, for instance — so this server reads the binary's own --help and passes only what your build advertises. Works on old and new versions alike.
• One wrong filter name aborts the whole crawl. Screaming Frog renames tab filters between versions, and an unrecognised name fails the run with a Java stack trace rather than skipping it. Every name is validated against your installed binary at crawl time, so an upgrade degrades instead of breaking.
• Its own --help output contains a poisoned entry. The binary lists a placeholder UNDEF:Unknown, and passing it back aborts the crawl with Using UNDEF as tab is not supported. It's filtered out.
• os.kill(pid, 0) is not a liveness probe on Windows. Any signal other than CTRL_C/CTRL_BREAK routes to TerminateProcess, so the usual "does this pid exist" idiom would kill the crawl and then report it finished. Windows uses tasklist to check and taskkill to cancel, and never signals. Detaching differs too: start_new_session is POSIX-only.
• everything mode is curated, not literal. The Spider lists ~1,150 tab filters, but 800+ are Custom Extraction / Custom Search / Custom JavaScript / AI filters that need a licence-gated config file and are permanently empty. Requesting them costs minutes and returns nothing, so those groups plus the API-dependent ones are excluded.

Development

git clone https://github.com/mshahiddigital/screamingfrog-audit-mcp
cd screamingfrog-audit-mcp
pip install -e ".[dev]"
pytest

The test suite runs on synthetic export fixtures, so it passes on a machine that has never had Screaming Frog installed.

Changelog

See CHANGELOG.md.

License

MIT. Not affiliated with or endorsed by Screaming Frog Ltd. You need your own copy of the SEO Spider; issue names, descriptions and fix guidance in the output are Screaming Frog's own.