ai-first-scraper-mcp
Plug Claude Desktop, Cursor, or Cline straight into an ad-free web scraper + search engine. Three tools, one line of config.
What it does
Adds three tools to any MCP-compatible agent:
| Tool | What it does |
|---|---|
fetch_page |
Fetch one URL → return clean Markdown (HTML or PDF). |
fetch_pages_batch |
Fetch up to 25 URLs in parallel → return Markdown for each. |
search_web |
Run a web search and return the top-k result pages already converted to Markdown. |
No more "the model called curl and then tried to parse 80kB of ad HTML." Your agent receives clean Markdown ready to reason about.
Backed by the ai-first-scraper and ai-first-search APIs.
Install
Fastest — uvx (no install, runs from PyPI on demand)
// claude_desktop_config.json / cline_mcp_settings.json / ~/.cursor/mcp.json
{
"mcpServers": {
"ai-first-scraper": {
"command": "uvx",
"args": ["ai-first-scraper-mcp"]
}
}
}
Restart your client (Claude Desktop / Cursor / Cline). The three tools above will appear automatically.
Alternative — pip install
pip install ai-first-scraper-mcp
{
"mcpServers": {
"ai-first-scraper": {
"command": "ai-first-scraper-mcp"
}
}
}
Where the config file lives
| Client | Config path |
|---|---|
| Claude Desktop (macOS) | ~/Library/Application Support/Claude/claude_desktop_config.json |
| Claude Desktop (Windows) | %APPDATA%\Claude\claude_desktop_config.json |
| Cursor | ~/.cursor/mcp.json |
| Cline (VS Code) | ~/Library/Application Support/Code/User/globalStorage/saoudrizwan.claude-dev/settings/cline_mcp_settings.json |
Point at your own backend (optional)
By default this server calls the public ai-first-scraper.onrender.com and
ai-first-search.onrender.com instances. If you want to self-host, set env
vars in your MCP config:
{
"mcpServers": {
"ai-first-scraper": {
"command": "uvx",
"args": ["ai-first-scraper-mcp"],
"env": {
"SCRAPER_URL": "https://your-scraper.example.com",
"SEARCH_URL": "https://your-search.example.com",
"AFS_TIMEOUT": "60"
}
}
}
}
Verify it works
Open your MCP client and ask the agent:
"Use the search_web tool to find the top 3 recent articles about MCP and summarize them in 5 bullets each."
You should see the agent call search_web, get back Markdown for each result,
and produce the summary without ever touching raw HTML.
Companion projects
- ai-first-scraper — the per-URL Markdown cleaner this MCP server fans out to.
- ai-first-search — search → scrape → markdown pipeline.
- mcp-rec — record & replay any MCP server's traffic for tests and bug reports.
- llm-cache-proxy — local cache for OpenAI/Anthropic API calls.
- promptlocker — lockfile for prompts.
- context-diff — see what blew up your Claude Code context window.
- agentwatch — overlay for browser AI agents.
Develop locally
git clone https://github.com/yubinkim444/ai-first-scraper-mcp.git
cd ai-first-scraper-mcp
uv sync # or: pip install -e .
ai-first-scraper-mcp # speaks MCP over stdio
To test against a local client, point its MCP config at the same command.
About the author
Built by yubinkim444, who also makes Kay's Records — an app for iOS and Android.
If this project saved you time, giving the app a try is the nicest way to say thanks.
License
MIT © yubinkim444
Metadata
Release files for ai-first-scraper-mcp 1.0.2
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | Provenance | Actions |
|---|---|---|---|---|
| ai_first_scraper_mcp-1.0.2.tar.gz | 5.7 kB | Without provenance | Details for ai_first_scraper_mcp-1.0.2.tar.gz |
Built distribution (wheel)
| File | Interpreter | ABI | Platform | Actions Reset |
|---|---|---|---|---|
| ai_first_scraper_mcp-1.0.2-py3-none-any.whl | Python 3 | none | any | Details for ai_first_scraper_mcp-1.0.2-py3-none-any.whl |
Total release size: 12.8 kB
Release files / ai_first_scraper_mcp-1.0.2.tar.gz
| Download URL | ai_first_scraper_mcp-1.0.2.tar.gz |
|---|---|
| Size | 5.7 kB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
87b1ea5a9b4d0ab98ab5378063a6ea93844fb3a1a515cb39aed106abcdeb4bfa
|
|
BLAKE2b-256 checksum How to use checksums |
bd4b92ccea1481850312fa8cd0e920ae288cee944bc3e6ab4acff7038470fee0
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/7.0.0 CPython/3.14.6
|
Release files / ai_first_scraper_mcp-1.0.2-py3-none-any.whl
| Download URL | ai_first_scraper_mcp-1.0.2-py3-none-any.whl |
|---|---|
| Size | 7.1 kB |
| Tags | Python 3 |
|
SHA-256 checksum How to use checksums |
2018c4c40290412ca2a05e40f050e6bc4609194e088f0a0cf3379e742e68b3f1
|
|
BLAKE2b-256 checksum How to use checksums |
392601e5f72bb34647c9f41bb619f5b47b3b028766d16045c3fcf841de589f81
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/7.0.0 CPython/3.14.6
|
Security for ai-first-scraper-mcp 1.0.2
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
No provenance found
None of the files in this release were uploaded with provenance metadata.
A note on package security
PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.
To learn more about protecting yourself from malicious actors, visit our package security documentation.