unlimited-search 1.0.0

MCP server for reading public web pages through resilient public routes.

unlimited-search

English | 한국어 | 中文 | 日本語 | Español

From public pages to usable signal.

unlimited-search is a Python CLI and MCP server for reading public web content when a normal direct fetch is not enough. It combines platform public routes, browser-like HTTP identities, content fallbacks, public archive fallbacks, and media metadata extraction behind one local tool.

It is built for agents and automation that need usable text from public URLs. It is not intended to bypass logins, paywalls, CAPTCHA, private networks, account restrictions, IP bans, or access controls.

Quickstart

Prerequisite: Python 3.12 or newer.

python -m pip install unlimited-search
unlimited-search read https://en.wikipedia.org/wiki/OpenAI --max-content-chars 800

The command returns JSON with page content, a verdict, request metadata, and an attempt trace.

For MCP clients, install the package and register the stdio server:

{
  "mcpServers": {
    "unlimited-search": {
      "command": "unlimited-search",
      "args": ["serve"]
    }
  }
}

What It Provides

Capability	What it does
Public platform routes	Uses unauthenticated public APIs, feeds, or metadata routes before generic fetching.
Browser-like fetching	Tries multiple TLS/browser identities, URL variants, referer strategies, and response validation.
Content fallbacks	Recovers public text through Jina Reader, RSS/Atom discovery, and embedded page metadata.
Archive fallbacks	Falls back to Wayback and archive.today/archive.ph public snapshots when live reads fail.
Media metadata	Uses yt-dlp --dump-json for public media URLs without downloading media.
MCP tools	Exposes single-URL reads, batch reads, diagnostics, and media extraction to MCP clients.

See Platform coverage for the full support matrix and known gaps.

Install

Install from PyPI:

python -m pip install unlimited-search

Update:

python -m pip install --upgrade unlimited-search

Remove:

python -m pip uninstall unlimited-search

CLI Usage

unlimited-search help
unlimited-search <command> [arguments]

Commands:

serve           Start the MCP stdio server
read <url>      Read a public URL
diagnose <url>  Diagnose access without full content
media <url>     Extract public media metadata
help            Show this help

Common examples:

unlimited-search read https://example.com --max-content-chars 1000
unlimited-search diagnose https://example.com
unlimited-search media https://www.youtube.com/watch?v=dQw4w9WgXcQ

Fallback-focused smoke tests:

unlimited-search read https://example.com --no-public-routes --max-attempts 0 --max-content-chars 500
unlimited-search read http://www.whitehouse.gov/1600/presidents/barackobama --no-public-routes --max-attempts 1 --max-content-chars 500

Useful flags:

Flag	Applies to	Purpose
--timeout SECONDS	read, diagnose, media	Set request timeout.
--max-attempts N	read, diagnose	Limit generic HTTP-grid attempts.
--max-content-chars N	read	Limit returned content size.
--no-public-routes	read, diagnose	Skip platform public routes.
--preferred-identity NAME	read, diagnose	Try a specific browser-like identity first.
--success-selector CSS	read	Treat matching CSS selectors as success signals. Can be repeated.

MCP Tools

The MCP server runs over stdio:

unlimited-search serve

Available tools:

Tool	Purpose
read_public_url	Read one public URL and return content, verdict, metadata, and trace.
read_public_urls	Read multiple public URLs in one tool call.
diagnose_access	Return a compact diagnosis and attempt trace without full content.
extract_media	Extract public media metadata through yt-dlp without downloading media.

More configuration examples are in MCP configuration.

How Reads Work

read_public_url tries the least invasive public routes first, then progressively broader recovery paths:

1. Platform-specific public routes for sites such as Reddit, X/Twitter, Bluesky, Hacker News, Google News, Stack Overflow, Wikipedia, GitHub, npm, PyPI, Wayback, Naver, Amazon, and Google Scholar.
2. A generic HTTP grid with browser-like TLS identities, URL variants, referer strategies, response validation, and selected HTTP/1.1 transport fallback.
3. Non-browser content fallbacks:
   • Jina Reader JSON content
   • RSS/Atom discovery through Jina external.alternate
   • common origin feed paths such as /feed, /rss, and /atom.xml
   • OGP, JSON-LD, Schema.org, and Next.js payload metadata salvage
4. Public archive fallbacks:
   • Wayback Available API
   • Wayback latest/direct snapshot
   • Wayback CDX latest 200 snapshot
   • archive.today/archive.ph best-effort snapshots
5. yt-dlp metadata extraction for known public media hosts.

Fallback successes are reported as weak_ok or suspect_ok, not strong_ok, because recovered content can be incomplete, stale, or metadata-only.

Safety Boundaries

unlimited-search is a public-content reader. It should stop or report failure when a target requires authentication, payment, CAPTCHA, private network access, or hard anti-abuse bypassing.

The reader rejects private, loopback, link-local, multicast, reserved, and metadata-service targets by default. Redirects are checked before fallback routes are allowed to continue.

Returned HTML, JSON, RSS, archive text, and metadata should be treated as untrusted content.

Development

uv sync --extra dev
uv run unlimited-search read https://example.com --max-content-chars 300
uv run unlimited-search serve
uv run pytest
uv build

Run the live eval set when changing routes or fallback behavior:

uv run python scripts/run_eval.py --list
uv run python scripts/run_eval.py
uv run python scripts/run_eval.py --markdown eval-results/report.md --csv eval-results/report.csv
uv run python scripts/run_eval.py --baseline eval-results/eval-20260707T000000Z.jsonl --fail-on-regression

The default eval cases live in scripts/eval_urls.yaml. Difficult sites such as NamuWiki, TikTok, Naver Search, Amazon, and Google Scholar are optional so remote blocking or rate limits do not fail the whole run.

Project Docs

• Platform coverage
• MCP configuration
• Troubleshooting
• Security
• Contributing
• Code of conduct
• Disclaimer
• Privacy
• Changelog
• License