Cluefinch MCP

Cluefinch is a local-first, version-aware research navigation layer for agents. It provides SearXNG search, HTML reading, and source collection without a separate LLM or paid search API. The agent plans, evaluates evidence, and writes conclusions; the server supplies research capabilities.

Repository | Issues | Agent workflow

Install

Requires Python 3.12.4+. After the first PyPI release, use an activated virtual environment:

pip install cluefinch
cluefinch

This starts an MCP stdio server, not an interactive shell or web interface. Connect it to an MCP client. Search requires a separate SearXNG instance with JSON output enabled. Its default address is http://127.0.0.1:8081; override it with MCP_SEARCH_SEARXNG_URL. The package does not install SearXNG. These commands describe installation after publication, not a claim that the package is already published. Neither uv nor Docker is required to install it.

Migrating from mcp-search

Do not install mcp-search and cluefinch in the same environment: both can own the mcp_search files. Prefer a new virtual environment. To reuse an existing one, run pip uninstall mcp-search before pip install cluefinch. The Python package remains mcp_search; python -m mcp_search and all MCP_SEARCH_ settings remain supported.

Install from source

From a checkout of this repository, with uv installed:

uv sync --locked
python scripts/init_env.py
docker compose up -d
uv run cluefinch

Docker is only needed for the supplied SearXNG deployment. Reuse an existing SearXNG instance if available. The sample binds to loopback port 8081. For a different port, set SEARXNG_PORT in .env and update the MCP client's MCP_SEARCH_SEARXNG_URL accordingly.

init_env.py creates a unique secret without overwriting an existing .env. Compose passes it as SEARXNG_SECRET. Never commit .env or your local searxng/settings.yml. SEARXNG_IMAGE accepts a tested tag or digest; the default searxng/searxng:latest is a development convenience, not a pinned release configuration. Upstream changes can affect search results.

Connect an MCP client

For clients using mcpServers, point to the interpreter in your environment:

{
  "mcpServers": {
    "cluefinch": {
      "command": "/absolute/path/.venv/bin/python",
      "args": ["-m", "mcp_search"],
      "env": {"MCP_SEARCH_SEARXNG_URL": "http://127.0.0.1:8081"}
    }
  }
}

On Windows, use a path such as C:\\path\\.venv\\Scripts\\python.exe. Client configuration formats vary. Alternatively configure the installed cluefinch executable with no arguments.

Tools

Tools publish typed input and output schemas, including nested sources and excerpts. Responses are validated before sending. Text JSON and MCP structuredContent have the same flat structure. Expected failures return error and hint instead of success fields.

topic alone does not start a search. Explicit URLs take priority; remaining candidate slots are shared round-robin between query variants. max_sources limits attempted candidates, not guaranteed successful sources. The server does not invent replacement queries or sources.

Reading and version checks

max_chars defaults to 10,000, capped at 20,000 in the default configuration. A preview reduces returned context, not the initial page download. Ready actions contain {tool, arguments} with the final URL, exact offset, reading budget, and expected_content_hash. Actions are optional; generating them makes no additional network request. Clients may qualify the local tool name with a server prefix.

Continuation preserves the effective reading budget. continuation: null means the retained text has ended. Expansion starts at the excerpt's start; its default budget is 3,000 characters, capped by MAX_FETCH_CHARS. You can adjust the budget while keeping the expected hash.

If the retained text differs, the server returns content_changed with no slice or automatic retry. Reacquire the source without the old hash and select coordinates again. A changed hash does not prove an old citation was false. A matching hash is not a freshness check against the origin. Cached text can still match after the website changes. Hashes do not identify cache hits or changes beyond the retained extraction.

Manual reading remains available through start_offset=response.next_start or start_offset=excerpt.start_char. Supply the corresponding source hash as expected_content_hash for server-side version checking. Without it, reading is unguarded. For preceding context, reduce the offset without going below zero. Offsets count Python Unicode characters, not bytes or UTF-16 code units.

truncated means more retained text remains after the slice. text_truncated means extraction exceeded the retained-text limit; pagination cannot recover that discarded tail. total_chars is the retained length.

Sources and limitations

source_id is src_ plus the first 16 hex characters of the normalized final URL's SHA-256. It is stable across calls and restarts, but does not identify a document version. content_hash hashes the entire retained extracted text.

Excerpts are literal slices: excerpt.text == text[start_char:end_char]. BM25 uses positive IDF and bounded passages from long paragraphs. When no positive matches exist, leading eligible passages are returned. There is no lemmatization or semantic relevance assessment. heading is local context, not a full outline. Author, publication date, and HTML canonical URL are not part of this contract.

source_type is a domain-based category, not a credibility rating. Unrecognized domains receive unknown. Domain search operators depend on external engine support. Search pagination is not implemented; max_results controls result count.

Inspect unresponsive_engines and collection gaps for provider failures, blocked/unreadable sources, duplicates, and limits. Empty gaps do not establish topic completeness. The agent decides whether evidence is sufficient.

After upgrading, restart the MCP connection so the client reloads schemas. Version 0.1.3 adds response fields; clients validating against an older strict schema must update it. Source installations should run uv sync --locked.

Configuration

Pass settings in the MCP process environment. Only Docker Compose automatically reads .env. Restart the server after changing settings. Numeric configuration values must be positive and finite.

MCP_SEARCH_UA sets the outgoing User-Agent for page downloads, including pages collected by research_collect. The default identifies Cluefinch and uses the installed package version. Override it for a deployment-specific identity and restart the MCP server. It does not change SearXNG's own requests to search engines.

Search requests are serialized with a five-second pause after completion, including failures. Page downloads use a separate five-second start-to-start interval per hostname and never overlap for that hostname. Up to three downloads to different hosts can run concurrently; redirects also pass through the limiter. Cache hits do not download the page or repeat the search. Limits are per MCP process, not shared across processes or subdomains. Waiting for a page-download slot counts toward DOWNLOAD_TIMEOUT.

These intervals reduce request frequency; they do not guarantee acceptance by a website. The server does not currently enforce robots.txt or Retry-After, and does not automatically retry failed downloads. Respect site and provider policies.

To add engines, update both SearXNG and MCP_SEARCH_ENGINES. When a call omits engines, search uses SearXNG's enabled engines. Caches are process-local with TTL and LRU limits. Character limits are not memory-size limits.

Support and security boundaries

Supports HTML/XHTML over HTTP(S). JavaScript rendering, PDF, authentication, CAPTCHA solving, and bypassing access restrictions are not implemented. Insufficient extracted text produces an error; this can indicate a short page as well as a JavaScript-rendered page. The transport requests gzip/identity; unsupported encodings and malformed or concatenated gzip fail explicitly. Decoding considers HTTP charset, BOM, and HTML meta-charset.

Outbound fetching rejects URL credentials, private/local/reserved and multicast IP addresses, scoped hosts, and mixed public/private DNS answers. Redirects are validated. Connections use validated numeric IPs while retaining the original hostname for Host, TLS SNI, and certificate verification. Environment proxies are disabled. Custom NAT64 prefixes or unusual routing require an infrastructure egress policy. Request limits apply to actual HTTP transitions, including redirects.

These controls do not filter prompt injection. Retrieved content is untrusted data, not instructions. The sample deployment is local, not a public hosted-service security profile. See the security policy.

Development

uv sync --locked
uv run pytest -q
uv run ruff check mcp_search tests scripts
uv run ruff format --check mcp_search tests scripts
uv build

Tests include real MCP stdio exchanges without external network access. uv run python scripts/smoke.py additionally checks live search and pages; it needs internet and SearXNG and accepts --url and --query overrides. CI checks Windows/Linux with Python 3.12/3.13.

See development checks and publishing.

License

Original code is licensed under Apache-2.0. NOTICE and third-party notices describe component boundaries. SearXNG is a separate AGPL-licensed service; the core license does not cover its code, dependencies, or retrieved content.

See contributing for Apache-2.0 and DCO requirements, and SearXNG integration requirements before bundling SearXNG or deploying a hosted service.