WHSearch

AI-native research search engine built as a lightweight modular monolith.

Current phase

v0.3.0 — all phases + hardening: keyless multi-provider discovery (DuckDuckGo lite/html + Wikipedia en/th routed + Google News + OpenAlex + arXiv + Crossref + YouTube + optional self-hosted SearXNG), robots-gated reader (HTML/PDF/plain/RSS, JS-shell L3 fallback), weighted RRF fusion, BM25 passage retrieval with section boosts, adaptive multi-round planning with evidence-gap follow-ups, Thai-aware claims/contradictions with PSL domains, claim-level verification, MCP agent (search, read_page, search_and_read, research), bounded TTL SQLite/FTS5 index, and explicit research budgets.

Design rules

• Evidence first: retrieve sources and passages before generating research conclusions.
• Online first: use external discovery initially; keep persistent storage bounded.
• Respect robots.txt, access policies, and conservative per-domain rate limits.
• Domain models and protocols must not depend on HTTP clients, providers, MCP, or extractors.
• MCP is an adapter layer; research/search logic stays in application modules.
• Optional integrations must not be required for importing the core domain.
• Resource limits are explicit so the system remains usable on low-memory machines.
• Keyless only: no paid API keys required for any default path.

Architecture

domain (models/protocols, no httpx/bs4/mcp)
  ^            ^            ^
search/reader/retrieval/research/evidence (pure app logic)
  ^            ^            ^
agent/service (plan → search → read → verify, budgeted)
  ^
application (composition root: providers, reader, index, agent)
  ^
mcp/server + tools (thin adapters) / infrastructure/http

Query DSL

site:, -exclude, "exact phrase", filetype:, OR (AND/NOT → spaces). site: becomes a domain allowlist; -/"phrase" filter locally so flaky HTML providers still work. recency_days boosts fresh dated results (undated kept) and the agent drops stale dated pages post-read.

Thai: th/en/mixed detection routes to one Wikipedia edition, sets kl/hl/gl/ceid + Accept-Language, and enables Thai-aware BM25/claims.

Planned phases

1. Foundation: contracts, configuration, logging, testing, architecture checks. Done.
2. Web discovery: provider abstraction and DuckDuckGo discovery. Done (+Wikipedia).
3. Web reader: robots policy, fetching, extraction, metadata, passages. Done.
4. Retrieval: passage ranking and deduplication. Done.
5. Research: adaptive query planning and stopping conditions. Done.
6. Evidence: claims, source independence, contradictions, verification. Done.
7. Search agent: orchestration through the MCP tools. Done.
8. Local index: SQLite/FTS5 bounded cache and reusable evidence. Done.
9. Autonomous research budgets and larger-scale discovery. Done (budgets + fan-out).
10. v0.3 hardening: weighted fusion, follow-up planning, Thai evidence, PDF/RSS, TTL caches, PSL domains, request IDs + JSON logs. Done.

Development

Use the repository virtual environment when available:

.venv/bin/python -m pytest
.venv/bin/python -m compileall -q src tests
.venv/bin/ruff check src tests

The quality gate must pass before moving to the next phase.

Install as an MCP server

Requires Python >= 3.12. After the whsearch package is published to PyPI, no manual install is needed — uvx fetches and runs it on first use:

{
  "mcpServers": {
    "whsearch": { "command": "uvx", "args": ["--from", "whsearch[mcp]", "whsearch"] }
  }
}

Alternatives:

uv tool install "whsearch[mcp]" && whsearch   # persistent install via uv
pipx install "whsearch[mcp]" && whsearch      # persistent install via pipx
pip install -e ".[mcp]" && whsearch           # from source

WHSEARCH_INDEX_PATH enables the persistent local index.

Configuration (env)

pip install -e ".[pdf]" enables PDF (pypdf).

Optional headless-browser fallback (L3)

Pages that render only via JavaScript (JS-shell SPAs) defeat static extraction. When the js extra is installed, the reader tries headless Chromium only for pages where static extraction yields almost nothing:

pip install -e ".[mcp,js]" && .venv/bin/playwright install chromium

Behavior and limits (all free, no keys):

• L1 trafilatura → L2 embedded JSON/meta → L3 headless, first hit wins.
• L3 triggers only below 200 extracted chars; rendered text must also clear it.
• At most 2 concurrent renders, 15s each, images/fonts/media blocked.
• Missing playwright (or any render failure) degrades silently to static text.
• WHSEARCH_BROWSER=0 disables it; WHSEARCH_BROWSER_TIMEOUT tunes seconds.
• WHSEARCH_CHROMIUM_PATH=/usr/bin/chromium reuses a system browser instead of downloading one (playwright install chromium).

Video vertical (YouTube, keyless)

Video-intent queries (youtube/video/คลิป, or site:youtube.com) also fan out to a first-party YouTube search provider (ytInitialData parsing — no key, no Invidious/Piped instances). Watch URLs read back as documents built from the video's own metadata + description, with MM:SS chapter lines as passage sections, so a multimodal caller knows where in the video to look:

• Direct captions are intentionally not fetched: YouTube's timedtext now requires proof-of-origin tokens and rate-limits keyless access (429).
• whsearch://stats reports reader.browser_installed/enabled/timeout.

License

GPL-3.0-or-later, see LICENSE. Copyright (C) 2026 WHSearch contributors. Per-file copyright holder names were intentionally left generic; update them to your name before publishing if you are the sole author.