websearch-kit 0.10.0

Web search, fetch, and research pipeline for LLMs — usable as a Python SDK, a standalone MCP server, and an Open WebUI plugin.

websearch-kit

Web search, fetch, and research pipeline for LLMs — one engine, three surfaces: a Python SDK, a standalone MCP server, and an Open WebUI plugin.

Query expansion → multi-provider search → SSRF-guarded concurrent fetching (40-UA browser profile, pinned-IP connect) → trafilatura extraction → BM25 rerank with adaptive context budgeting → numbered, citable context for your LLM.

No fail-silent: a call either raises a typed error or returns a result where every dropped, blocked, truncated, or substituted item is enumerated as a structured Degradation. On the live web that looks like:

ok        : True   partial: True
sources   : 10                       # 5 fetched pages + 5 relevance-filtered snippets
warnings  :
  - [fetch] https://cloud.google.com/...: response exceeded byte cap (1054971 > 1048576 bytes)
stats     : 10 raw -> 10 unique, 5 fetched, context 23471 chars,
            timings {'search': 854, 'fetch': 1662, 'extract': 878, 'rank': 3}

Status: 0.4.0. See SPEC.md, CHANGELOG.md.

Features

• One engine, three surfaces — the SDK core is the only pipeline; the MCP server and Open WebUI adapters are thin translators over it, so behavior and config semantics never drift between surfaces.
• Full research pipeline — research() runs search → fetch → extract → rank → assemble in one call and returns a numbered [N] context block with 1:1 source citations, ready to drop into a prompt.
• Multi-provider search — zero-key ddgs out of the box; keyed Tavily / Brave / Serper / Exa and self-hosted SearXNG via config; ordered fallback chains with per-provider circuit breakers.
• Hardened fetching — SSRF guard (private / reserved / metadata IP ranges blocked at connect time with pinned-IP enforcement), per-response byte caps, rotating 40-UA browser profile, concurrent fetches with deadline budgeting.
• Quality extraction & ranking — trafilatura article extraction, BM25 reranking (golden-tested math), adaptive context budgeting: the most relevant pages get more of the character budget, marginal ones shrink, noise is dropped.
• Hybrid dense reranking (optional) — semantic_rerank fuses BM25 with dense-embedding cosine via reciprocal-rank fusion, so synonym/paraphrase matches surface that pure lexical ranking misses (measured +0.021 nDCG@10 on the eval corpus). Local ONNX (fastembed, the [rerank] extra) or any OpenAI-compatible /embeddings endpoint; off by default, a zero-BM25 source still drops as noise, and an embedding failure degrades to pure BM25. GPU is optional (embedding_device="cuda", the [rerank-gpu] extra).
• Content deduplication — on by default: a word-shingle SimHash drops the same article reaching you under different URLs (syndicated wire stories, mirrors, scraped copies) that URL-identity dedup can't catch, while leaving distinct same-topic articles alone. dedup_max_hamming=0 disables it.
• Chunk-level assembly (default) — each page's budget is spent on its most relevant paragraphs (in document order, with visible [...] gap markers) instead of keeping the page head, so an answer past the budget line still ships. assembly_mode="pages" restores the pre-0.5 head-truncation.
• Char- or token-budgeting — the per-source budget is measured in characters by default; set budget_unit="tokens" to spend it in model tokens instead, so CJK/code-heavy pages ship the context they were actually allotted. Exact counts via tiktoken (the [tokens] extra), else a dependency-free CJK-aware heuristic. chars mode is byte-identical to before.
• Recency-aware ranking — pages carry their own declared publication dates (extracted from page metadata, explicit tags only — never guessed) merged with provider dates; a decay-weighted boost ranks fresher answers higher by default, works on any provider, and never penalizes undated pages. recency_boost=0 restores pure BM25.
• Date/time/location awareness — every LLM-facing surface knows now and here: prompts carry the current date/time (configured timezone) and an opt-in location hint; every research context opens with a Research performed: <ISO-8601> header; the MCP server surfaces the locale in its instructions/tool descriptions so calling models localize queries with zero extra calls.
• No-fail-silent contract — every degradation (blocked URL, truncated page, provider fallback, budget cut) is a typed, enumerable warning; nothing disappears without a trace.
• LLM query expansion (optional) — expand a question into multiple search queries via any OpenAI-compatible endpoint or an injected callback.
• Caching — in-memory by default, sqlite persistence a config flag away.
• Typed throughout — pyright-strict clean, structured results on every surface (Pydantic models in the SDK, JSON structured output over MCP).
• 890+ tests including live-web smoke suites, hand-computed golden tests, and a recorded-corpus retrieval-quality eval gate (recall / nDCG / MRR / span-recall).

How to use

Python SDK

pip install "websearch-kit[ddgs]"   # ddgs = the zero-API-key search provider

import asyncio
from websearch_kit import SearchKit

async def main():
    async with SearchKit() as kit:          # zero-config: ddgs, no keys, no LLM
        report = await kit.research("RISC-V vs ARM datacenter adoption")
        print(report.context)               # numbered [N] block for your LLM
        for s in report.sources:
            print(f"[{s.n}] {s.title} — {s.url}")
        print(report.warnings)              # everything the run degraded on

asyncio.run(main())

Beyond research(), the kit exposes the pipeline stages individually:

results = await kit.search("python 3.14 free threading", count=5)     # snippets only
pages   = await kit.fetch(["https://docs.python.org/3.14/whatsnew/"])  # URLs, extracted
status  = await kit.health()                                           # provider probe

Prefer blocking code? SyncSearchKit mirrors the async API 1:1. Keyed providers, fallback chains, sqlite caching, and LLM query expansion are all config away — see docs/deployment/sdk.md and examples/.

MCP server

Add to your MCP client config (Claude Code, Claude Desktop, or any MCP client):

{
  "mcpServers": {
    "websearch-kit": {
      "command": "uvx",
      "args": ["--from", "websearch-kit[mcp,ddgs]", "websearch-kit-mcp"]
    }
  }
}

Four read-only tools with typed structured output, over stdio or streamable HTTP:

Tool	What it does
web_search	Snippet-level results — context-economical
fetch_page	Read one URL as markdown, cursor pagination for long pages
research	Full pipeline → [N] context block + one resource link per citation
health	Provider latency, circuit-breaker state, config checks

For HTTP transport, scaling, and hardening flags see docs/deployment/mcp.md and examples/mcp_config_examples.md.

Open WebUI

Import adapters/owui/websearch_kit_filter.json via Admin Panel → Functions → Import (OWUI's import expects its JSON export format — or create a new Function and paste websearch_kit_filter.py instead). It pip-installs this SDK automatically via its frontmatter requirements: line and searches key-free out of the box (ddgs); valves switch it to your instance's configured web search or a keyed provider.

Toggle the pill to research every message, or trigger one-off:

?? quantum routers --count 12 --lang en --reply de --fresh week

A Tool variant for model-invoked (agentic) use ships alongside it. See docs/deployment/owui.md.

Documentation

• SPEC.md — normative behavioral contract (pipeline semantics, ranking math, degradation codes, SSRF ruleset)
• docs/architecture.md — how the engine is layered
• docs/domains/ — one standard document per domain
• docs/adr/ — the ten load-bearing architecture decisions
• VERSIONING.md — SemVer policy and public-API definition
• SECURITY.md — SSRF posture and threat model

License

MIT — with a CI-enforced permissive-only dependency policy (no GPL/AGPL; trafilatura>=1.8.0 pinned for its Apache-2.0 relicense).