websearch-kit 0.35.0

Web search, fetch, and research pipeline for LLMs — usable as a Python SDK, a standalone MCP server, and an Open WebUI plugin.

websearch-kit

Web search, fetch, and research pipeline for LLMs — one engine, three surfaces: a Python SDK, a standalone MCP server, and an Open WebUI plugin.

Query expansion → multi-provider search → SSRF-guarded concurrent fetching (40-UA browser profile, pinned-IP connect) → trafilatura extraction → BM25 rerank with adaptive context budgeting → numbered, citable context for your LLM.

No fail-silent: a call either raises a typed error or returns a result where every dropped, blocked, truncated, or substituted item is enumerated as a structured Degradation. On the live web that looks like:

ok        : True   partial: True
sources   : 10                       # 5 fetched pages + 5 relevance-filtered snippets
warnings  :
  - [fetch] https://cloud.google.com/...: response exceeded byte cap (1054971 > 1048576 bytes)
stats     : 10 raw -> 10 unique, 5 fetched, context 23471 chars,
            timings {'search': 854, 'fetch': 1662, 'extract': 878, 'rank': 3}

Status: 0.31.0. See SPEC.md, CHANGELOG.md.

Features

• One engine, three surfaces — the SDK core is the only pipeline; the MCP server and Open WebUI adapters are thin translators over it, so behavior and config semantics never drift between surfaces.
• Full research pipeline — research() runs search → fetch → extract → rank → assemble in one call and returns a numbered [N] context block with 1:1 source citations, ready to drop into a prompt.
• Agentic deep research — deep_research() gathers via an agentic see-decide loop (one context steers search on the evidence so far, then MERGE → REDUCE), returning one synthesized, [N]-cited answer, so research quality stops depending on the calling model's own loop. (A planned PLAN→fan-out→REFLECT mode is a one-flag fallback.) Orchestration lives in code; an LLM (a Reasoner — an injected generate callback to the host's own model, or an OpenAI-compatible endpoint) is a stateless worker called once per scoped step. Inner research reuse keeps every guard; fully fail-soft. On all three surfaces: SDK (deep_research()), MCP (deep_research tool), and OWUI (the per-chat Search toggle pill, or the --deep flag).
• Deep research report — deep_report() is the deepest mode: a structural agent that OUTLINEs the question into sections, researches each (reusing the deep_research engine) against one shared global [N] registry, then assembles a multi-section, cited markdown report with a bounded completeness critique. For broad, report-grade questions where depth and structure matter, not a single answer. Slowest mode; same orchestrator requirement; bounded by deep_report_* config; fully fail-soft. On all three surfaces: SDK (deep_report()), MCP (deep_report tool), and OWUI (the per-chat Deep Research toggle pill).
• Preferred sources — for questions whose best source is authoritative but not lexically matchy (forecasts, prices, scores, gov stats, API docs), a preferred flag seeds those sources into the pool and protects them from the zero-BM25 drop + budget starvation. Fed by a config regex → domains registry and the deep-research orchestrator's own per-question proposals; discovery via site: search / direct URL, always through the SSRF guard. Default-on, fail-soft (ADR 0019).
• Performance profile — one performance_profile lever (economy/balanced/performance) scales the engine to your host. balanced is today's defaults; performance is a capacity tier for a fast host — it raises fetch concurrency + time budget so bigger jobs finish, holding the quality knobs at balanced (the quality benchmark found deeper context/rounds don't improve answers on a small orchestrator and can hurt format-sensitive tasks); economy lightens for a small/slow host. Any individual knob you set still wins over the preset (ADR 0021).
• Multi-provider search — zero-key ddgs out of the box; keyed Tavily / Brave / Serper / Exa and self-hosted SearXNG via config; ordered fallback chains with per-provider circuit breakers.
• Hardened fetching — SSRF guard (private / reserved / metadata IP ranges blocked at connect time with pinned-IP enforcement), per-response byte caps, rotating 40-UA browser profile, concurrent fetches with deadline budgeting.
• TLS-impersonation profile (optional) — fetch_profile="impersonate" (the [impersonate] extra) fetches via curl_cffi Chrome TLS impersonation to recover Cloudflare-fronted / Medium pages that 403 the default fetch on its TLS fingerprint (a full browser header set flips none of them — the block is on the ClientHello). The SSRF guard is fully preserved: the validated IP is pinned via libcurl CURLOPT_RESOLVE with SNI + certificate verification kept on the real hostname, every redirect hop re-vetted, and proxy_url rejected (it would void the pin). Off by default; robots-off like browser.
• Quality extraction & ranking — trafilatura article extraction, BM25 reranking (golden-tested math), adaptive context budgeting: the most relevant pages get more of the character budget, marginal ones shrink, noise is dropped.
• Hybrid dense reranking (optional) — semantic_rerank fuses BM25 with dense-embedding cosine via reciprocal-rank fusion, so synonym/paraphrase matches surface that pure lexical ranking misses (measured +0.021 nDCG@10 on the eval corpus). Local ONNX (fastembed, the [rerank] extra) or any OpenAI-compatible /embeddings endpoint; off by default, a zero-BM25 source still drops as noise, and an embedding failure degrades to pure BM25. GPU is optional (embedding_device="cuda", the [rerank-gpu] extra).
• Content deduplication — on by default: a word-shingle SimHash drops the same article reaching you under different URLs (syndicated wire stories, mirrors, scraped copies) that URL-identity dedup can't catch, while leaving distinct same-topic articles alone. dedup_max_hamming=0 disables it.
• Chunk-level assembly (default) — each page's budget is spent on its most relevant paragraphs (in document order, with visible [...] gap markers) instead of keeping the page head, so an answer past the budget line still ships. assembly_mode="pages" restores the pre-0.5 head-truncation.
• Char- or token-budgeting — the per-source budget is measured in characters by default; set budget_unit="tokens" to spend it in model tokens instead, so CJK/code-heavy pages ship the context they were actually allotted. Exact counts via tiktoken (the [tokens] extra), else a dependency-free CJK-aware heuristic. chars mode is byte-identical to before.
• Code & table fidelity — structured content survives the pipeline coherent on doc pages: fenced code keeps its markers and structural lines (the prose cleaner used to drop them), original indentation is restored from the source <pre> with a language info-string (```python), even when trafilatura glues text onto a fence delimiter; markdown tables are kept as atomic chunks with their header carried onto every piece of an oversized table, so rows never reach the model orphaned from their columns.
• Prompt-injection flagging — on by default (detect_injection): fetched content is scanned for injection-like text (instruction overrides, role/persona spoofing, prompt-extraction) and a match is surfaced as an info degradation. It flags, never censors — a best-effort signal on untrusted web data, not a safety boundary.
• Recency-aware ranking — pages carry their own declared publication dates (extracted from page metadata, explicit tags only — never guessed) merged with provider dates; a decay-weighted boost ranks fresher answers higher by default, works on any provider, and never penalizes undated pages. recency_boost=0 restores pure BM25.
• Date/time/location awareness — every LLM-facing surface knows now and here: prompts carry the current date/time (configured timezone) and an opt-in location hint; every research context opens with a Research performed: <ISO-8601> header; the MCP server surfaces the locale in its instructions/tool descriptions so calling models localize queries with zero extra calls.
• No-fail-silent contract — every degradation (blocked URL, truncated page, provider fallback, budget cut) is a typed, enumerable warning; nothing disappears without a trace.
• LLM query expansion (optional) — expand a question into multiple search queries via any OpenAI-compatible endpoint or an injected callback.
• Caching — in-memory by default, sqlite persistence a config flag away.
• OpenTelemetry metrics (optional) — metrics_enabled + the [otel] extra pushes per-stage latency (by provider), fetch outcomes, cache hits, degradations, and budget utilization to an OTLP collector, derived from the run stats the pipeline already produces. Off by default, MCP-server-only, fail-soft (a metrics fault never fails a tool call).
• Typed throughout — pyright-strict clean, structured results on every surface (Pydantic models in the SDK, JSON structured output over MCP).
• 1000+ tests including live-web smoke suites, hand-computed golden tests, and a recorded-corpus retrieval-quality eval gate (recall / nDCG / MRR / span-recall).

How to use

Python SDK

pip install "websearch-kit[ddgs]"   # ddgs = the zero-API-key search provider

import asyncio
from websearch_kit import SearchKit

async def main():
    async with SearchKit() as kit:          # zero-config: ddgs, no keys, no LLM
        report = await kit.research("RISC-V vs ARM datacenter adoption")
        print(report.context)               # numbered [N] block for your LLM
        for s in report.sources:
            print(f"[{s.n}] {s.title} — {s.url}")
        print(report.warnings)              # everything the run degraded on

asyncio.run(main())

Beyond research(), the kit exposes the pipeline stages individually:

results = await kit.search("python 3.14 free threading", count=5)     # snippets only
pages   = await kit.fetch(["https://docs.python.org/3.14/whatsnew/"])  # URLs, extracted
status  = await kit.health()                                           # provider probe

Prefer blocking code? SyncSearchKit mirrors the async API 1:1. Keyed providers, fallback chains, sqlite caching, and LLM query expansion are all config away — see docs/deployment/sdk.md and examples/.

MCP server

Add to your MCP client config (Claude Code, Claude Desktop, or any MCP client):

{
  "mcpServers": {
    "websearch-kit": {
      "command": "uvx",
      "args": ["--from", "websearch-kit[mcp,ddgs]", "websearch-kit-mcp"]
    }
  }
}

Requires Python ≥ 3.10. If your client launches uvx against an older host Python (e.g. LM Studio on macOS → system 3.9), prepend "--python", "3.12" to args — uv downloads a modern interpreter if needed.

Six read-only tools with typed structured output, over stdio or streamable HTTP:

Tool	What it does
web_search	Snippet-level results — context-economical
fetch_page	Read one URL as markdown, cursor pagination for long pages
research	Full pipeline → [N] context block + one resource link per citation
deep_research	Agentic multi-step research → one synthesized, [N]-cited answer (needs an orchestrator LLM)
deep_report	Deepest mode → a multi-section, [N]-cited markdown report for broad questions (needs an orchestrator LLM)
health	Provider latency, circuit-breaker state, config checks

For HTTP transport, scaling, and hardening flags see docs/deployment/mcp.md and examples/mcp_config_examples.md; a ready-to-use system prompt for these tools is in examples/system_prompt.md.

Open WebUI

Import adapters/owui/websearch_kit_filter.json via Admin Panel → Functions → Import (OWUI's import expects its JSON export format — or create a new Function and paste websearch_kit_filter.py instead). It pip-installs this SDK automatically via its frontmatter requirements: line and searches key-free out of the box (ddgs); valves switch it to your instance's configured web search or a keyed provider.

Toggle the pill to research every message, or trigger one-off:

?? quantum routers --count 12 --lang en --reply de --fresh week

A Tool variant for model-invoked (agentic) use ships alongside it. See docs/deployment/owui.md.

Documentation

• SPEC.md — normative behavioral contract (pipeline semantics, ranking math, degradation codes, SSRF ruleset)
• docs/architecture.md — how the engine is layered
• docs/domains/ — one standard document per domain
• docs/adr/ — the load-bearing architecture decisions
• docs/design/ — design notes and measurement investigations, including retrieval-gap-investigation.md (production retrieval re-baselined to ~93%) and thinking-evaluation.md (where native thinking helps — deterministic compounding — and why it's absent from this pipeline)
• VERSIONING.md — SemVer policy and public-API definition
• SECURITY.md — SSRF posture and threat model

License

MIT — with a CI-enforced permissive-only dependency policy (no GPL/AGPL; trafilatura>=1.8.0 pinned for its Apache-2.0 relicense).