websearch-skill 0.2.6

Open-source multi-engine web search + extraction SKILL and TOOL for AI agents (uv-native, contract-driven, harness-portable)

websearch-skill

Open-source multi-engine web search and content extraction for AI agents, built as isolated layers connected only by versioned JSON Schema contracts.

Status: early. First public release 2026-06-22; current version in CHANGELOG.md. The keyless search, the clean-Markdown reader, the five agent tools, and the opt-in egress proxy work today and are covered by the test suite. The hard anti-bot tiers and local rerank are not built yet (see Roadmap). Pin a version and try it in a sandbox before wiring it into anything sensitive.

What it is

A self-hosted web search and page reader for AI agents, with no API keys and no paywall. Point it at a query and it fans out across many search engines, fuses and dedups the results, then fetches and extracts pages into clean Markdown. Everything runs locally; queries do not go to a vendor.

Three commands are the whole surface:

• web_search finds pages: ranked, deduplicated results across engines, each with a reusable handle.
• web_fetch reads a page: clean Markdown, fenced as untrusted, paginated so a long page never overflows context.
• web_open pages back through a document you already fetched, from cache, without hitting the network again.

Two extra keyless tools cover what general web search does not:

• arxiv searches arXiv papers and returns structured metadata (authors, abstract, categories, abstract and PDF links).
• github searches GitHub repositories and returns typed fields you can sort on (stars, language, topics).

Engines, out of the box, no keys

Search works the moment you install it. The default engine is the keyless ddgs metasearch library, which by itself spans Google, Brave, DuckDuckGo, Yandex, Yahoo, Startpage, Mojeek, Wikipedia, and Grokipedia. Each query is served by whichever of those respond. No API key, no account, no service to run, and no engine flags: the agent surface (web-search) is plug-and-play. Picking a subset is a power-user knob on the lower-level search command via --ddgs-backends google,brave,mojeek, not on web-search. Which of them actually answer from your connection today is a question websearch doctor answers per provider, and it varies.

For broader and more reliable search you can run your own SearXNG (~280 engines, your own server, still no keys), and the router fuses it with ddgs and de-correlates the engines they share. websearch searxng up sets one up and starts it on any machine with git and Python; on a Docker host, docker/searxng/ is the curated stack, including a probe that enables every engine that answers on your connection. See Self-hosting SearXNG below. Public SearXNG instances are deliberately not a default: most disable the JSON API and rate-limit automated clients, so depending on them would break on a fresh install.

It is MIT-licensed and open source. On top of the keyless default you can add a self-hosted SearXNG, keyed engines (Brave, Exa, Tavily), or a paid egress adapter, all behind the same contracts, none of them required.

Among 2026 agentic-search APIs the top tier is statistically tied on result quality, so cost and latency are the real differentiators; the scorecard below is how I measure that, axis by axis. Where this helps is cost (about zero at the software layer), privacy and self-hosting, multi-engine recall plus dedup, clean extraction, and the unprotected majority of the web. Hard anti-bot at scale is out of scope: it stays a swappable paid egress adapter, because residential proxies and captcha solving have no reliable free equivalent.

Layer status

Layer	What it does	Status
Layer 1: Search	Multi-engine router (keyless ddgs across many engines, optional self-hosted SearXNG), canonicalize, dedup, de-correlated RRF fusion	Built
Layer 2A: Fetch + Extract	Tiered fetch (httpx, curl_cffi impersonation), Trafilatura extraction to Markdown + metadata	Built
Layer 2B: Format + Store	Paginated Markdown + JSON sidecar, progressive-disclosure index/resolver, MinHash dedup, ephemeral SQLite-FTS5 store	Built
Layer 3: Agent I/O	Consolidated web_search/web_fetch/web_open, untrusted-content fence, optional MCP stdio server, SKILL.md	Built
Extra tools	Keyless arxiv (paper search) and github (repo search), standalone over the same Envelope	Built
Doctor	websearch doctor: per-capability self-test of the optional layers, every engine, the tools, both fetch tiers, and MCP	Built

Contracts are frozen as JSON Schema 2020-12: envelope@1.0.0, search@1.1.0, fetch@1.2.0, extract@1.0.0, format@1.0.0, store@1.0.0, agent-io@1.1.0, arxiv@1.1.0, github@1.1.0, doctor@1.0.0. Every response is wrapped in one Envelope { contract_version, ok, data, error, meta }.

Layer 1: Search

A thin router fans a normalized request out to per-engine adapters behind an EngineAdapter port, canonicalizes URLs, dedups with provenance merge, and fuses results with provenance-aware weighted Reciprocal Rank Fusion (RRF, k=60). The keyless default is ddgs, a metasearch library that is itself multi-engine (Google, Brave, DuckDuckGo, Yandex, Yahoo, Startpage, Mojeek, Wikipedia, Grokipedia). On this lower-level search command, --ddgs-backends forces a subset and --engines picks which adapters (ddgs, searxng) run; the agent-facing web-search command takes none of those and just uses the keyless default. Point WEBSEARCH_SEARXNG_URL at a self-hosted SearXNG to add it as a second, broader engine, and the router fuses both. Keyed engines (Brave, Exa, Tavily, and others) are a planned drop-in behind the same port.

ddgs and SearXNG do the same job (query many engines and merge) in different forms: ddgs is a keyless Python library that runs in-process, while SearXNG is a separate server you host that covers far more engines. They are interchangeable adapters behind this port; run either, or both fused. SearXNG only overlaps with this search layer, it does not fetch or extract pages, so it never replaces the rest of the tool.

The load-bearing decision is de-correlation. SearXNG and ddgs both lean on the same upstream crawlers (Google, Bing), so a naive union lets a consensus bonus amplify the same crawler agreeing with itself, and the fused ranking can end up worse than a single well-tuned engine. The fix: a result's sources are grouped by correlation_group, each group contributes one RRF term at its best rank, and the consensus bonus scales only with the number of distinct groups. SearXNG and ddgs agreeing counts as one independent vote; SearXNG and a neural index agreeing counts as two. The router records the de-correlation in a warning.

Every result carries full per-engine provenance (which engine returned it, at what rank), so the ranking is auditable.

Layer 2A: Fetch + Extract

Two decoupled sub-ports, each independently swappable.

Fetch escalates by tier and escalates only when it detects an anti-bot block. Tier 0 is plain httpx; on a detected challenge it escalates to curl_cffi (browser TLS/JA3 impersonation). It does not escalate on a 404 or on a terminal block (rate limit, auth required, legal/geo block), since a stealthier client from the same egress will not help. Block detection reads header markers first, then gated body markers for Cloudflare, DataDome, PerimeterX, Akamai, and Imperva (Imperva can return a block with HTTP 200, so its body markers are scanned on any status). Browser and stealth tiers (Crawl4AI, nodriver) are named in the contract enum but stay opt-in, not in the base install.

Extract defaults to Trafilatura, a heuristic extractor. Per the May 2026 WCXB benchmark, heuristic extractors beat neural extractors on both quality and cost (neural runs tens to hundreds of times more expensive), and Trafilatura lands at about 0.79 F1 on CPU in roughly 100ms; article pages saturate around 0.93 F1. The adapter parses the raw HTML once with lxml to recover the raw schema.org JSON-LD blocks and og:type (Trafilatura folds JSON-LD into metadata and never exposes the raw blocks), runs Trafilatura for the Markdown body and plain text plus metadata, then computes:

• a heuristic quality_score (0..1) from runtime signals (text density, word-count saturation, paragraph count, inverse link density, JSON-LD presence, clean title) with hard vetoes for soft-404s and shells; below about 0.80 a page is a fallback candidate.
• a cheap page_type resolved from JSON-LD @type, then og:type, then URL shape.

Neural extract engines (crawl4ai, jina_readerlm, and others) are named in the contract enum but stay opt-in. The default dependency closure is permissive: Apache-2.0 (trafilatura) plus MIT/BSD/MPL deps.

There is no output-length cap anywhere. content_markdown is never truncated; --max-bytes is a transport guard only, not a content cap, and --max-bytes 0 lifts even that. The same convention runs through every knob: --max-results 0, --page-size-tokens 0, and --per-page 0 mean "no limit" (or the provider's own maximum where one exists, like GitHub's 100 per page and arXiv's 2000 per request).

Layer 2B: Format + Store

Two decoupled sub-ports that turn results into something an agent can actually read, and keep the full pages around for follow-up.

Format takes vendor-neutral results (the union of what Layer 1 and Layer 2A produce) and renders one layout-stable Markdown document plus a parallel JSON sidecar holding the same data. Results are ordered by descending relevance and paginated, because lost-in-the-middle is still real in 2026 even on long-context models. Near-duplicate dedup runs first: byte-exact (normalized SHA-256), then a pure-Python MinHash over word 4-gram shingles (128 permutations, Jaccard 0.9) clustered with union-find. The best-scored page in a cluster is kept and the rest are recorded as dropped_duplicates, so the fold is auditable. The 0.9 threshold is deliberately conservative: it folds genuinely near-identical pages (mirrors, syndication), not merely topically similar ones, so two pages that just read alike will not collapse at the default. Lower jaccard_threshold for boilerplate-heavy corpora. Dedup is pure Python by design (no datasketch), so the default install stays dependency-light. Progressive disclosure chooses how much to inline:

• auto (default) inlines full bodies when the page fits a token budget, otherwise switches to an index (a preview plus a stable id).
• index always shows a preview and a resolve hint; full always inlines.

The optional anthropic_search_result_blocks view maps 1:1 onto Anthropic search_result content blocks (source as a bare string, at least one non-empty text block, citations all-or-nothing). It is off by default and is a derived view, not the canonical shape, so installing into a non-Anthropic harness costs nothing.

There is no output-length cap here either. The sidecar carries the full body verbatim in every mode, and the store keeps the full Markdown. body_char_budget only offloads the rendered Markdown view to the resolver (with a hint), and --no-truncate turns even that off. Nothing is summarized or discarded.

Store is an ephemeral page index behind a PageIndex port (add / search / get / resolve_index). There is no database for the per-query result set, which is tens of rows of plain Python; the store is used only as the progressive-disclosure index over fetched pages. The default adapter is SQLite FTS5 over an in-memory connection: it ships in the Python stdlib with BM25 and needs no third-party package. FTS5 is not compiled into every SQLite build, so the adapter probes for it at runtime and falls back to a pure-Python BM25 index that returns the identical shapes. Adds are idempotent on url plus content hash, an arbitrary query is escaped so FTS5 operators never raise a syntax error, and persistence is just passing a file path. A vector or Rust backend (sqlite-vec, tantivy) plugs in behind the same port, opt-in.

Layer 3: Agent I/O

One consolidated surface over Layers 1, 2A, and 2B: web_search (find), web_fetch (read a URL), and web_open (page through an already-fetched document). Each returns the same Envelope. The identical core is exposed three ways: the websearch web-search / web-fetch / web-open CLI, a FastMCP stdio server (websearch mcp, bundled in the base install), and a portable SKILL.md written to the Agent Skills standard (name plus description, so it loads in Claude Code, Codex, OpenCode, and others).

The cross-layer key is a human-readable handle (site~shorthash, for example en.wikipedia.org~3a1f9c2b5e6f), never an opaque UUID. web_fetch indexes the full page into the Layer 2B store and returns one token-budget page; web_open pages through the rest from that store, by handle, without re-fetching. The split is lossless: pagination is progressive disclosure, not a cap, and the whole body stays reachable page by page. The lower-level search / fetch / open commands remain as the per-layer surfaces for debugging and composition.

Fetched page text is untrusted, so web_fetch and web_open wrap each page in a fence (see Security). On the MCP face the page also rides the tool-result channel, which models are trained to treat with skepticism.

Extra keyless tools

Two standalone tools cover sources that general web search handles poorly, both keyless and over the same Envelope:

• arxiv searches arXiv via the official Atom API and returns structured papers (title, authors, abstract, categories, abstract and PDF links). It supports field-targeted search (--field title|author|abstract) and sorting by date or relevance, uses GET so it benefits from arXiv's cache, and backs off on the 2026 rate limiting.
• github searches GitHub repositories via the unauthenticated REST API and returns typed fields (full name, stars, forks, language, topics, updated date). Unauthenticated search is about 10 requests per minute; on a rate limit it returns a clean rate_limited error instead of hammering. Code search needs a token and is intentionally left out of the keyless path.

uv run websearch arxiv "diffusion models for protein design" --max-results 5 --sort-by submittedDate
uv run websearch github "vector database" --language Rust --sort stars --per-page 10

Reddit and X (Twitter) have no keyless, terms-clean search path in 2026 (Reddit's anonymous JSON endpoints return 403 as of May 2026; X needs a paid API or a logged-in account), so there is deliberately no dedicated tool for them. Search the open web with a site filter instead:

uv run websearch web-search "rust async" --site reddit.com
uv run websearch web-search "frontier model release" --site x.com

Install

Pick the route that matches how you use it. Everything is keyless and needs internet; the only hard requirement is uv. Full per-harness instructions (Claude Code, Codex, OpenCode, Cursor, Hermes, OpenClaw, the MCP registry, and PyPI publishing) are in docs/INSTALL.md.

As a CLI tool, no install (uvx builds an ephemeral env and runs it):

uvx websearch-skill web-search "your query"
# or straight from git, for an unreleased commit:
uvx --from git+https://github.com/hec-ovi/websearch-skill websearch web-search "your query"

As an agent skill across 40+ agents via the skills CLI (it installs the skills/web-search/ directory into each detected agent):

npx skills add hec-ovi/websearch-skill            # all detected agents
npx skills add hec-ovi/websearch-skill -a claude-code -a codex -s web-search

As a Claude Code plugin (bundles the skill and the MCP server in one install):

/plugin marketplace add hec-ovi/websearch-skill
/plugin install web-search@websearch-skill

As an MCP server (FastMCP stdio, bundled in the base install). Point any MCP client at:

{ "mcpServers": { "web-search": { "command": "uvx", "args": ["websearch-skill", "mcp"] } } }

It is also published in the MCP Registry as io.github.hec-ovi/web-search, so registry-aware clients can discover and install it by name.

From source (development, uv-native):

git clone https://github.com/hec-ovi/websearch-skill
cd websearch-skill
uv sync
uv run websearch web-search "your query"

Quickstart

Search works with no setup: the keyless ddgs metasearch is the default. The agent-facing web-search (Layer 3) needs no engine flags. The lower-level search (Layer 1) is for debugging and power use, and is the only command that takes --engines, --ddgs-backends, and --no-ddgs:

# Layer 1: search (keyless, multi-engine via ddgs)
uv run websearch search "rust ownership" --json

# force specific keyless engines (search command only, not web-search)
uv run websearch search "rust ownership" --ddgs-backends google,brave,mojeek

# add a self-hosted SearXNG (see docker/searxng/) as a second, broader engine
export WEBSEARCH_SEARXNG_URL=http://127.0.0.1:8888
uv run websearch search "rust ownership" --engines searxng,ddgs

# Layer 2A: fetch + extract one page to clean Markdown
uv run websearch fetch "https://en.wikipedia.org/wiki/Rust_(programming_language)" --json

# Layer 2B: open several pages into one paginated, deduped, LLM-ready document,
# and full-text search the passages across them
uv run websearch open \
  "https://en.wikipedia.org/wiki/Rust_(programming_language)" \
  "https://doc.rust-lang.org/book/ch04-00-understanding-ownership.html" \
  --search "ownership borrow checker"

# Layer 3: the consolidated, fenced, handle-keyed agent face
uv run websearch web-search "rust ownership" --json
uv run websearch web-fetch "https://doc.rust-lang.org/book/ch04-01-what-is-ownership.html" \
  --page-size-tokens 4000 --persist-path /tmp/idx.sqlite
# page through the rest of a fetched doc by its handle, from cache (no refetch)
uv run websearch web-open "doc.rust-lang.org~<hash>" --page 2 --persist-path /tmp/idx.sqlite

# extra keyless tools: arXiv papers and GitHub repos
uv run websearch arxiv "mixture of experts scaling laws" --max-results 5
uv run websearch github "llm agent framework" --language Python --sort stars

# or run as an MCP server (FastMCP stdio; bundled, no extra needed)
uv run websearch mcp

# check what actually works on this machine, engine by engine
uv run websearch doctor

Every command prints a compact human view by default, or the raw JSON Envelope with --json (exit 0 on success, 1 on a request-level error). For the fetch command, --output-format {markdown,text,json} selects the body representation the human view prints (text emits the plain-text rendering), and --quiet prints only the extracted body, for piping. For the open command, --mode {auto,index,full} controls progressive disclosure, --no-truncate inlines every full body, --search QUERY runs a BM25 passage search over the opened pages, and --anthropic-blocks adds the Anthropic search_result view to the sidecar. The agent-facing web-search takes --max-results, --detail, --freshness, --site, --language, --country, --safesearch, --offset, and --searxng-url; the engine-selection flags (--engines, --ddgs-backends, --no-ddgs) live only on the lower-level search command, alongside --searxng-url (or WEBSEARCH_SEARXNG_URL). See uv run websearch <command> --help for the full flag list.

A fetch --json response looks like:

{
  "contract_version": "1.0.0",
  "ok": true,
  "data": {
    "source": { "final_url": "https://...", "status": 200, "fetched_via": "http", "blocked": false },
    "result": {
      "title": "Rust (programming language)",
      "page_type": "article",
      "quality_score": 0.91,
      "word_count": 8123,
      "content_markdown": "# Rust ...",
      "metadata": { "og_type": "article" }
    }
  },
  "error": null,
  "meta": { "layer": "extract", "backend": "http", "elapsed_ms": 412 }
}

As a library:

from websearch.layer1_search import build_router, SearchRequest

router = build_router(searxng_url="http://127.0.0.1:8888", enable_ddgs=True)
envelope = router.search(SearchRequest(query="rust ownership", count=10))
for r in envelope.data["results"]:
    print(r["fused_score"], r["url"], [s["engine"] for s in r["sources"]])

Layer 2B (format + store) the same way. run returns an Envelope; its data is the JSON FormatPayload, so markdown and the lossless sidecar are plain dict access:

from websearch.layer2_format import (
    FormatRequest, ResultInput, build_format_pipeline,
    build_page_index, StoreConfig, PageInput, SearchPageRequest,
)

env = build_format_pipeline().run(
    FormatRequest(
        query="rust ownership",
        results=[ResultInput(url="https://x.test/a", title="A", score=0.9, body_markdown="# A ...")],
    )
)
document = env.data["markdown"]       # the layout-stable document
sidecar = env.data["sidecar"]         # identical data; full bodies verbatim, never capped

# an ephemeral page index for passage search and resolve-by-id
index = build_page_index(StoreConfig())          # in-memory SQLite FTS5 (BM25)
index.add([PageInput(url="https://x.test/a", markdown="# A ...")])
hits = index.search(SearchPageRequest(query="borrow checker"))

Optional layers (all off by default)

The base install searches the keyless engines over a direct connection. Three switches add to that, each off until you set it, each verified by websearch doctor:

Layer	Switch	What it adds
VPN	WEBSEARCH_VPN=nordvpn or any	declares that egress should be tunneled, so the doctor verifies it instead of assuming it. It routes nothing on its own: the tunnel is your VPN app's job
Egress proxy	WEBSEARCH_PROXY=<url> or nordvpn	one proxy for every network path the tool opens
SearXNG	WEBSEARCH_SEARXNG_URL=<url>	a self-hosted metasearch instance joins the Layer-1 fanout. websearch searxng up starts one and sets this for you, with or without Docker

All three read from a gitignored .env in the working directory if you have one (copy .env.example), which keeps NordVPN service credentials out of your shell history. An exported variable always beats the file, and --vpn / --proxy / --searxng-url beat both for a single run.

Self-hosting SearXNG

You never need this; the keyless ddgs engines work out of the box. Run your own SearXNG when you want the broadest, most reliable search: every engine SearXNG can reach, your own server, no public rate limits, and still no API keys. It is also the tool's second opinion: websearch doctor uses it to tell a stale ddgs parser apart from a provider blocking your IP.

Two ways to run one. Pick by whether the machine has Docker.

No Docker (works anywhere, including an agent sandbox):

websearch searxng up                          # clone, install, start detached, wire it in
websearch web-search "your query"             # now fuses SearXNG + ddgs
websearch searxng status                      # where it lives, whether it answers
websearch searxng down                        # stop it

The first up clones upstream SearXNG and builds a virtualenv beside it, about 15 to 30 seconds and a few hundred MB; later ones only start it. Everything lands in one state directory (WEBSEARCH_SEARXNG_HOME, defaulting beside WEBSEARCH_ENV_FILE or in the XDG cache), and WEBSEARCH_SEARXNG_URL is written into your env file for you, so the next search picks it up with nothing else to do. The server is started in its own session rather than as a child of the command, which is what lets it survive an agent CLI killing the process group of every shell command it runs. WEBSEARCH_SEARXNG_PORT moves it off 8888. The same thing is on the MCP face as searxng_setup, so an agent that only speaks MCP can set it up itself.

With Docker:

./docker/searxng/searxng.sh up                # generates a per-machine secret, waits for health
export WEBSEARCH_SEARXNG_URL=http://127.0.0.1:8888
uv run websearch web-search "your query"

One container, nothing installed on the host, and it can route SearXNG's own engine requests through your egress proxy. SearXNG ships ~280 engines but leaves most of them off, so a stock instance answers a general query from about six of them (and on a home connection Brave and Startpage return a CAPTCHA). searxng.sh engines probes every engine from your own connection and enables the ones that answer, recording the reason next to each one it skips. Measured here: a general query went from 26 results across 2 engines to 155-240 across 21-29, in 3 to 4 seconds.

The container is kept in its box: config mounted read-only, runtime state in a Docker volume rather than the repo, all capabilities dropped, loopback-only by default, and no secret committed. Torrent trackers and shadow libraries stay out of the default fanout (still queryable by name). Details and the checklist for exposing it beyond localhost are in docker/searxng/.

Both bind to loopback and turn the JSON API on. The difference is the engine list: the no-Docker path runs upstream defaults (83 engines enabled of 278 here), while the Docker stack's probe enables everything that answers from your own connection (213 of 279 on the same machine, same day). The Docker stack also routes SearXNG's own engine requests through your egress proxy. Use it when you have Docker; use websearch searxng up when you do not.

Egress proxy

One switch, WEBSEARCH_PROXY, routes every network path (search engines, fetch tiers, arxiv, github, and the MCP tools) through a proxy:

export WEBSEARCH_PROXY=socks5h://user:pass@host:1080   # any proxy URL (http:// works too)
export WEBSEARCH_PROXY=nordvpn                         # NordVPN shorthand, see below
export WEBSEARCH_PROXY=off                             # or unset it: direct connection

The nordvpn shorthand builds the SOCKS5 URL for you from NORDVPN_USER and NORDVPN_PASS. These are the service credentials shown in the Nord Account dashboard under "Set up NordVPN manually", not your account login. NORDVPN_HOST picks a specific server (default nl.socks.nordhold.net; any of the official *.socks.nordhold.net hosts on port 1080 works).

Every network command also takes --proxy <url|nordvpn|off>, which overrides the variable for that run, so --proxy off gets you a direct connection without unsetting anything. Prefer socks5h:// over socks5://: it resolves DNS through the proxy, so hostnames never hit your local resolver. A per-request fetch --proxy still wins over the process-wide default.

Nothing bypasses it. With WEBSEARCH_PROXY set, every path that leaves the machine goes through it: both fetch tiers, the ddgs engines, arXiv, GitHub, the MCP tools, and the doctor's own probes. Two consequences that are easy to get wrong and are tested for:

• No DNS leak. The SSRF guard used to resolve every hostname locally before fetching, which showed your resolver, and therefore your ISP, every site you visited while the traffic itself was tunneled. Behind a proxy it no longer resolves: socks5h:// resolves at the exit node, so the local lookup never decided the route anyway, and skipping it removes the leak. Literal IPs are still refused without any lookup, so http://127.0.0.1 and the 169.254.169.254 metadata endpoint stay blocked.
• Fail closed. Point the proxy at a dead port and every component errors rather than falling back to a direct connection. That includes the two that open sockets in native code, ddgs (Rust) and curl_cffi (libcurl), which a Python-level check cannot see.

The one deliberate exception is opt-in: websearch doctor --baseline makes a single direct request to learn your own exit IP so the proxied one can be compared against it. It is off by default, because handing your real address to an echo service while paying for a VPN to hide it is the wrong trade.

With a self-hosted SearXNG there are two hops and the proxy applies to the one that matters. The client's hop to 127.0.0.1 is never proxied, because asking a remote exit node to reach 127.0.0.1 gets you its localhost, not yours. The hop that actually reaches Google and Bing is SearXNG's own, out of the container, and docker/searxng/searxng.sh up routes that one through the same WEBSEARCH_PROXY (expanding the nordvpn shorthand for you). searxng.sh egress prints both IPs so you can see it. Set SEARXNG_OUTGOING_PROXY=off for direct container egress; a SearXNG on a public address still goes through the proxy on the client hop too.

VPN

WEBSEARCH_VPN does not route anything. It records what you expect, so a tunnel that silently drops fails a check instead of quietly leaking your real IP. nordvpn is verified against NordVPN's own keyless connection endpoint, which reports whether the caller is inside their network; any only asserts that a tunnel interface is up, because no third party can confirm an arbitrary provider by name. The doctor checks the path the tool actually uses: through the egress proxy when one is set, direct otherwise.

nordvpn works on every platform, since it is an HTTP check. any reads interface names, which are meaningful on Linux (tun, wg, nordlynx) and macOS (utun) but not on Windows, where they look like ethernet_32770; there it reports the tunnel as unconfirmed rather than guessing.

websearch doctor

One command that says what works right now, capability by capability:

uv run websearch doctor                          # everything
uv run websearch doctor --quick                  # skip the engine fanout, tools, and fetch tiers
uv run websearch doctor --check engines --check proxy
uv run websearch doctor --json                   # the same run as an Envelope

It prints the three optional layers' state, then checks Python and the dependency closure, direct internet and the exit IP, the egress proxy and whether the exit IP actually moved, the declared VPN, a self-hosted SearXNG (health, active engine count, live JSON query), each ddgs provider on its own, arXiv and GitHub, both fetch tiers, and the MCP tool registration. Exit code is 1 only when something failed: an optional layer that is off is skipped, and one rate-limited provider is a warning. Proxy credentials never reach the output, including inside HTTP client error text.

The part worth having is the cross-check. ddgs reports "No results found" for a CAPTCHA, for a rate limit, and for an HTTP 200 whose markup its parser no longer reads, and those need opposite fixes. With SearXNG running, the doctor asks it about exactly the providers that went quiet, and SearXNG's scrapers are maintained separately:

warn  engine:ddgs:google     No results found. (SearXNG reached it: 260 results)
                             -> ddgs's parser, not a block on your IP. Query it through SearXNG.
warn  engine:ddgs:startpage  No results found. (SearXNG got nothing from it either)
                             -> startpage is refusing this IP. Only a different egress exit changes that.

Two runs on one home connection in July 2026, minutes apart, over the nine ddgs providers: Brave, Yahoo, and Yandex answered both times. Google and Mojeek were silent through ddgs both times and returned 181 and 10 results through SearXNG, so ddgs cannot read their pages any more. Startpage and Wikipedia were empty in both parsers, which is a block, not a parser. DuckDuckGo and Grokipedia flipped between the two runs, and moving the client onto NordVPN's shared SOCKS exit cost one more provider. None of that is stable enough to write into a table, which is why it is a command.

Security

A fetch tool an agent can point anywhere is an SSRF and prompt-injection surface, so:

• SSRF guard (built): fetch enforces an http(s) scheme allowlist and resolves every host, refusing private, loopback, link-local (the 169.254.169.254 cloud-metadata endpoint), reserved, and multicast addresses. Redirects are followed manually with the same check on each hop, so a public URL cannot redirect into the internal network. Override per request with --allow-private-hosts for deliberate internal fetches.
• Untrusted content (built, Layer 3): fetched page text is untrusted input and is never presented as instructions. web_fetch and web_open wrap each page in a fence built from the 2026 primary-source guidance: a per-instance 128-bit random nonce in the open and close markers (so injected text cannot forge the close), a data-only directive, and neutralization of any copy of the marker inside the body, with optional datamarking (--datamark) for higher resistance. This reduces, but does not eliminate, indirect prompt injection: it prevents the boundary breakout, not persuasion. The real guarantees are channel separation (the MCP face delivers content through the tool-result channel), least privilege, and cutting exfiltration paths. The lower-level fetch command still returns the clean body unmodified, so piping and composition stay clean.

Architecture

Each layer is a folder with a port (a capability-named interface) and one or more adapters behind it, connected only by versioned JSON Schema 2020-12 contracts. Port fields are capability-named (snippet, fused_score, sources); a backend's native shape is mapped onto the port inside that backend's adapter. The default deployment runs in-process for speed, and because layers are coupled only through their contracts, a layer can later move to a subprocess or a local service without its neighbors changing. Additive contract changes are MINOR (consumers ignore unknown fields); a removal, rename, or type change is MAJOR, and consumer-driven contract tests fail any producer change that breaks a recorded fixture. Full design in docs/ARCHITECTURE.md.

The 7-axis scorecard

Rather than a single claim that it is better, here is how I measure it, axis by axis, including where it is only at parity or behind:

Axis	This tool	Notes
Retrieval quality	At top-tier parity on the common case	2026 leaders are statistically tied on quality
Freshness	On-demand recency filter	per-engine, best-effort
Extraction recall / noise	Competitive (Trafilatura ~0.79 F1, articles ~0.93)	heuristic beats neural on quality and cost
Anti-bot success	About 70 to 90% without paid proxies	near-cloud with a plugged-in residential adapter
End-to-end latency	The unprotected common case (where leaders actually differ)	multi-engine + fuse adds some cost
Cost	About zero at the software layer	infra documented separately
Citation accuracy	Source-anchored, deduped results	no fabricated URLs

Top agentic-search APIs are tied on quality, and hard anti-bot at scale is irreducibly paid (even Firecrawl scores about 34% on independently tested protected sites). This project concedes the protected long tail to a swappable paid egress adapter and does not claim to beat everything. The built-in proxy egress is scoped to search geo-targeting and rate-limit rotation, not anti-bot for page fetches (commercial VPNs use datacenter IPs that anti-bot systems flag); the paid residential adapter for the long tail is the planned piece.

Benchmark

A same-query, same-moment head-to-head against the web search built into Claude Code (a hosted, paid baseline) is written up in docs/BENCHMARK.md, with the exact commands so you can rerun it. The short version: on finding relevant, fresh pages the two are comparable; this tool adds clean on-demand extraction, multi-engine fusion, the arxiv and github tools, and runs free and locally, while the hosted search is frictionless and writes a summary in one call. It wins on cost, privacy, and control; on raw retrieval the gap is small.

Roadmap

Built: harness packaging ships the SKILL.md plus the bundled tool via npx skills add, a Claude Code plugin and marketplace, the MCP registry (server.json), and PyPI/uvx, with per-harness MCP registration documented in docs/INSTALL.md.

Also built: the opt-in egress proxy (WEBSEARCH_PROXY, --proxy, NordVPN shorthand) covering every network path.

Planned, not built yet:

• Paid egress adapter: a residential-proxy adapter for the protected long tail, plus optional gluetun / wg-netns network-namespace isolation.
• Local rerank: a cross-encoder pass to turn multi-engine recall into precision.
• More engines: keyed adapters (Brave, Exa, Tavily) behind the existing EngineAdapter port; an optional neural index.

Development

uv sync          # install deps (including the dev group)
uv run pytest    # 667 tests, no network
uv run ruff check .

CI runs ruff and pytest on Python 3.11, 3.12, and 3.13 via uv, on Linux only. The package itself is pure Python with no OS-specific imports, and its two compiled dependencies (curl_cffi, and primp under ddgs) publish wheels for Linux, macOS, and Windows on both x86-64 and arm64, so macOS and Windows should work; they are not covered by CI, so run uv run pytest and uv run websearch doctor there before trusting it. Two platform notes: websearch doctor reads network interface names for WEBSEARCH_VPN=any, which Windows does not expose usefully (it reports the tunnel as unconfirmed there, while WEBSEARCH_VPN=nordvpn works everywhere because it is an HTTP check), and the Docker SearXNG stack is driven by a bash script that needs WSL or Git Bash on Windows (websearch searxng up is Python and has no such requirement).

The contract tests validate real output against the frozen JSON Schemas, so a change that breaks a contract shape fails CI. Build one isolated layer at a time, against its versioned contract; adding or swapping an engine or an extractor touches only its adapter module.

License

MIT. See LICENSE. Optional anti-bot tiers that depend on AGPL components (for example nodriver) stay as out-of-band adapters you install separately, not bundled into the MIT core.