websearch-skill 0.3.0

Open-source multi-engine web search + extraction SKILL and CLI for AI agents (uv-native, contract-driven, harness-portable, self-hosted SearXNG and egress proxy)

websearch-skill

Open-source multi-engine web search and content extraction for AI agents. Internal layers communicate through versioned JSON Schema contracts.

Status: early. First public release 2026-06-22; current version in CHANGELOG.md. Available now: keyless search, Markdown extraction, five agent commands, self-hosted SearXNG, and an opt-in egress proxy. These are covered by the test suite. There is no MCP server; use the CLI directly or install the skill (see No MCP below). Hard anti-bot tiers and local reranking remain on the roadmap. Pin a version and test it in a sandbox before using it with sensitive data.

What it is

A self-hosted web search and page reader for AI agents, with no API keys or paid search service. It queries multiple engines, combines and deduplicates the results, then fetches pages as clean Markdown. The CLI runs locally and sends queries directly to the configured engines.

Core commands:

• init reads the env file, starts local SearXNG, and reports available capabilities.
• web_search finds pages: ranked, deduplicated results across engines, each with a reusable handle.
• web_fetch reads a page: clean Markdown, fenced as untrusted, paginated so a long page never overflows context.
• web_open pages back through a document you already fetched, from cache, without hitting the network again.

Additional keyless commands:

• arxiv searches arXiv papers and returns structured metadata (authors, abstract, categories, abstract and PDF links).
• github searches GitHub repositories and returns typed fields you can sort on (stars, language, topics).

Engines, out of the box, no keys

The default engine is the keyless ddgs metasearch library, which spans Google, Brave, DuckDuckGo, Yandex, Yahoo, Startpage, Mojeek, Wikipedia, and Grokipedia. Each query uses the providers that respond. The agent-facing web-search command requires no account, service, or engine flags. The lower-level search command can restrict providers with --ddgs-backends google,brave,mojeek. Provider availability varies by network; websearch doctor reports it per provider.

For broader and more reliable search you can run your own SearXNG (~280 engines, your own server, still no keys), and the router fuses it with ddgs and de-correlates the engines they share. websearch searxng up sets one up and starts it on any machine with git and Python; on a Docker host, docker/searxng/ is the curated stack, including a probe that enables every engine that answers on your connection. See Self-hosting SearXNG below. Public SearXNG instances are deliberately not a default: most disable the JSON API and rate-limit automated clients, so depending on them would break on a fresh install.

The project is MIT-licensed. Optional integrations include self-hosted SearXNG, keyed engines (Brave, Exa, Tavily), and a paid egress adapter.

The scorecard below compares result quality, latency, cost, privacy, and retrieval limits. The software layer is free and self-hosted, with multi-engine search, deduplication, and Markdown extraction. Protected sites may require the planned paid egress adapter; residential proxies and CAPTCHA solving have no reliable free equivalent.

Layer status

Layer	What it does	Status
Layer 1: Search	Multi-engine router (keyless ddgs across many engines, optional self-hosted SearXNG), canonicalize, dedup, de-correlated RRF fusion	Built
Layer 2A: Fetch + Extract	Tiered fetch (httpx, curl_cffi impersonation), Trafilatura extraction to Markdown + metadata	Built
Layer 2B: Format + Store	Paginated Markdown + JSON sidecar, progressive-disclosure index/resolver, MinHash dedup, ephemeral SQLite-FTS5 store	Built
Layer 3: Agent I/O	Consolidated web_search/web_fetch/web_open, untrusted-content fence, SKILL.md	Built
Extra tools	Keyless arxiv (paper search) and github (repo search), standalone over the same Envelope	Built
Doctor	websearch doctor: per-capability self-test of the optional layers, every engine, the tools, and both fetch tiers	Built
Init	websearch init: reads the env file, starts SearXNG, runs diagnostics, and reports capability state	Built

Contracts are frozen as JSON Schema 2020-12: envelope@1.0.0, search@1.1.0, fetch@1.2.0, extract@1.0.0, format@1.0.0, store@1.0.0, agent-io@1.1.0, arxiv@1.1.0, github@1.1.0, doctor@2.0.0, searxng@1.0.0, init@1.0.0. Every response is wrapped in one Envelope { contract_version, ok, data, error, meta }.

Layer 1: Search

Search normalizes a request, sends it to each configured EngineAdapter, canonicalizes and deduplicates URLs, and combines the rankings with weighted Reciprocal Rank Fusion (RRF, k=60). The keyless default is ddgs, a metasearch library for Google, Brave, DuckDuckGo, Yandex, Yahoo, Startpage, Mojeek, Wikipedia, and Grokipedia. On the lower-level search command, --ddgs-backends selects a provider subset and --engines selects the ddgs and SearXNG adapters. The agent-facing web-search command uses the defaults. Set WEBSEARCH_SEARXNG_URL to add a self-hosted SearXNG instance. Keyed engines such as Brave, Exa, and Tavily are planned.

ddgs is a keyless Python library that runs in-process. SearXNG is a separate self-hosted server with a larger engine catalog. Search can use either adapter or combine both. SearXNG provides search results only; page fetching and extraction remain in Layer 2A.

De-correlation prevents duplicate upstream data from distorting the ranking. SearXNG and ddgs both use crawlers such as Google and Bing, so a naive union can count the same source twice. Results are grouped by correlation_group; each group contributes one RRF term at its best rank, and the consensus bonus uses only the number of distinct groups. SearXNG and ddgs agreeing counts as one independent vote; SearXNG and a neural index agreeing counts as two. The response includes a warning when de-correlation is applied.

Each result records which engines returned it and its rank within each engine.

Layer 2A: Fetch + Extract

Fetching and extraction use separate interfaces and can be replaced independently.

Fetch starts with httpx and retries with curl_cffi browser impersonation only when it detects an anti-bot challenge. It does not retry 404s, rate limits, authentication failures, or legal and geographic blocks because the same network address is unlikely to change the result. Detection checks response headers, then known body markers for Cloudflare, DataDome, PerimeterX, Akamai, and Imperva. Imperva body markers are checked on every status because its blocks can return HTTP 200. Browser and stealth tiers (Crawl4AI, nodriver) are present in the contract but are not part of the base install.

Extract defaults to Trafilatura, a heuristic extractor. Per the May 2026 WCXB benchmark, heuristic extractors beat neural extractors on both quality and cost (neural runs tens to hundreds of times more expensive), and Trafilatura lands at about 0.79 F1 on CPU in roughly 100ms; article pages saturate around 0.93 F1. The adapter parses the raw HTML once with lxml to recover the raw schema.org JSON-LD blocks and og:type (Trafilatura folds JSON-LD into metadata and never exposes the raw blocks), runs Trafilatura for the Markdown body and plain text plus metadata, then computes:

• a heuristic quality_score (0..1) from runtime signals (text density, word-count saturation, paragraph count, inverse link density, JSON-LD presence, clean title) with hard vetoes for soft-404s and shells; below about 0.80 a page is a fallback candidate.
• a cheap page_type resolved from JSON-LD @type, then og:type, then URL shape.

Neural extract engines (crawl4ai, jina_readerlm, and others) are named in the contract enum but stay opt-in. The default dependency closure is permissive: Apache-2.0 (trafilatura) plus MIT/BSD/MPL deps.

Extraction does not truncate content_markdown. --max-bytes limits the downloaded response size, and --max-bytes 0 disables that transport limit. A value of 0 also disables --max-results, --page-size-tokens, and --per-page, subject to provider limits such as GitHub's 100 results per page and arXiv's 2,000 results per request.

Layer 2B: Format + Store

This layer formats results for agents and stores full pages for later access.

Format takes vendor-neutral results from Layers 1 and 2A and renders a layout-stable Markdown document plus a JSON sidecar containing the same data. Results are ordered by relevance and paginated according to the configured token budget. Near-duplicate deduplication runs first: byte-exact (normalized SHA-256), then pure-Python MinHash over word 4-gram shingles (128 permutations, Jaccard 0.9) clustered with union-find. The highest-scored page in each cluster is kept and the rest are listed in dropped_duplicates. The default threshold merges near-identical pages such as mirrors and syndicated copies, while keeping pages that are only topically similar. Lower jaccard_threshold for boilerplate-heavy corpora. The implementation does not depend on datasketch. Progressive disclosure controls how much to inline:

• auto (default) inlines full bodies when the page fits a token budget, otherwise switches to an index (a preview plus a stable id).
• index always shows a preview and a resolve hint; full always inlines.

The optional anthropic_search_result_blocks view maps 1:1 onto Anthropic search_result content blocks (source as a bare string, at least one non-empty text block, citations all-or-nothing). It is off by default and separate from the canonical response shape.

The sidecar and store retain the full body in every mode. body_char_budget moves content from the rendered Markdown view into the resolver and adds a lookup hint; it does not modify the stored content. --no-truncate keeps the full body inline.

Store indexes fetched pages behind a PageIndex interface (add / search / get / resolve_index). Per-query result sets remain as plain Python objects; only fetched pages enter the index. The default adapter uses SQLite FTS5 with BM25 and requires no third-party package. If the local SQLite build lacks FTS5, it falls back to a pure-Python BM25 index with the same response shape. Adds are idempotent by URL and content hash, queries are escaped before reaching FTS5, and a file path enables persistence. Optional vector or Rust backends can implement the same interface.

Layer 3: Agent I/O

Layer 3 exposes web_search (find), web_fetch (read a URL), and web_open (page through an already-fetched document). Each returns the same Envelope. The CLI and portable SKILL.md use the same implementation. The skill follows the Agent Skills format and can be installed in Claude Code, Codex, OpenCode, and other compatible agents.

Commands refer to pages with a human-readable handle (site~shorthash, for example en.wikipedia.org~3a1f9c2b5e6f). web_fetch indexes the full page in the Layer 2B store and returns one token-budget page; web_open reads later pages from that store without fetching the URL again. Because each command runs in a new process, the store uses a file by default beside the env file or in the XDG cache. --persist-path off keeps it in memory. Pagination does not discard content; the full body remains available page by page. The lower-level search / fetch / open commands remain available for debugging and composition.

Fetched page text is untrusted, so web_fetch and web_open wrap each page in a fence (see Security). The fenced content is returned as tool output, separate from user instructions.

Extra keyless tools

Two standalone tools cover sources that general web search handles poorly, both keyless and over the same Envelope:

• arxiv searches arXiv via the official Atom API and returns structured papers (title, authors, abstract, categories, abstract and PDF links). It supports field-targeted search (--field title|author|abstract) and sorting by date or relevance, uses GET so it benefits from arXiv's cache, and backs off on the 2026 rate limiting.
• github searches GitHub repositories via the unauthenticated REST API and returns typed fields (full name, stars, forks, language, topics, updated date). Unauthenticated search is about 10 requests per minute; rate-limited requests return a rate_limited error. Code search needs a token and is not included in the keyless path.

uv run websearch arxiv "diffusion models for protein design" --max-results 5 --sort-by submittedDate
uv run websearch github "vector database" --language Rust --sort stars --per-page 10

There are no dedicated Reddit or X tools because neither has a keyless search API compatible with its terms in 2026 (Reddit's anonymous JSON endpoints return 403 as of May 2026; X requires a paid API or a logged-in account). Use an open-web site filter:

uv run websearch web-search "rust async" --site reddit.com
uv run websearch web-search "frontier model release" --site x.com

No MCP (since 0.3.0)

Version 0.3.0 removed the MCP server. The server was a long-lived process that read configuration at startup and cached its engine set. Changes to the proxy, self-hosted SearXNG, engine selection, or env file required a client restart. CLI commands start a new process and read that configuration on every call. websearch init handles setup and reports the resulting capabilities.

For an existing websearch mcp installation, install the skill with npx skills add hec-ovi/websearch-skill; it exposes the commands through the agent's shell.

Install

Pick the route that matches how you use it. Everything is keyless and needs internet; the only hard requirement is uv. Full per-harness instructions (Claude Code, Codex, OpenCode, Cursor, Hermes, OpenClaw, and PyPI publishing) are in docs/INSTALL.md.

As a CLI tool, no install (uvx builds an ephemeral env and runs it):

uvx websearch-skill web-search "your query"
# or straight from git, for an unreleased commit:
uvx --from git+https://github.com/hec-ovi/websearch-skill websearch web-search "your query"

As an agent skill across 40+ agents via the skills CLI (it installs the skills/web-search/ directory into each detected agent):

npx skills add hec-ovi/websearch-skill            # all detected agents
npx skills add hec-ovi/websearch-skill -a claude-code -a codex -s web-search

As a Claude Code plugin:

/plugin marketplace add hec-ovi/websearch-skill
/plugin install web-search@websearch-skill

From source (development, uv-native):

git clone https://github.com/hec-ovi/websearch-skill
cd websearch-skill
uv sync
uv run websearch web-search "your query"

Quickstart

Search works with no setup: the keyless ddgs metasearch is the default. The agent-facing web-search (Layer 3) needs no engine flags. The lower-level search (Layer 1) is for debugging and power use, and is the only command that takes --engines, --ddgs-backends, and --no-ddgs:

# Layer 1: search (keyless, multi-engine via ddgs)
uv run websearch search "rust ownership" --json

# force specific keyless engines (search command only, not web-search)
uv run websearch search "rust ownership" --ddgs-backends google,brave,mojeek

# add a self-hosted SearXNG (see docker/searxng/) as a second, broader engine
export WEBSEARCH_SEARXNG_URL=http://127.0.0.1:8888
uv run websearch search "rust ownership" --engines searxng,ddgs

# Layer 2A: fetch + extract one page to clean Markdown
uv run websearch fetch "https://en.wikipedia.org/wiki/Rust_(programming_language)" --json

# Layer 2B: open several pages into one paginated, deduplicated Markdown document,
# and full-text search the passages across them
uv run websearch open \
  "https://en.wikipedia.org/wiki/Rust_(programming_language)" \
  "https://doc.rust-lang.org/book/ch04-00-understanding-ownership.html" \
  --search "ownership borrow checker"

# Layer 3: the fenced, handle-keyed agent interface
uv run websearch web-search "rust ownership" --json
uv run websearch web-fetch "https://doc.rust-lang.org/book/ch04-01-what-is-ownership.html" \
  --page-size-tokens 4000
# page through the rest of a fetched doc by its handle, from cache (no refetch, no flags:
# the page index is on disk by default, so a later command resolves the same handle)
uv run websearch web-open "doc.rust-lang.org~<hash>" --page 2

# extra keyless tools: arXiv papers and GitHub repos
uv run websearch arxiv "mixture of experts scaling laws" --max-results 5
uv run websearch github "llm agent framework" --language Python --sort stars

# initialize local services and report available capabilities
uv run websearch init

# check configured capabilities and providers
uv run websearch doctor

Every command prints a compact human view by default, or the raw JSON Envelope with --json (exit 0 on success, 1 on a request-level error). For the fetch command, --output-format {markdown,text,json} selects the body representation the human view prints (text emits the plain-text rendering), and --quiet prints only the extracted body, for piping. For the open command, --mode {auto,index,full} controls progressive disclosure, --no-truncate inlines every full body, --search QUERY runs a BM25 passage search over the opened pages, and --anthropic-blocks adds the Anthropic search_result view to the sidecar. The agent-facing web-search takes --max-results, --detail, --freshness, --site, --language, --country, --safesearch, --offset, and --searxng-url; the engine-selection flags (--engines, --ddgs-backends, --no-ddgs) live only on the lower-level search command, alongside --searxng-url (or WEBSEARCH_SEARXNG_URL). See uv run websearch <command> --help for the full flag list.

A fetch --json response looks like:

{
  "contract_version": "1.0.0",
  "ok": true,
  "data": {
    "source": { "final_url": "https://...", "status": 200, "fetched_via": "http", "blocked": false },
    "result": {
      "title": "Rust (programming language)",
      "page_type": "article",
      "quality_score": 0.91,
      "word_count": 8123,
      "content_markdown": "# Rust ...",
      "metadata": { "og_type": "article" }
    }
  },
  "error": null,
  "meta": { "layer": "extract", "backend": "http", "elapsed_ms": 412 }
}

As a library:

from websearch.layer1_search import build_router, SearchRequest

router = build_router(searxng_url="http://127.0.0.1:8888", enable_ddgs=True)
envelope = router.search(SearchRequest(query="rust ownership", count=10))
for r in envelope.data["results"]:
    print(r["fused_score"], r["url"], [s["engine"] for s in r["sources"]])

Use Layer 2B (format + store) the same way. run returns an Envelope; its data is the JSON FormatPayload, so markdown and the full sidecar use normal dictionary access:

from websearch.layer2_format import (
    FormatRequest, ResultInput, build_format_pipeline,
    build_page_index, StoreConfig, PageInput, SearchPageRequest,
)

env = build_format_pipeline().run(
    FormatRequest(
        query="rust ownership",
        results=[ResultInput(url="https://x.test/a", title="A", score=0.9, body_markdown="# A ...")],
    )
)
document = env.data["markdown"]       # the layout-stable document
sidecar = env.data["sidecar"]         # identical data with full page bodies

# an ephemeral page index for passage search and resolve-by-id
index = build_page_index(StoreConfig())          # in-memory SQLite FTS5 (BM25)
index.add([PageInput(url="https://x.test/a", markdown="# A ...")])
hits = index.search(SearchPageRequest(query="borrow checker"))

Optional layers (all off by default)

The base install searches keyless engines over a direct connection. Three optional settings add VPN verification, proxy routing, or self-hosted SearXNG. websearch doctor verifies each configured setting:

Layer	Switch	What it adds
VPN	WEBSEARCH_VPN=nordvpn or any	declares that egress should be tunneled, so the doctor verifies it instead of assuming it. It routes nothing on its own: the tunnel is your VPN app's job
Egress proxy	WEBSEARCH_PROXY=<url> or nordvpn	one proxy for every network path the tool opens
SearXNG	WEBSEARCH_SEARXNG_URL=<url>	a self-hosted metasearch instance joins the Layer-1 fanout. websearch searxng up starts one and sets this for you, with or without Docker

All three read from a gitignored .env in the working directory if you have one (copy .env.example), which keeps NordVPN service credentials out of your shell history. An exported variable always beats the file, and --vpn / --proxy / --searxng-url beat both for a single run.

Self-hosting SearXNG

SearXNG is optional; the keyless ddgs engines work without it. A self-hosted SearXNG adds more engines without relying on a public instance or API keys. websearch doctor can compare its results with ddgs to distinguish a parser failure from a provider blocking your IP.

Choose the setup based on Docker availability.

No Docker (works anywhere, including an agent sandbox):

websearch searxng up                          # clone, install, start detached, wire it in
websearch web-search "your query"             # now fuses SearXNG + ddgs
websearch searxng status                      # where it lives, whether it answers
websearch searxng down                        # stop it

The first up clones upstream SearXNG and builds a virtualenv beside it, about 15 to 30 seconds and a few hundred MB; later ones only start it. Everything lands in one state directory (WEBSEARCH_SEARXNG_HOME, defaulting beside WEBSEARCH_ENV_FILE or in the XDG cache), and WEBSEARCH_SEARXNG_URL is written into your env file for you, so the next search picks it up with nothing else to do. The server is started in its own session rather than as a child of the command, which is what lets it survive an agent CLI killing the process group of every shell command it runs. WEBSEARCH_SEARXNG_PORT moves it off 8888. websearch init does this step for you as part of bringing everything up.

With Docker:

./docker/searxng/searxng.sh up                # generates a per-machine secret, waits for health
export WEBSEARCH_SEARXNG_URL=http://127.0.0.1:8888
uv run websearch web-search "your query"

One container, nothing installed on the host, and it can route SearXNG's own engine requests through your egress proxy. SearXNG ships ~280 engines but leaves most of them off, so a stock instance answers a general query from about six of them (and on a home connection Brave and Startpage return a CAPTCHA). searxng.sh engines probes every engine from your own connection and enables the ones that answer, recording the reason next to each one it skips. Measured here: a general query went from 26 results across 2 engines to 155-240 across 21-29, in 3 to 4 seconds.

The container mounts its configuration read-only, stores runtime state in a Docker volume, drops all capabilities, and binds to loopback by default. No secret is committed. Torrent trackers and shadow libraries are excluded from the default engine set but remain queryable by name. See docker/searxng/ before exposing it beyond localhost.

Both bind to loopback and turn the JSON API on. The difference is the engine list: the no-Docker path runs upstream defaults (83 engines enabled of 278 here), while the Docker stack's probe enables everything that answers from your own connection (213 of 279 on the same machine, same day). The Docker stack also routes SearXNG's own engine requests through your egress proxy. Use it when you have Docker; use websearch searxng up when you do not.

Egress proxy

One switch, WEBSEARCH_PROXY, routes every network path (search engines, fetch tiers, arxiv, github) through a proxy:

export WEBSEARCH_PROXY=socks5h://user:pass@host:1080   # any proxy URL (http:// works too)
export WEBSEARCH_PROXY=nordvpn                         # NordVPN shorthand, see below
export WEBSEARCH_PROXY=off                             # or unset it: direct connection

The nordvpn shorthand builds the SOCKS5 URL for you from NORDVPN_USER and NORDVPN_PASS. These are the service credentials shown in the Nord Account dashboard under "Set up NordVPN manually", not your account login. NORDVPN_HOST picks a specific server (default nl.socks.nordhold.net; any of the official *.socks.nordhold.net hosts on port 1080 works).

Every network command also takes --proxy <url|nordvpn|off>, which overrides the variable for that run, so --proxy off gets you a direct connection without unsetting anything. Prefer socks5h:// over socks5://: it resolves DNS through the proxy, so hostnames never hit your local resolver. A per-request fetch --proxy still wins over the process-wide default.

With WEBSEARCH_PROXY set, all outbound paths use it: both fetch tiers, the ddgs engines, arXiv, GitHub, and the doctor's probes. Tests cover two important behaviors:

• DNS resolution. With a proxy configured, the SSRF guard does not resolve hostnames locally. socks5h:// resolves them at the exit node. Literal IPs are still refused without a lookup, including http://127.0.0.1 and the 169.254.169.254 metadata endpoint.
• Failure behavior. If the proxy is unavailable, every component returns an error instead of retrying over a direct connection. This includes ddgs (Rust) and curl_cffi (libcurl), which open sockets outside Python.

The opt-in websearch doctor --baseline check makes one direct request to compare the direct and proxied exit IPs. It is disabled by default because it sends the direct IP to an echo service.

A self-hosted SearXNG request has two hops. The client connects directly to local SearXNG at 127.0.0.1; proxying that address would target the proxy server's localhost. SearXNG then connects to upstream engines from its container, and docker/searxng/searxng.sh up routes that hop through WEBSEARCH_PROXY (including the nordvpn shorthand). searxng.sh egress prints both IPs. Set SEARXNG_OUTGOING_PROXY=off for direct container egress. A remote SearXNG address still uses the client proxy.

VPN

WEBSEARCH_VPN records the expected tunnel state but does not configure routing. If the tunnel drops, the doctor reports a failure. nordvpn is verified against NordVPN's keyless connection endpoint, which reports whether the caller is inside its network. any checks only for an active tunnel interface because there is no provider-specific endpoint. The doctor checks through the egress proxy when one is set and directly otherwise.

nordvpn works on every platform, since it is an HTTP check. any reads interface names, which are meaningful on Linux (tun, wg, nordlynx) and macOS (utun) but not on Windows, where they look like ethernet_32770; there it reports the tunnel as unconfirmed rather than guessing.

websearch init

websearch init reads the configured env file, starts local SearXNG unless --skip-searxng is set, and runs the doctor checks:

uv run websearch init                # bring everything up, then report
uv run websearch init --json         # the same run as an Envelope
uv run websearch init --quick        # skip the slow sweep; unprobed capabilities read "unknown"

  ok    env       read /config/websearch.env
  ok    searxng   SearXNG 2026.7.25 answering at http://127.0.0.1:8888, 83 of 278 engines
  warn  doctor    18 ok, 4 warn, 0 fail, 3 skipped

capabilities
  web_search  ok
  web_fetch   ok
  arxiv       ok
  github      ok
  searxng     ok
  proxy       ok
  vpn         off

ready: search online (5 of 9 keyless providers + SearXNG, 83 engines); fetch, arxiv, github ok; egress through socks5h://***:***@nl.socks.nordhold.net:1080

Wait for data.ready. data.state is ready, degraded (search works but a requested capability is missing), or broken (search does not work). data.capabilities lists available commands, data.next_actions lists remediation steps, and data.doctor contains the full diagnostic payload. The exit code is 0 when search works, including a degraded result, and 1 when it does not.

websearch doctor

websearch doctor checks current capability and provider status:

uv run websearch doctor                          # everything
uv run websearch doctor --quick                  # skip the engine fanout, tools, and fetch tiers
uv run websearch doctor --check engines --check proxy
uv run websearch doctor --json                   # the same run as an Envelope

It prints the three optional layers' state, then checks Python and the dependency closure, direct internet and the exit IP, the egress proxy and whether the exit IP actually moved, the declared VPN, a self-hosted SearXNG (health, active engine count, live JSON query), each ddgs provider on its own, arXiv and GitHub, and both fetch tiers. Exit code is 1 only when something failed: an optional layer that is off is skipped, and one rate-limited provider is a warning. Proxy credentials never reach the output, including inside HTTP client error text.

The SearXNG cross-check helps classify ddgs failures. ddgs reports "No results found" for CAPTCHAs, rate limits, and successful responses whose markup its parser cannot read. With SearXNG running, the doctor queries the same provider through SearXNG's separately maintained scraper:

warn  engine:ddgs:google     No results found. (SearXNG reached it: 260 results)
                             -> ddgs's parser, not a block on your IP. Query it through SearXNG.
warn  engine:ddgs:startpage  No results found. (SearXNG got nothing from it either)
                             -> startpage is refusing this IP. Only a different egress exit changes that.

Two runs on one home connection in July 2026, minutes apart, produced different results across the nine ddgs providers. Brave, Yahoo, and Yandex answered both times. Google and Mojeek were silent through ddgs but returned 181 and 10 results through SearXNG, indicating ddgs parser failures. Startpage and Wikipedia were empty through both paths, indicating a block. DuckDuckGo and Grokipedia changed between runs, and NordVPN's shared SOCKS exit lost one additional provider. Run the doctor for current results.

Security

Fetch accepts arbitrary URLs and page content, so it includes SSRF and prompt-injection controls:

• SSRF guard (built): fetch enforces an http(s) scheme allowlist and resolves every host, refusing private, loopback, link-local (the 169.254.169.254 cloud-metadata endpoint), reserved, and multicast addresses. Redirects are followed manually with the same check on each hop, so a public URL cannot redirect into the internal network. Override per request with --allow-private-hosts for deliberate internal fetches.
• Untrusted content (built, Layer 3): fetched page text is never presented as instructions. web_fetch and web_open wrap each page in a fence based on 2026 primary-source guidance: a per-instance 128-bit random nonce in the opening and closing markers, a data-only directive, and neutralization of marker copies inside the body. Optional datamarking (--datamark) adds resistance. The fence prevents boundary breakout but cannot prevent persuasion. Tool-output separation, least privilege, and blocked exfiltration paths are still required. The lower-level fetch command returns the extracted body without the fence for piping and composition.

Architecture

Each layer is a folder with a port (a capability-named interface) and one or more adapters behind it, connected only by versioned JSON Schema 2020-12 contracts. Port fields are capability-named (snippet, fused_score, sources); a backend's native shape is mapped onto the port inside that backend's adapter. The default deployment runs in-process for speed, and because layers are coupled only through their contracts, a layer can later move to a subprocess or a local service without its neighbors changing. Additive contract changes are MINOR (consumers ignore unknown fields); a removal, rename, or type change is MAJOR, and consumer-driven contract tests fail any producer change that breaks a recorded fixture. Full design in docs/ARCHITECTURE.md.

The 7-axis scorecard

The scorecard separates retrieval quality, cost, latency, and operational limits:

Axis	This tool	Notes
Retrieval quality	Comparable to leading APIs for common queries	2026 leaders are statistically tied on quality
Freshness	On-demand recency filter	per-engine, best-effort
Extraction recall / noise	Competitive (Trafilatura ~0.79 F1, articles ~0.93)	heuristic beats neural on quality and cost
Anti-bot success	About 70 to 90% without paid proxies	protected sites require residential egress
End-to-end latency	Varies with responsive engines	multi-engine fanout and fusion add latency
Cost	About zero at the software layer	infra documented separately
Citation accuracy	Source-anchored, deduped results	no fabricated URLs

Protected sites usually require paid residential egress (even Firecrawl scores about 34% on independently tested protected sites). The built-in proxy supports search geo-targeting and rate-limit rotation. It does not bypass page-level anti-bot systems because commercial VPNs use datacenter IPs that those systems commonly flag. A residential egress adapter is planned.

Benchmark

docs/BENCHMARK.md compares the same queries at the same time against the hosted web search in Claude Code and includes commands for reproduction. Both found comparable relevant and recent pages. This project adds local extraction, multi-engine fusion, and the arxiv and github commands; the hosted search returns a summary in one call.

Roadmap

Current distribution options include npx skills add, a Claude Code plugin and marketplace, and PyPI/uvx. See docs/INSTALL.md for each supported agent.

The opt-in egress proxy (WEBSEARCH_PROXY, --proxy, NordVPN shorthand) covers every outbound network path.

Planned, not built yet:

• Paid egress adapter: a residential-proxy adapter for the protected long tail, plus optional gluetun / wg-netns network-namespace isolation.
• Local rerank: a cross-encoder pass to turn multi-engine recall into precision.
• More engines: keyed adapters (Brave, Exa, Tavily) behind the existing EngineAdapter port; an optional neural index.

Development

uv sync          # install deps (including the dev group)
uv run pytest    # 748 tests, no network
uv run ruff check .

CI runs ruff and pytest on Python 3.11, 3.12, and 3.13 via uv, on Linux only. The package itself is pure Python with no OS-specific imports, and its two compiled dependencies (curl_cffi, and primp under ddgs) publish wheels for Linux, macOS, and Windows on both x86-64 and arm64, so macOS and Windows should work; they are not covered by CI, so run uv run pytest and uv run websearch doctor there before trusting it. Two platform notes: websearch doctor reads network interface names for WEBSEARCH_VPN=any, which Windows does not expose usefully (it reports the tunnel as unconfirmed there, while WEBSEARCH_VPN=nordvpn works everywhere because it is an HTTP check), and the Docker SearXNG stack is driven by a bash script that needs WSL or Git Bash on Windows (websearch searxng up is Python and has no such requirement).

The contract tests validate real output against the frozen JSON Schemas, so a change that breaks a contract shape fails CI. Build one isolated layer at a time, against its versioned contract; adding or swapping an engine or an extractor touches only its adapter module.

License

MIT. See LICENSE. Optional anti-bot tiers that depend on AGPL components (for example nodriver) stay as out-of-band adapters you install separately, not bundled into the MIT core.