Why

Most scraping tools assume one engine. webget assumes the web is messy:

FETCH_AUTO
├── http       → fast path (httpx + trafilatura/markdownify), no browser
├── crawl4ai   → Playwright browser, JS rendering, persistent auth sessions
└── firecrawl  → optional cloud fallback (needs WEBGET_FIRECRAWL_KEY)

Every fetch classifies what it hit - success, login_required, challenge, blocked, or error - and reports it in machine-readable JSON. webget never pretends an empty page is success, and it never solves CAPTCHAs or evades anti-bot systems; it tells you honestly what happened.

Install

Requires Python 3.11+.

PyPI (webget-cli)

The CLI command is webget; the PyPI package name is webget-cli (the bare webget name is taken by an unrelated package).

# pip - HTTP fast path + search only (no browser)
pip install webget-cli

# pip - full stack with Crawl4AI/Playwright browser fallback
pip install "webget-cli[browser]"

# uv tool - isolated executable on your PATH
# (single spec with extras: do NOT use `--with "webget-cli[browser]"`,
#  uv re-resolves the same package there and can clobber the main install
#  with a stale version)
uv tool install "webget-cli[browser,mcp]"

After install, webget is available as a command:

webget --help

Browser runtime (optional)

Crawl4AI drives a Playwright Chromium. pip install "webget-cli[browser]" installs the Python packages; the browser binary itself is downloaded separately:

python -m playwright install chromium

Without the browser extra, webget still works for search and plain HTTP fetches. A fetch that needs the browser (JS rendering, --profile sessions, login) prints a clear warning telling you how to install it.

From source (development)

git clone https://github.com/DavidPandleton/webget
cd webget
uv pip install -e ".[dev,browser]"

Usage

webget s "rust async runtime"             # search (ddgs metasearch, top 5)
webget s "rust async runtime" --engine brave,duckduckgo   # subset: faster than auto
webget s "rust async runtime" --json      # machine-readable, includes engine provenance
webget u https://example.com              # scrape (auto: http -> crawl4ai)
webget su "llm inference" 5               # search + scrape top 5, parallel
cat urls.txt | webget u -                 # batch scrape, one browser instance
webget fetch https://example.com --json   # machine-readable result

Long aliases: search = s, fetch = u, search-fetch = su.

Search engines

webget searches through ddgs, a metasearch library that aggregates several keyless engines. The default auto queries all of them; naming a subset is faster and skips engines that are having a bad hour.

webget s "query" --engine brave,duckduckgo

Do not hardcode engine names. The set ddgs offers changes between its releases, and webget validates against the registry at runtime rather than a built-in list. Pass a deliberately bogus name to see what your installed version supports:

$ webget s "query" --engine bogus
warning: unknown search engine(s): bogus - using auto
  (known: brave, duckduckgo, google, grokipedia, mojeek, startpage, wikipedia, yahoo)

Unknown names degrade to auto with a warning instead of failing, so a typo never kills a search.

Failover is automatic. If the engine you named fails, or returns zero results, webget tries the remaining engines until a time budget is spent (15s by default, WEBGET_FAILOVER_BUDGET_S to override) and returns the first non-empty set. The budget replaced a fixed "3 engines" cap: a count is either too many (3 x 20s timeout = a minute of dead air) or too few (a live engine sitting fifth in line never got reached). At least two alternates are always tried, however fast the budget expires. This is not paranoia: engine reachability depends on where you are, not just whether a service is up. A benchmark from one residential connection found only 2 of 9 engines reachable, and auto survived purely because they did.

Every substitution is announced, never silent:

$ webget s "rust programming language book" --engine google
warning: engine 'google' failed (No results found.); fell back to 'brave'
1. The Rust Programming Language - doc.rust-lang.org/book/

Results carry provenance. Because failover can answer from an engine you did not ask for, --json reports which one actually did:

$ webget s "linux kernel" --engine google --json
{
  "results": [ ... ],
  "engine": "grokipedia",         # who actually answered
  "requested_engine": "google",
  "failed_over": true
}

This matters most over MCP, where a stderr warning is invisible to the agent: the payload is the only signal that the results came from elsewhere.

Failover order is learned, not hardcoded. webget keeps a small health ledger per install (~/.local/state/webget/engine_health.json) recording whether each engine answered and how fast. When failover kicks in, engines are tried best-first by that record instead of in registry order. The ledger is advisory - it reorders candidates, it never removes one - and it decays: an engine marked dead last week is retried with no penalty this week, because the blocking that made it dead is exactly what changes. There is no baked-in ranking anywhere in the source; the same webget learns a different order on your machine than on the author's, which is the point.

Diagnostics: python -c "from webget import health; print(health.health_snapshot())"

MCP note (breaking in 0.13.0): the search and search_fetch tools now return an object {results, engine, requested_engine, failed_over} instead of a bare list, so provenance has somewhere to live. Errors are {error, results: []}.

Provenance is per-call, not per-result: ddgs merges every engine's hits into one list and discards which engine supplied each hit, so per-hit attribution is not obtainable through its public API.

Options

Authenticated sessions (profiles)

# interactive login: browser opens, YOU log in manually, session persists
webget login https://campus.example --profile campus

# list profiles and their session status
webget profiles
webget profiles --json

# later fetches reuse the session - even on the HTTP fast path
webget fetch https://campus.example/dashboard --profile campus --json

# log out ONE domain, keep the rest of the profile
webget logout https://campus.example --profile campus

webget login never stores passwords and never fills forms. A visible browser opens, you authenticate yourself, then press Enter in the terminal and webget persists the session. Persistent profiles live in ~/.local/share/webget/profiles/<name>; session cookies are exported to storage_state.json inside the profile after each browser run, so the fast path can reuse them. Secrets are never printed.

JSON output

--json returns a dict keyed by URL, so batch results are easy to inspect:

{
  "https://campus.example/dashboard": {
    "status": "success",
    "method": "crawl4ai",
    "cached": false,
    "attempts": 1,
    "auth": {
      "profile": "campus",
      "authenticated": true,
      "state": "success"
    },
    "metadata": {
      "author": "Jane Doe",
      "published_at": "2026-09-01",
      "site_name": "Campus Portal",
      "language": "id"
    },
    "error": null
  }
}

metadata (author, published date, site name, language) comes from trafilatura extraction on the HTTP path; values are null when unknown or when the winning strategy was not HTTP.

Status values: success | login_required | challenge | blocked | error.

The HTTP path also handles non-HTML: JSON becomes a pretty code block, CSV becomes a GFM table, RSS/Atom feeds become a link list, PDFs are extracted per page (via pypdf), plain text passes through. Valid non-HTML payloads count as success even when short.

Status detection rules

Cache

Results are cached in ~/.cache/webget/ (sha1 of url + profile + options, TTL 1h, eviction at 500 files). The cache is content-level, not strategy-level, and isolated per profile - public, campus, and work fetches never collide. Failures are never cached.

Privacy note: cached content is plaintext JSON on disk. If you fetch authenticated/personal pages, use --no-cache.

MCP server

webget_mcp.py exposes the same ladder as an MCP server (search, fetch, search_fetch), so agents like opencode can search and scrape without API keys:

pip install "webget-cli[mcp]"
# or as a uv tool with the MCP server + browser fallback:
uv tool install "webget-cli[browser,mcp]"

Register as a local MCP server in opencode:

{
  "mcp": {
    "webget": {
      "type": "local",
      "command": ["webget-mcp"],
      "enabled": true
    }
  }
}

Then prompt with use webget for search and scrape tasks. Run the server standalone with webget-mcp (stdio transport) or python webget_mcp.py.

Breaking change in 0.13.0: search returns an object, not a list

search and search_fetch used to return a bare JSON array of results. They now return an object, because a list has nowhere to carry provenance:

// <= 0.12.1  ->  a list
[ { "title": "...", "href": "...", "body": "..." }, ... ]

// >= 0.13.0  ->  an object
{
  "results": [ { "title": "...", "href": "...", "body": "..." }, ... ],
  "engine": "brave",            // the engine that actually answered
  "requested_engine": "google", // what was asked for
  "failed_over": true           // an alternative was attempted
}

Error responses changed shape too:

// <= 0.12.1  ->  a plain error string / empty list
// >= 0.13.0
{
  "error": "SearchError: RequestError(...)",
  "results": []
}

How to migrate. If you read the tool result as an array, read .results instead. In JavaScript that is result.results rather than result; in Python data["results"] rather than data. If you only iterate the hits, the change is mechanical.

Why it had to break. Failover means the engine that answers is not always the engine that was asked for, and over MCP there is no stderr for a warning to land on, so the payload is the only place that fact can live. Keeping the list shape would have meant silently returning results from an unexpected engine with no way for a client to notice. An additive field was not possible: a JSON array cannot carry sibling keys.

fetch is unchanged and still returns its string payload. The Python API is unchanged: search() still returns a list, and search_with_provenance() is the new opt-in that returns (results, provenance).

Authenticated sessions (profiles)

MCP tools can use locally stored login sessions. Create one first with the CLI, or let the agent create it via the MCP login tool:

webget login https://portal.example.com --profile portal

Then the agent can discover sessions and fetch authenticated pages:

• list_profiles - lists available sessions (name, last used, size, status). Cookie values are never returned.
• login(url, profile) - opens a browser session (headful by default so a human can log in), navigates to url, and persists the session once the login handshake's cookies appear (or after wait_seconds, whichever comes first). MCP stdin is the JSON-RPC stream, so there is no Enter keypress; the flow polls for cookies instead.
• fetch(..., profile="portal") / search_fetch(..., profile="portal") - scrape using that session.

agent: "check my portal for new announcements"
  1. list_profiles            -> portal (authenticated)
  2. login(https://portal.example.com, profile="portal")  # if not listed
  3. fetch(url, profile="portal")

Invalid profile names and unknown profiles are hard errors (no silent anonymous fallback). Sessions are local to the machine running the MCP server.

Development

make dev        # install runtime + dev deps
make test       # pytest (pure logic, no network needed)
make lint       # ruff

• Single-file Python (webget_cli.py), no build step, runs via uv run.
• Lazy imports: --strategy http never pays the Crawl4AI import cost.
• Crawl4AI 0.9.2's export_storage_state() is broken (wrong attribute); webget works around it by reaching into browser_manager directly.

Contributing

Found a bug or have an idea? Open an issue - we have templates. Pull requests welcome, see CONTRIBUTING.md.

License

Apache-2.0

Note: dependency licenses are separate (e.g. certifi MPL-2.0, tqdm MPL-2.0 AND MIT, scipy's bundled GCC-runtime GPL-with-exception via the browser extra); each dependency keeps its own license/notice.