Why

Most scraping tools assume one engine. webget assumes the web is messy:

FETCH_AUTO
├── http       → fast path (httpx + trafilatura/markdownify), no browser
├── crawl4ai   → Playwright browser, JS rendering, persistent auth sessions
└── firecrawl  → optional cloud fallback (needs WEBGET_FIRECRAWL_KEY)

Every fetch classifies what it hit - success, login_required, challenge, blocked, or error - and reports it in machine-readable JSON. webget never pretends an empty page is success, and it never solves CAPTCHAs or evades anti-bot systems; it tells you honestly what happened.

Install

Requires Python 3.11+.

PyPI (webget-cli)

The CLI command is webget; the PyPI package name is webget-cli (the bare webget name is taken by an unrelated package).

# pip - HTTP fast path + search only (no browser)
pip install webget-cli

# pip - full stack with Crawl4AI/Playwright browser fallback
pip install "webget-cli[browser]"

# uv tool - isolated executable on your PATH
# (single spec with extras: do NOT use `--with "webget-cli[browser]"`,
#  uv re-resolves the same package there and can clobber the main install
#  with a stale version)
uv tool install "webget-cli[browser,mcp]"

After install, webget is available as a command:

webget --help

Browser runtime (optional)

Crawl4AI drives a Playwright Chromium. pip install "webget-cli[browser]" installs the Python packages; the browser binary itself is downloaded separately:

python -m playwright install chromium

Without the browser extra, webget still works for search and plain HTTP fetches. A fetch that needs the browser (JS rendering, --profile sessions, login) prints a clear warning telling you how to install it.

From source (development)

git clone https://github.com/DavidPandleton/webget
cd webget
uv pip install -e ".[dev,browser]"

Usage

webget s "rust async runtime"             # search DuckDuckGo (top 5)
webget u https://example.com              # scrape (auto: http -> crawl4ai)
webget su "llm inference" 5               # search + scrape top 5, parallel
cat urls.txt | webget u -                 # batch scrape, one browser instance
webget fetch https://example.com --json   # machine-readable result

Long aliases: search = s, fetch = u, search-fetch = su.

Options

Authenticated sessions (profiles)

# interactive login: browser opens, YOU log in manually, session persists
webget login https://campus.example --profile campus

# list profiles and their session status
webget profiles
webget profiles --json

# later fetches reuse the session - even on the HTTP fast path
webget fetch https://campus.example/dashboard --profile campus --json

# log out ONE domain, keep the rest of the profile
webget logout https://campus.example --profile campus

webget login never stores passwords and never fills forms. A visible browser opens, you authenticate yourself, then press Enter in the terminal and webget persists the session. Persistent profiles live in ~/.local/share/webget/profiles/<name>; session cookies are exported to storage_state.json inside the profile after each browser run, so the fast path can reuse them. Secrets are never printed.

JSON output

--json returns a dict keyed by URL, so batch results are easy to inspect:

{
  "https://campus.example/dashboard": {
    "status": "success",
    "method": "crawl4ai",
    "cached": false,
    "attempts": 1,
    "auth": {
      "profile": "campus",
      "authenticated": true,
      "state": "success"
    },
    "error": null
  }
}

Status values: success | login_required | challenge | blocked | error.

Status detection rules

Cache

Results are cached in ~/.cache/webget/ (sha1 of url + profile + options, TTL 1h, eviction at 500 files). The cache is content-level, not strategy-level, and isolated per profile - public, campus, and work fetches never collide. Failures are never cached.

Privacy note: cached content is plaintext JSON on disk. If you fetch authenticated/personal pages, use --no-cache.

MCP server

webget_mcp.py exposes the same ladder as an MCP server (search, fetch, search_fetch), so agents like opencode can search and scrape without API keys:

pip install "webget-cli[mcp]"
# or as a uv tool with the MCP server + browser fallback:
uv tool install "webget-cli[browser,mcp]"

Register as a local MCP server in opencode:

{
  "mcp": {
    "webget": {
      "type": "local",
      "command": ["webget-mcp"],
      "enabled": true
    }
  }
}

Then prompt with use webget for search and scrape tasks. Run the server standalone with webget-mcp (stdio transport) or python webget_mcp.py.

Authenticated sessions (profiles)

MCP tools can use locally stored login sessions. Create one first with the CLI, or let the agent create it via the MCP login tool:

webget login https://portal.example.com --profile portal

Then the agent can discover sessions and fetch authenticated pages:

• list_profiles - lists available sessions (name, last used, size, status). Cookie values are never returned.
• login(url, profile) - opens a browser session (headful by default so a human can log in), navigates to url, and persists the session once the login handshake's cookies appear (or after wait_seconds, whichever comes first). MCP stdin is the JSON-RPC stream, so there is no Enter keypress; the flow polls for cookies instead.
• fetch(..., profile="portal") / search_fetch(..., profile="portal") - scrape using that session.

agent: "check my portal for new announcements"
  1. list_profiles            -> portal (authenticated)
  2. login(https://portal.example.com, profile="portal")  # if not listed
  3. fetch(url, profile="portal")

Invalid profile names and unknown profiles are hard errors (no silent anonymous fallback). Sessions are local to the machine running the MCP server.

Development

make dev        # install runtime + dev deps
make test       # pytest (pure logic, no network needed)
make lint       # ruff

• Single-file Python (webget_cli.py), no build step, runs via uv run.
• Lazy imports: --strategy http never pays the Crawl4AI import cost.
• Crawl4AI 0.9.2's export_storage_state() is broken (wrong attribute); webget works around it by reaching into browser_manager directly.

Contributing

Found a bug or have an idea? Open an issue - we have templates. Pull requests welcome, see CONTRIBUTING.md.

License

Apache-2.0

Note: dependency licenses are separate (e.g. certifi MPL-2.0, tqdm MPL-2.0 AND MIT, scipy's bundled GCC-runtime GPL-with-exception via the browser extra); each dependency keeps its own license/notice.