Why

Most scraping tools assume one engine. webget assumes the web is messy:

FETCH_AUTO
├── http       → fast path (httpx + trafilatura/html2text), no browser
├── crawl4ai   → Playwright browser, JS rendering, persistent auth sessions
└── firecrawl  → optional cloud fallback (needs WEBGET_FIRECRAWL_KEY)

Every fetch classifies what it hit - success, login_required, challenge, blocked, or error - and reports it in machine-readable JSON. webget never pretends an empty page is success, and it never solves CAPTCHAs or evades anti-bot systems; it tells you honestly what happened.

Install

Requires Python 3.11+.

PyPI (webget-cli)

The CLI command is webget; the PyPI package name is webget-cli (the bare webget name is taken by an unrelated package).

# pip - HTTP fast path + search only (no browser)
pip install webget-cli

# pip - full stack with Crawl4AI/Playwright browser fallback
pip install "webget-cli[browser]"

# uv tool - isolated executable on your PATH
uv tool install webget-cli --with "webget-cli[browser]"

After install, webget is available as a command:

webget --help

Browser runtime (optional)

Crawl4AI drives a Playwright Chromium. pip install "webget-cli[browser]" installs the Python packages; the browser binary itself is downloaded separately:

python -m playwright install chromium

Without the browser extra, webget still works for search and plain HTTP fetches. A fetch that needs the browser (JS rendering, --profile sessions, login) prints a clear warning telling you how to install it.

From source (development)

git clone https://github.com/DavidPandleton/webget
cd webget
uv pip install -e ".[dev,browser]"

Usage

webget s "rust async runtime"             # search DuckDuckGo (top 5)
webget u https://example.com              # scrape (auto: http -> crawl4ai)
webget su "llm inference" 5               # search + scrape top 5, parallel
cat urls.txt | webget u -                 # batch scrape, one browser instance
webget fetch https://example.com --json   # machine-readable result

Long aliases: search = s, fetch = u, search-fetch = su.

Options

Authenticated sessions (profiles)

# interactive login: browser opens, YOU log in manually, session persists
webget login https://campus.example --profile campus

# list profiles and their session status
webget profiles
webget profiles --json

# later fetches reuse the session - even on the HTTP fast path
webget fetch https://campus.example/dashboard --profile campus --json

# log out ONE domain, keep the rest of the profile
webget logout https://campus.example --profile campus

webget login never stores passwords and never fills forms. A visible browser opens, you authenticate yourself, then press Enter in the terminal and webget persists the session. Persistent profiles live in ~/.local/share/webget/profiles/<name>; session cookies are exported to storage_state.json inside the profile after each browser run, so the fast path can reuse them. Secrets are never printed.

JSON output

--json returns a dict keyed by URL, so batch results are easy to inspect:

{
  "https://campus.example/dashboard": {
    "status": "success",
    "method": "crawl4ai",
    "cached": false,
    "attempts": 1,
    "auth": {
      "profile": "campus",
      "authenticated": true,
      "state": "success"
    },
    "error": null
  }
}

Status values: success | login_required | challenge | blocked | error.

Status detection rules

Cache

Results are cached in ~/.cache/webget/ (sha1 of url + profile + options, TTL 1h, eviction at 500 files). The cache is content-level, not strategy-level, and isolated per profile - public, campus, and work fetches never collide. Failures are never cached.

Privacy note: cached content is plaintext JSON on disk. If you fetch authenticated/personal pages, use --no-cache.

Development

make dev        # install runtime + dev deps
make test       # pytest (pure logic, no network needed)
make lint       # ruff

• Single-file Python (webget_cli.py), no build step, runs via uv run.
• Lazy imports: --strategy http never pays the Crawl4AI import cost.
• Crawl4AI 0.9.2's export_storage_state() is broken (wrong attribute); webget works around it by reaching into browser_manager directly.

Contributing

Found a bug or have an idea? Open an issue - we have templates. Pull requests welcome, see CONTRIBUTING.md.

License

MIT