fetchquest 0.1.0

The web fetcher that always finds a way

FetchQuest

FetchQuest is a resilient command-line web fetcher that escalates from plain HTTP to browser-like requests and, when needed, a real Playwright browser. It returns readable content instead of leaving the caller to diagnose blocks, JavaScript shells, rendering delays, and vanished pages.

$ fetchquest https://example.com/app --strategy playwright --wait-selector "#results" -v
# https://example.com/app → playwright (status 200)

Why FetchQuest

• Automatic escalation — try lightweight HTTP first, add browser headers when needed, and launch Chromium for JavaScript-rendered pages.
• Content-aware results — preserve raw responses, convert pages to readable text, or extract the main article content without surrounding boilerplate.
• Bounded browser readiness — require load, network-idle, or a visible CSS selector while keeping all waits inside one hard deadline.
• Reliable bulk work — fetch URL lists, continue through individual failures, avoid filename collisions, and write a Markdown report.
• Careful archive recovery — the opt-in Wayback strategy runs only after the original resource returns 404 or 410.
• Agent-friendly — use the CLI directly or install the included Claude Code plugin as a resilient replacement for WebFetch.

Install

Run a tagged release without installing it:

uvx fetchquest https://example.com

For a persistent command:

uv tool install fetchquest
fetchquest https://example.com
fetchquest --version

To run the latest unreleased source directly from main:

uvx --from git+https://github.com/slukyano/fetchquest fetchquest https://example.com

The Playwright strategy needs a one-time Chromium installation:

uvx playwright install chromium

Python 3.14 or newer and uv are required.

Quick start

# Fetch readable text to stdout
fetchquest https://example.com

# Save raw HTML and readable text
fetchquest https://example.com \
  --output pages/ \
  --formats raw,text

# Extract only the main article content
fetchquest https://example.com --formats text-trim

# Require JavaScript-rendered content to appear
fetchquest https://example.com/app \
  --strategy playwright \
  --wait-selector "#results article"

# Fetch a list and recover genuinely missing pages from Wayback
fetchquest --input urls.txt \
  --output pages/ \
  --formats raw,text \
  --strategy smart-cache

Each input-file line contains a URL and an optional output prefix:

https://example.com/pricing pricing
https://example.com/about

How the smart chain works

1. Fetch with plain httpx.
2. If that does not return HTTP 200, retry with browser-like headers.
3. When a successful HTML response is only an empty JavaScript shell, render it with Playwright.
4. When both HTTP strategies fail because of TLS, try Playwright.
5. With smart-cache, consult Wayback only if an original HTTP response was 404 or 410.

Retries apply to transient failures such as HTTP 429, HTTP 503, and network timeouts. The per-URL --timeout remains the hard deadline across strategy attempts, retries, browser startup, required waits, render settling, and capture.

Strategies and formats

Strategy	Behavior
smart	Automatic HTTP, browser-header, and Playwright chain
smart-cache	Smart chain plus Wayback for original 404/410 responses
simple	Plain httpx request
ua-spoof	httpx with browser-like request headers
playwright	Headless Chromium
cache	Wayback Machine only

Format	Behavior
text	Readable Markdown-like text; the default
text-trim	Main content without navigation, footers, or ads
raw	Original response bytes

Run fetchquest --help for all options. The CLI specification documents exact defaults, constraints, output behavior, readiness semantics, and exit codes.

Claude Code plugin

Install FetchQuest as a Claude Code plugin:

claude plugin marketplace add slukyano/fetchquest
claude plugin install fetchquest@fetchquest

The included fetch skill teaches Claude when to use FetchQuest and how to select formats, strategies, and browser-readiness controls.

Development

Setup, validation, network-test, documentation, and pull-request guidance lives in CONTRIBUTING.md. Project documentation starts at docs/index.md, and user-visible changes are tracked in CHANGELOG.md.

License

MIT