webeater 0.2.1

A web content extraction tool designed to fetch and process web pages efficiently

WebEater (weat)

WebEater is a Python library for backend services that need to fetch structured content (Markdown text or a JSON dict) from arbitrary URLs as fast as possible. It exposes a small async API (Webeater.create, engine.get, engine.get_many, engine.shutdown) designed to be built once at service startup and reused across many requests. A weat / webeater CLI ships in the same package for one-shot extraction and interactive use, but the library is the primary target.

Main Features

• Library + CLI in one package — same engine, two surfaces.
• Hybrid HTTP-first renderer with lazy escalation to a JS engine (zendriver default, playwright opt-in, selenium legacy). Static pages skip the browser boot entirely. The default renderer chain (hybrid + zendriver + bs) was selected by the project's pipeline matrix bench (python tests/bench_all.py --preset realistic) on 2026-05-18 as the best-on-average pipeline across the realistic URL pool.
• One supported extractor: WebeaterBeautifulSoup (default). The experimental WebeaterFastBS is deprecated as of 2026-05-24 (D16) and scheduled for removal in v0.3.0 — see "Extractor selection" below.
• Output as Markdown text or a structured JSON dict (title, content, images, links, fetch_time).
• Concurrent batch fetch via Webeater.get_many(urls, max_concurrent=N) (single engine) or WebeaterPool (N engines, true cross-engine parallelism for SPA-heavy workloads).
• Layered hints system for noise removal and main-content selectors.
• Tested on Python 3.9 through 3.12.

Lifecycle and usage patterns

How you use webeater drives which costs dominate. The warm pattern (build the engine once, reuse it across many get() calls) saves the multi-second cost of importing the JS engine (zendriver by default) and booting the browser context per request — this is the recommended shape for backend services. The cold pattern (one full lifecycle per invocation) pays those costs every time and is the natural fit for the CLI or for serverless invocations where state cannot be kept.

See metak-shared/architecture.md ("Lifecycle and usage patterns") for the full breakdown, concurrency notes, and per-request data flow.

Install

pip install webeater

The default JS engine is zendriver, which uses the host's installed Chrome — no separate browser-binary install step. If you want Playwright instead (WeatConfig.js_engine="playwright"), install its browser binaries with:

playwright install chromium

Selenium also remains supported and uses the host's installed Chrome plus Selenium Manager (no extra install step). See "JavaScript engine" below for how to switch engines.

Quick Start (library, recommended pattern)

Build the engine once at process startup. Reuse it across requests. This is the cost model that matters for FastAPI apps, worker processes, daemons, and any long-running service.

import asyncio
from webeater import Webeater


async def main():
    # Build once. Multi-second boot cost paid here.
    engine = await Webeater.create()
    try:
        for url in [
            "https://en.wikipedia.org/wiki/Python_(programming_language)",
            "https://docs.python.org/3/library/asyncio.html",
            "https://news.ycombinator.com/",
        ]:
            result = await engine.get(url, return_dict=True)
            print(result["title"], "—", result["fetch_time"])
    finally:
        # Releases Chromium and the httpx client.
        await engine.shutdown()


asyncio.run(main())

For concurrent batches against a single engine, use engine.get_many(urls, max_concurrent=8) — the HTTP fast path runs in parallel, and the JS engine path is serialised internally so one browser context stays safe. For true cross-engine concurrency (e.g. SPA-heavy workloads where every request escalates to the JS engine), use WebeaterPool — see the "Backend integration" section below.

Quick Start (one-off / CLI use)

For a single URL with no service to keep state in, the cold pattern is fine — but be aware that every invocation pays the full import + browser-boot cost, which usually dwarfs the actual fetch.

import asyncio
from webeater import Webeater


async def main():
    engine = await Webeater.create()
    try:
        print(await engine.get("https://example.com"))
    finally:
        await engine.shutdown()


asyncio.run(main())

For genuine one-off invocations from a shell, the weat / webeater CLI (see next section) is the natural fit — same engine, no Python boilerplate.

Quick Start (CLI)

weat https://example.com

Fetches the page and prints the extracted text to the console.

CLI Options

• url (positional): URL to fetch. If omitted, starts an interactive prompt.
• -c, --config FILE (default: weat.json): Config file.
• --hints FILE [FILE ...]: Additional hint files (space-separated).
• --debug: Enable debug logging.
• --silent: Suppress debug/info; print only result or errors (for scripts).
• --json: Return content as JSON instead of plain text.
• --content-only: Return only the main extracted content (skip images/links).

webeater https://example.com
webeater --json --content-only https://example.com
webeater -c weat.json --hints hints/news.json hints/sports.json https://example.com

Interactive mode (when no URL is provided): enter a URL when prompted. Per-request prefix shortcuts: j!<url> (JSON), c!<url> (content only), jc!<url> or cj!<url> (JSON + content only). q quits. URLs must start with http:// or https://.

Help and Contributions

For questions or discussions, start a Discussion. For bugs or contributions, open an Issue.

Develop with Source

Clone the repository at https://github.com/tiagrib/webeater.git, then install the development dependencies:

pip install -r requirements.txt

Tested on Python 3.9 through 3.12 (daily on 3.12.3).

Configuration and Advanced documentation

WebEater uses a weat.json configuration file (in the current working directory) to manage its settings. The file is read on construction and written back after every load, with defaults omitted to keep it clean.

For detailed documentation on the hints system, see the Hints Documentation.

Backend integration

For daemons and concurrent-request services, see INTEGRATION.md — it covers the warm-pool / WeatService pattern end-to-end (FastAPI, queue consumers, sync bridges, shutdown, telemetry, sizing, and what not to do). The section below is a brief introduction to the lower-level WebeaterPool primitive that WeatService is built on.

For a long-running service (FastAPI / Starlette / aiohttp / a worker daemon), build a WebeaterPool once at startup and tear it down at shutdown. Each pool member holds its own browser context, so requests that escalate to the JS engine run in parallel up to the pool size.

import asyncio
from fastapi import FastAPI
from webeater import WebeaterPool

app = FastAPI()
pool: WebeaterPool | None = None


@app.on_event("startup")
async def startup():
    global pool
    pool = await WebeaterPool.create(size=4)


@app.on_event("shutdown")
async def shutdown():
    if pool is not None:
        await pool.shutdown()


@app.get("/fetch")
async def fetch(url: str):
    assert pool is not None
    return await pool.get(url, return_dict=True)

Pool size should be tuned to expected concurrency. Each pool member holds one browser context, so memory scales linearly with pool size. size=4 is a reasonable default for a single FastAPI worker.

For a single shared engine without per-request parallelism through the JS engine, the simpler Webeater.create() + engine.get() pattern still works — but WebeaterPool is the recommended shape for backend services because the JS-engine escalation path is serialised inside a single engine and saturates quickly under load.

Service: concurrent request dispatch (WeatService)

WeatService (added in v0.2.0) sits one level above WebeaterPool and adds a request queue with per-request callbacks and futures, so you can drive a warm pool of N engines from any transport — FastAPI, a Redis queue consumer, a CLI batch processor, a thread bridge from sync code — without writing your own dispatcher. Each submit(url, ...) returns a ServiceFuture immediately; an optional callback fires when the request completes.

import asyncio
from fastapi import FastAPI, HTTPException
from webeater import WeatService

app = FastAPI()
svc: WeatService | None = None


@app.on_event("startup")
async def _startup() -> None:
    global svc
    svc = await WeatService.create(pool_size=4)


@app.on_event("shutdown")
async def _shutdown() -> None:
    if svc is not None:
        await svc.shutdown(drain=True)


@app.get("/fetch")
async def fetch(url: str) -> dict:
    assert svc is not None
    result = await svc.submit(url)
    if result.error is not None:
        raise HTTPException(502, str(result.error))
    return result.result

Fire-and-forget with a callback (queue-consumer pattern):

from webeater import WeatService, ServiceResult


async def deliver(result: ServiceResult) -> None:
    await my_outbound_queue.put(result)


svc = await WeatService.create(pool_size=16, on_result=deliver, on_error=deliver)

async for url in incoming_urls:
    svc.submit(url)  # no await -- result delivered via callback

Bridging from sync code (thread):

import asyncio
from webeater import WeatService

svc = await WeatService.create(pool_size=4)
loop = asyncio.get_running_loop()


def from_sync_thread(url: str):
    fut = asyncio.run_coroutine_threadsafe(svc.submit_async(url), loop)
    return fut.result(timeout=30)

Sizing: each pool member spawns a Chrome subprocess via the JS engine, so memory scales roughly linearly with pool_size. See tests/bench_service_memory.py (requires pip install webeater[dev] for psutil) for the per-worker RSS sweep against your hardware. The full surface — submit / submit_async / submit_many / shutdown / telemetry properties / ServiceResult fields / callback semantics — is documented in metak-shared/api-contracts/service.md.

Extractor selection

Two content extractors, picked via extractor in weat.json:

• bs (default) — the BeautifulSoup-walking extractor. Stable, ships the production output shape, and is the documented default since 2026-05-24 (D16).
• fastbs — Deprecated (will be removed in v0.3.0). Provided no measurable advantage over the bs extractor on real-world content and can occasionally clip the output or get gated by anti-bot responses. Still functional through v0.2.x; constructing WeatConfig(extractor="fastbs") emits a DeprecationWarning. Users should drop the extractor field from their weat.json (or set extractor="bs") to silence the warning.

{"extractor": "fastbs"}

The default (bs) is omitted from the saved weat.json.

Renderer selection

WebEater ships with four HTML renderers, picked via renderer in weat.json:

• hybrid (default) — HTTP fast path via httpx, then an optional Chrome-impersonating HTTP middle tier (curl_cffi, on by default since T30 — see "Browser-impersonating HTTP middle tier" below), then lazy escalation to a JS-rendering engine when the response looks JS-rendered (empty body, SPA shell markers, SPA-framework shell with thin visible text, anti-bot challenge interstitial, very short visible text, or HTTP errors). Static pages skip the browser boot entirely. The escalation engine is chosen by js_engine (below).
• zendriver — always zendriver (async-first, CDP-direct, undetected-by-default) against the host's installed Chrome. Used as the default JS engine inside hybrid since 2026-05-18 (see "JavaScript engine" below).
• playwright — always headless Chromium via Playwright. Async-native and CDP-direct. Requires playwright install chromium to fetch browser binaries.
• selenium — always headless Chrome via Selenium. Legacy, kept for compatibility; slower than zendriver / Playwright.

{"renderer": "selenium"}

The default (hybrid) is omitted from the saved weat.json.

JavaScript engine

When renderer is hybrid, js_engine picks which engine HybridRenderer escalates to: zendriver (default — async-first, CDP-direct, undetected-by-default, uses the host's Chrome install), playwright (opt-in — async-native, CDP-direct, headless Chromium; requires playwright install chromium), or selenium (legacy).

{"js_engine": "playwright"}

The default (zendriver) is omitted from the saved weat.json. The flip from playwright to zendriver on 2026-05-18 is documented in metak-orchestrator/DECISIONS.md (D9 amendment) and was driven by the realistic-preset run of tests/bench_all.py.

Browser-impersonating HTTP middle tier

When renderer is hybrid, a curl_cffi-backed middle tier between httpx and the JS engine impersonates Chrome's TLS / JA3 fingerprint to bypass sites that 403 plain httpx on TLS-fingerprint detection alone (notably Wikimedia). On by default since T30 (2026-05-18) — the new SPA-framework branch in HybridRenderer._looks_js_rendered detects partially-hydrated SPA responses and escalates them to the JS engine, closing the regression that previously held back this default. Opt out via:

{"use_impersonate_tier": false}

User-Agent override

All renderers impersonate a recent stable Chrome on Windows by default. Override per config ({"user_agent": "MyCrawler/1.0"}) or programmatically via WeatConfig(user_agent="MyCrawler/1.0"). Sites with TLS fingerprinting plus JS challenges (Cloudflare-grade) still require the full-browser escalation path, which the hybrid renderer already does on demand.

Benchmarks

Standalone bench scripts under tests/ — none are part of CI; run manually. The project's success metric is full end-to-end wall-clock for one engine.get(url) request; the full-query benches measure exactly that. The diagnostic benches isolate individual components and are useful for finding hot spots but should not be used to justify default-affecting decisions (see metak-shared/overview.md "What 'fast' means here").

Full-query benches (canonical — represent user-facing latency):

• bench_pipeline_breakdown.py — canonical full-query bench. Real per-iteration render + extract per URL. This is the bench whose numbers represent user-facing latency; run it with --preset realistic for the headline picture.
• bench_end_to_end.py — full-query across a config matrix (v0.1.1 baseline vs current default, etc.).
• bench_all.py — unified entry point that runs the others and prints one summary. Wraps every bench listed in this section.
• bench_renderers_live.py — real side-by-side wall-clock per URL across every renderer option (no simulation). Opt in to running it from bench_all.py via --include-renderer-live.
• bench_warm_vs_cold.py — warm-vs-cold lifecycle cost. Measures Webeater.create + get + shutdown (cold) versus get on a pre-built engine (warm) per renderer config, so you can see how much per-request cost a backend service saves by keeping the engine alive. Opt in to running it from bench_all.py via --include-warm-cold.
• bench_pool.py — pool concurrency. Compares three batched paths against the same URL set: serial (one engine, one URL at a time), Webeater.get_many (one engine, concurrent), and WebeaterPool.get_many (N engines, true cross-engine concurrency). Opt in to running it from bench_all.py via --include-pool.

Diagnostic-only benches

These measure individual components in isolation. Useful for finding hot spots, not for justifying default-affecting decisions — use the full-query benches above for that (see metak-shared/overview.md).

• bench_extractors.py — extractor on a synthetic HTML fixture.
• bench_extractors_live.py — extractor on real HTML, fetched once and replayed in isolation.
• bench_renderers.py — architectural simulation: cost shapes of Selenium / Playwright escalations without launching browsers.