onyxweb

URL in, fully-rendered HTML out. A Rust + Chromium (CDP) engine with a typed Python API, built for high-throughput recon, scraping, and change detection.

No Node process like Playwright, no WebDriver like Selenium. One install, one process, ~8.5 URL/s.

Installation

uv add onyxweb              # or: pip install onyxweb
uv run onyxweb --install    # one-time: fetch the pinned chrome-headless-shell (~180 MB)

Python 3.10+. Wheels for linux (x86_64, aarch64), macOS (arm64), Windows x64. Anything else builds from source and needs rustup.

Embedding onyxweb in another tool? await onyxweb.aensure_chrome(dest=...) installs the browser wherever you want and returns the path for Client(chrome_path=...); find_chrome() is a no-network "is it installed?" check.

Quickstart

import onyxweb

r    = onyxweb.fetch("https://example.com")       # rendered HTML, post-JS
png  = onyxweb.screenshot("https://example.com")  # png / jpeg / webp
both = onyxweb.fetch_all("https://example.com")   # both, from one page visit

r.title                               # "Example Domain"
r.text                                # visible text; script and style source left out
r.links, r.images, r.scripts          # lazy buckets of records (see Page buckets)

# CSS + BeautifulSoup-style queries, parsed and run in Rust
r.dom.find_all("a", limit=10)

RenderResult is not a str. Pass r.html to regex, lxml, or BS4. str(r), "x" in r, and len(r) still work.

Examples

1) Sweep a lot of URLs

with onyxweb.Client(concurrency=16) as c:
    for r in c.batch(urls, capture="html"):
        if isinstance(r, Exception):   # batch never raises; failures land in place
            continue
        print(r.status_code, r.title)

2) Async, or N threads on one Client

async with onyxweb.AsyncClient(concurrency=16) as ac:
    results = await asyncio.gather(*(ac.fetch(u) for u in urls))

The GIL is released for all Rust work, so a thread pool over one Client runs genuinely parallel.

3) Get past a WAF

from onyxweb.presets.full import stealth

with onyxweb.Client(**stealth.BASIC) as c:      # real Chrome, automation tells stripped
    r = c.fetch("https://www.tesla.com/")

4) Drive the page before capture

r = client.fetch(
    url,
    scripts=[HOOK_JS],                          # runs before any page script
    post_load_scripts=["document.title"],       # returns land in r.post_load_results
    actions=[onyxweb.Click(selector="#login")], # CDP-trusted click / fill / hover / wait
    block_urls=["*://*.tracker.example/*"],
    extra_headers={"Referer": "https://ref.example/"},
)

Per-call settings are reverted before the tab returns to the pool, so nothing leaks between fetches.

Page buckets

A captured page is sorted into 9 lazy buckets: scripts, styles, links, images, iframes, forms, meta, comments, json_ld. Sizing or printing one costs nothing until you read its records.

r.overview(prnt=True)                 # count and size of every bucket, no records built
r.content.scripts                     # inline half: source that lives in the document
r.resources.scripts                   # external half: URLs the page loads
r.resources.all()                     # everything the browser fetches, in document order

r.scripts.search("apiKey")            # records containing a string, matched in Rust
r.scripts.matches(r'"apiKey":"(\w+)"', regex=True)[0].value   # just the captured key

Every URL-bearing record carries url (absolute) and raw (as authored). Search patterns use Rust's regex crate: linear time, no lookaround.

Anti-bot

r.anti_bot is populated on every fetch, whether or not you try to get past anything, so a plain fetch tells you a host sits behind Akamai.

r.anti_bot   # AntiBot(vendor="cloudflare", kind="challenge", resolved=True) or None

bypass_anti_bot=True waits out challenge interstitials and self-heals hard blocks by dropping only the anti-bot cookies, then retrying once. Vendor-agnostic (Akamai, Cloudflare, DataDome, PerimeterX, Imperva).

In testing the full engine cleared Akamai, DataDome, PerimeterX, and Cloudflare on sites like tesla.com and ticketmaster. It does not beat Kasada or an interactive captcha, and says so through .anti_bot instead of handing back a challenge stub.

Two engines

onyxweb.Client(engine="shell")   # default: bundled chrome-headless-shell, fast and light
onyxweb.Client(engine="full")    # real Chrome, --headless=new, beats WAFs the shell can't

Presets

Spread into Client(...). Organized engine-first, since the two engines want opposite recipes.

onyxweb --preset list        # print every preset and exit

The response

Every result carries the whole HTTP response, shaped to match blasthttp so it drops straight into BBOT's HTTP_RESPONSE.

r.metadata.status_code, r.metadata.protocol, r.metadata.remote_ip
r.metadata.redirect_chain, r.metadata.cert_info, r.metadata.body_hashes
r.headers["content-type"], r.headers.cookies, r.headers.raw

Hashes (md5 / mmh3 / sha256) are computed in Rust and match Python's hashlib and mmh3.hash() byte for byte.

CLI

onyxweb https://example.com                  # HTML to stdout
onyxweb https://example.com -o page.html -s shot.png
onyxweb https://example.com --json           # HTML + metadata as JSON
onyxweb --help                               # every config knob is a flag

Configuration

Flat kwargs, a ClientConfig object, or ONYXWEB_* environment variables.

onyxweb.Client(viewport=(1920, 1080), locale="en-GB", proxy="http://user:pass@host:8080")

client.config is a live view: assign at any depth and the next fetch uses it. Launch-only fields (concurrency, chrome options) raise ValueError instead of failing silently. Full field list with docs: python/onyxweb/config.py.

Two knobs worth knowing, both off by default, since outerHTML drops this content:

onyxweb.Client(include_shadow_dom=True)   # web components
onyxweb.Client(include_iframes=True)      # same-origin iframes

Errors

try:
    r = client.fetch(url)
except TimeoutError:            # navigation + CDP timeouts
    ...
except onyxweb.OnyxwebError:    # subclasses RuntimeError; carries .url and .kind
    ...

Development

uv sync                        # venv, deps, Rust extension in editable mode
uv run onyxweb-download-chrome
uv run pytest                  # tests are Python end-to-end; no Rust unit tests, on purpose
uv run pytest -m real_sites    # integration tests against live sites (opt-in)

Editing src/*.rs rebuilds on the next uv run. Set ONYXWEB_LOG=debug for engine logs. Benchmarks and the engine comparison that led here are in BENCHMARKS.md.

License

BSD 3-Clause. The bundled chrome-headless-shell is also BSD-3-Clause (Google's Chrome for Testing).