headless-driver

The fast multi-engine search scraper for Python

Search results from Brave, DuckDuckGo, Yahoo, Mojeek, Google and Bing in about a second — no browser, no API key.
Real browser TLS fingerprints, consensus ranking across engines, honest block detection and per-engine circuit breakers. Plus Selenium and Playwright automation, screenshots, PDF, structured extraction and a colourful CLI.

Documentation · Quick start · Performance · CLI · Engines · Roadmap · PyPI

Why headless-driver

• Fast — engines are fetched over HTTP with a real browser's TLS and HTTP/2 fingerprint (curl_cffi), not by driving Chrome: about a second per query instead of six to nine, megabytes instead of a gigabyte, and thread-safe.
• Gets answers where others get CAPTCHAs — from cloud and datacentre addresses (AWS, ECS, GitHub Actions) a Python TLS stack claiming to be Chrome is refused on sight. headless-driver presents a consistent browser, rotates between Chrome, Edge, Safari and Firefox profiles after a refusal, and asks each engine the way its own front end does.
• Accurate — mode="aggregate" asks one engine per independent index at once and ranks results by how many agree. A LinkedIn profile returned by Brave, DuckDuckGo and Mojeek is far likelier to be the right person.
• Honest — "nobody has an answer" (empty) and "they refused me" (blocked, rate_limited with Retry-After, unparsed) are different results. HTTP 403/429/503 are never reported as "no results", so you never store a false negative.
• Polite by design — a per-engine circuit breaker stands an engine down after refusals and escalates the pause; Chrome is not launched when the address itself is throttled.
• Browser automation when you need it — Selenium or Playwright, stealth, proxies (with credentials under Playwright), screenshots, PDF, declarative extract() and JSON-API capture.

Features

• 11 engines: Brave, DuckDuckGo (HTML POST form, Lite, JS), Yahoo, Mojeek, Google (basic Search-App endpoint and browser), Startpage, Yandex, Bing
• Three browserless transports — impersonate (default when installed), plain http, or a browser — plus Selenium or Playwright for JavaScript engines
• Two search modes — first (walk a fallback chain) and aggregate (parallel fan-out, provider de-duplication, consensus votes, deadline)
• site: is enforced: engines that ignore it are skipped, off-target rows dropped
• Structured SearchResponse: per-engine attempts with status, HTTP code, Retry-After, transport and timing; blocked, cooling, skipped
• Per-engine EngineHealth circuit breaker shared across a process, soft-block detection (verify_empty, probe()), regional results (region="uk-en")
• Clean text: HTML entities, Unicode normalisation, zero-width characters, and click-tracking redirects (DuckDuckGo, Google, Bing, Yahoo) all unwrapped; ads dropped
• ScraperPool, search_batch, JSON/CSV export
• headless-driver CLI: search, engines, doctor, bench, extract, shot, pdf — coloured everywhere, --json for piping
• Coloured, levelled logs — for this package and, on request, for your own logging setup (colorize_logging(), ColorFormatter)
• Tested: 360+ unit tests, ~90% coverage, and an end-to-end suite that scrapes a local site with the installed wheel on Linux, macOS and Windows

Install

pip install "headless-driver[impersonate]"   # recommended: fast browserless search
pip install "headless-driver[playwright]"    # Playwright backend (then: playwright install chromium)
pip install "headless-driver[all]"           # everything
pip install headless-driver                  # Selenium only

Needs Python 3.9+. Search needs no browser at all; browser automation needs an installed Chrome or Chromium (a matching ChromeDriver is downloaded automatically), or Playwright's own build via playwright install chromium.

On Debian/Ubuntu you can install a system browser and driver with:

sudo apt update && sudo apt install -y chromium chromium-driver

Quick start

from headless import Headless

with Headless() as driver:
    driver.get("https://example.com")
    print(driver.title)

Search the web — no browser needed:

from headless import AdvancedSearchScraper

with AdvancedSearchScraper(max_results=5) as scraper:
    response = scraper.search("python headless browser")

    for item in response:                 # iterates like a list
        print(item["title"], item["url"])

    print("answered by", response.engine)
    if response.blocked:                  # every engine refused, not "no results"
        print("blocked:", [str(a) for a in response.refused])

Ask several engines at once and rank by agreement:

with AdvancedSearchScraper(region="uk-en") as scraper:
    response = scraper.search('site:linkedin.com/in "Jane Doe" Credo Capital',
                              mode="aggregate", deadline=8)
    for hit in response:
        print(hit["votes"], hit["engines"], hit["url"])
    # 3 ['brave', 'duckduckgo', 'mojeek'] https://www.linkedin.com/in/jane-doe

Being told you were blocked is the difference between backing off and recording a false negative. Engines CAPTCHA cloud address ranges, so the same code that works on a laptop can get nothing from ECS — response.blocked, response.rate_limited, response.retry_after and response.cooling say which happened.

Search in parallel:

from headless import ScraperPool

with ScraperPool(size=4) as pool:
    for query, response in pool.map(["python asyncio", "python typing"]):
        print(query, response.engine, len(response))

Playwright, with structured extraction:

from headless.playwright_driver import PlaywrightBrowser

with PlaywrightBrowser(block_resources=True) as browser:
    rows = browser.extract("https://news.ycombinator.com",
                           {"title": ".titleline > a", "link": ".titleline > a@href"},
                           item_selector="tr.athing")
    api = browser.capture_json("https://example.com/app", r"/api/")   # read the site's own JSON
    browser.screenshot("https://example.com", "page.png", full_page=True)

Screenshot and PDF with Selenium:

from headless import ExtendedHeadless

hl = ExtendedHeadless(stealth=True)
driver = hl.get_driver()
driver.get("https://example.com")

hl.screenshot("example.png")
hl.save_pdf("example.pdf")
hl.quit()

Performance

Measured 2026-09-29 from a laptop, three queries against Yahoo, then a five-engine aggregate search. Your numbers will differ; the ratios are the point.

The plain-requests row is why impersonation is the default: a Python TLS handshake behind a Chrome User-Agent is refused on sight. Measure it from your own servers with headless-driver bench.

Command line

headless-driver search "python headless browser" -n 5
headless-driver search 'site:linkedin.com/in "jane doe"' --mode aggregate --region uk-en
headless-driver search "selenium stealth" --json | jq -r '.results[].url'
headless-driver search "python" --transport impersonate   # no browser at all
headless-driver engines                    # engines, capabilities, fallback order
headless-driver doctor                     # check chrome, driver, transports, connectivity
headless-driver doctor --engines           # check the engines still parse
headless-driver bench --min-ok-rate 0.5    # which engines answer from this address
headless-driver extract https://example.com -f title=h1 -f link=a@href --json
headless-driver shot https://example.com -o page.png --window 1280x720
headless-driver pdf  https://example.com -o page.pdf --browser playwright

doctor is the quickest way to explain a failing run:

── browser ──────────────────────────────────────────────
 ✓ chrome              Google Chrome 151.0.7922.140
 ✓ chromedriver        ChromeDriver 148.0.7778.179
 ! version match       chrome 151 vs driver 148 - a matching driver will be downloaded
── connectivity ─────────────────────────────────────────
 ✓ html.duckduckgo.com  reachable
 ✓ www.bing.com         reachable
── smoke test ───────────────────────────────────────────
 ✓ launch and navigate  page title 'ok'

 13 passed, 1 warning  +++++++++++++~

Colour switches off automatically when output is not a terminal, so piping gives plain text. NO_COLOR=1 or --no-color disables it; FORCE_COLOR=1 forces it on. Diagnostics go to stderr, keeping --json on stdout clean. Exit status is 0 on success, 1 when nothing could be produced, 2 for a usage error.

See DOCS.md for every command and flag.

Search engines

Searching starts at Brave, which answers from datacentre addresses and honours site: paths. If an engine refuses, has nothing, or stalls, the next is tried — browserless engines first, browser-only ones late, and Bing, which ignores site: paths, last:

brave → duckduckgo → mojeek → yahoo → google_basic → duckduckgo_lite
      → duckduckgo_js → startpage → google → yandex → bing

AdvancedSearchScraper(search_engine="yahoo")                 # start elsewhere
AdvancedSearchScraper(fallback=False)                        # single engine
AdvancedSearchScraper(fallback_engines=["mojeek", "yahoo"])  # custom order
AdvancedSearchScraper(mode="aggregate")                      # fan out by default
scraper.search("query", engine="duckduckgo")                 # force, one-off
scraper.engines_honoring_site()                              # trust these for absence

Each engine declares what it can do, so you do not have to discover it by observation:

scraper.capabilities("yahoo")
# {"js": False, "snippets": True, "honors_site": True, "provider": "bing", "method": "GET", ...}

Add your own engine with register_engine(): a spec can set the HTTP method, form or query builder, cookies, headers, a per-request URL builder, a pre-flight hook, a redirect unwrapper and "no results" markers. Details in DOCS.md.

More

• Full documentation — every class, argument and CLI flag
• Changelog — what changed in 1.1 and 1.0
• Roadmap — what is coming: async API, Puppeteer/Cypress bridges, MCP server, and more
• llms.txt — a compact API summary for LLMs and coding assistants
• Search results — SearchResponse, blocked, per-engine attempts
• Transports — impersonation, plain HTTP, Selenium, Playwright
• Aggregate mode and the circuit breaker
• Playwright — extract(), capture_json(), stealth, tracing
• Deployment — the Dockerfile, and skipping Chrome entirely
• Python API — Headless, ExtendedHeadless, MultiDriverManager, AdvancedSearchScraper, SearchScraper
• Timeouts and troubleshooting
• Running the tests

Support

If this project saves you time, you can support its development:

• Sponsor on GitHub — github.com/nuhmanpk
• Buy me a coffee — buymeacoffee.com/nuhmanpk

Starring the repository helps too.

License

MIT — see LICENSE.

Happy coding 🚀