headless-driver

Headless Chrome automation and a multi-engine search scraper for Python.
Proxy and stealth support, automatic driver installation, screenshots and PDF export, multi-driver management, and a colourful CLI.

Documentation · Quick start · CLI · Engines · PyPI · Issues

Features

• Headless and windowed Chrome management, with temporary or persistent profiles
• HTTP and SOCKS proxy support, plus optional stealth mode
• Automatic ChromeDriver installation, and recovery when the installed driver does not match your Chrome
• Screenshot and PDF export via Chrome DevTools
• Download folder management
• Multi-driver manager for many isolated browser instances
• Search scraping across 8 engines, starting with DuckDuckGo's no-JavaScript endpoint and falling back automatically when one blocks you
• Tells you when you were blocked — "nobody has an answer" and "everybody refused me" are different results, not both an empty list
• Browserless mode — server-rendered engines fetched over HTTP: sub-second instead of seconds, megabytes instead of a gigabyte, and thread-safe
• ScraperPool for genuinely parallel searching, one browser per worker
• Standard logging throughout: silent until your application asks
• Bounded page-load timeouts, so a stalled engine fails over instead of hanging
• A headless-driver CLI with coloured output, --json for piping, and a doctor command that diagnoses your setup
• Colour everywhere it makes sense — macOS, Linux, Windows consoles and CI log viewers — with no extra dependency, degrading to plain ASCII when it does not

Install

pip install headless-driver              # browser only
pip install "headless-driver[http]"      # recommended: adds browserless mode

Needs Python 3.9+ and an installed Chrome or Chromium. A matching ChromeDriver is downloaded automatically when required.

On Debian/Ubuntu you can install a system browser and driver with:

sudo apt update && sudo apt install -y chromium chromium-driver

Quick start

from headless import Headless

with Headless() as driver:
    driver.get("https://example.com")
    print(driver.title)

Search the web — no browser needed for most engines:

from headless import AdvancedSearchScraper

with AdvancedSearchScraper(max_results=5) as scraper:
    response = scraper.search("python headless browser")

    for item in response:                 # iterates like a list
        print(item["title"], item["url"])

    print("answered by", response.engine)
    if response.blocked:                  # every engine refused, not "no results"
        print("blocked:", [a.reason for a in response.refused])

Being told you were blocked is the difference between backing off and recording a false negative. Engines CAPTCHA cloud address ranges, so the same code that works on a laptop returns nothing from ECS — response.blocked says which happened.

Search in parallel:

from headless import ScraperPool

with ScraperPool(size=4) as pool:
    for query, response in pool.map(["python asyncio", "python typing"]):
        print(query, response.engine, len(response))

Screenshot and PDF:

from headless import ExtendedHeadless

hl = ExtendedHeadless(stealth=True)
driver = hl.get_driver()
driver.get("https://example.com")

hl.screenshot("example.png")
hl.save_pdf("example.pdf")
hl.quit()

Command line

headless-driver search "python headless browser" -n 5
headless-driver search "selenium stealth" --json | jq -r '.results[].url'
headless-driver search "python" --transport http   # no browser at all
headless-driver engines                    # engines, capabilities, fallback order
headless-driver doctor                     # check chrome, driver, connectivity
headless-driver doctor --engines           # check the engines still parse
headless-driver shot https://example.com -o page.png --window 1280x720
headless-driver pdf  https://example.com -o page.pdf

doctor is the quickest way to explain a failing run:

── browser ──────────────────────────────────────────────
 ✓ chrome              Google Chrome 151.0.7922.140
 ✓ chromedriver        ChromeDriver 148.0.7778.179
 ! version match       chrome 151 vs driver 148 - a matching driver will be downloaded
── connectivity ─────────────────────────────────────────
 ✓ html.duckduckgo.com  reachable
 ✓ www.bing.com         reachable
── smoke test ───────────────────────────────────────────
 ✓ launch and navigate  page title 'ok'

 13 passed, 1 warning  +++++++++++++~

Colour switches off automatically when output is not a terminal, so piping gives plain text. NO_COLOR=1 or --no-color disables it; FORCE_COLOR=1 forces it on. Diagnostics go to stderr, keeping --json on stdout clean. Exit status is 0 on success, 1 when nothing could be produced, 2 for a usage error.

See DOCS.md for every command and flag.

Search engines

Searching starts at DuckDuckGo's no-JavaScript endpoint, which returns server-rendered HTML and so is much faster than the JavaScript front end. If an engine serves a bot check, returns nothing, or stalls, the next one is tried:

duckduckgo → duckduckgo_lite → bing → mojeek
           → duckduckgo_js → startpage → google → yandex

search() returns [] only once every engine has been tried.

AdvancedSearchScraper(search_engine="bing")                # start elsewhere
AdvancedSearchScraper(fallback=False)                      # single engine
AdvancedSearchScraper(fallback_engines=["bing", "mojeek"])  # custom order
scraper.search("query", engine="bing")                     # force, one-off

Add your own engine with register_engine(). Full details, including the result dict and redirect handling, are in DOCS.md.

Each engine declares what it can do — whether it needs a browser (js) and whether it returns description text (snippets) — so you do not have to discover it by observation:

scraper.capabilities("duckduckgo_lite")
# {"js": False, "snippets": False, ...}   -> fast, but match on titles

More

• Full documentation — every class, argument and CLI flag
• Changelog — what changed in 1.0, and the two breaking changes
• Search results — SearchResponse, blocked, per-engine attempts
• Transports — running without Chrome
• Deployment — the Dockerfile, and skipping Chrome entirely
• Python API — Headless, ExtendedHeadless, MultiDriverManager, AdvancedSearchScraper, SearchScraper
• Timeouts and troubleshooting
• Running the tests

Support

If this project saves you time, you can support its development:

• Sponsor on GitHub — github.com/nuhmanpk
• Buy me a coffee — buymeacoffee.com/nuhmanpk

Starring the repository helps too.

License

MIT — see LICENSE.

Happy coding 🚀