headless-driver

Headless Chrome automation and a multi-engine search scraper for Python.
Proxy and stealth support, automatic driver installation, screenshots and PDF export, multi-driver management, and a colourful CLI.

Documentation · Quick start · CLI · Engines · PyPI · Issues

Features

• Headless and windowed Chrome management, with temporary or persistent profiles
• HTTP and SOCKS proxy support, plus optional stealth mode
• Automatic ChromeDriver installation, and recovery when the installed driver does not match your Chrome
• Screenshot and PDF export via Chrome DevTools
• Download folder management
• Multi-driver manager for many isolated browser instances
• Search scraping across 8 engines, starting with DuckDuckGo's no-JavaScript endpoint and falling back automatically when one blocks you
• Bounded page-load timeouts, so a stalled engine fails over instead of hanging
• A headless-driver CLI with coloured output, --json for piping, and a doctor command that diagnoses your setup

Install

pip install headless-driver

Needs Python 3.9+ and an installed Chrome or Chromium. A matching ChromeDriver is downloaded automatically when required.

On Debian/Ubuntu you can install a system browser and driver with:

sudo apt update && sudo apt install -y chromium chromium-driver

Quick start

from headless import Headless

with Headless() as driver:
    driver.get("https://example.com")
    print(driver.title)

Search the web:

from headless import AdvancedSearchScraper

with AdvancedSearchScraper(max_results=5) as scraper:
    for item in scraper.search("python headless browser"):
        print(item["title"], item["url"])
    print("answered by", scraper.last_engine)

Screenshot and PDF:

from headless import ExtendedHeadless

hl = ExtendedHeadless(stealth=True)
driver = hl.get_driver()
driver.get("https://example.com")

hl.screenshot("example.png")
hl.save_pdf("example.pdf")
hl.quit()

Command line

headless-driver search "python headless browser" -n 5
headless-driver search "selenium stealth" --json | jq -r '.results[].url'
headless-driver engines                    # engine list and fallback order
headless-driver doctor                     # check chrome, driver, connectivity
headless-driver shot https://example.com -o page.png --window 1280x720
headless-driver pdf  https://example.com -o page.pdf

doctor is the quickest way to explain a failing run:

── browser ──────────────────────────────────────────────
 ✓ chrome              Google Chrome 151.0.7922.140
 ✓ chromedriver        ChromeDriver 148.0.7778.179
 ! version match       chrome 151 vs driver 148 - a matching driver will be downloaded
── connectivity ─────────────────────────────────────────
 ✓ html.duckduckgo.com  reachable
 ✓ www.bing.com         reachable
── smoke test ───────────────────────────────────────────
 ✓ launch and navigate  page title 'ok'

 13 passed, 1 warning  +++++++++++++~

Colour switches off automatically when output is not a terminal, so piping gives plain text. NO_COLOR=1 or --no-color disables it; FORCE_COLOR=1 forces it on. Diagnostics go to stderr, keeping --json on stdout clean. Exit status is 0 on success, 1 when nothing could be produced, 2 for a usage error.

See DOCS.md for every command and flag.

Search engines

Searching starts at DuckDuckGo's no-JavaScript endpoint, which returns server-rendered HTML and so is much faster than the JavaScript front end. If an engine serves a bot check, returns nothing, or stalls, the next one is tried:

duckduckgo → duckduckgo_lite → bing → mojeek
           → duckduckgo_js → startpage → google → yandex

search() returns [] only once every engine has been tried.

AdvancedSearchScraper(search_engine="bing")                # start elsewhere
AdvancedSearchScraper(fallback=False)                      # single engine
AdvancedSearchScraper(fallback_engines=["bing", "mojeek"])  # custom order
scraper.search("query", engine="bing")                     # force, one-off

Add your own engine with register_engine(). Full details, including the result dict and redirect handling, are in DOCS.md.

More

• Full documentation — every class, argument and CLI flag
• Python API — Headless, ExtendedHeadless, MultiDriverManager, AdvancedSearchScraper, SearchScraper
• Timeouts and troubleshooting
• Running the tests

Support

If this project saves you time, you can support its development:

• Sponsor on GitHub — github.com/nuhmanpk
• Buy me a coffee — buymeacoffee.com/nuhmanpk

Starring the repository helps too.

License

MIT — see LICENSE.

Happy coding 🚀