headless-driver

Web search for Python and AI agents

Search the web, read any page as clean Markdown, and give your AI agent both.
Free, fast, no API key.

Documentation · Quick start · Performance · CLI · Engines · Roadmap · PyPI

What you can do

• Search the web from Python — Brave, DuckDuckGo, Yahoo, Mojeek, Google and Bing. No API key, about a second per search, no browser needed.
• Get answers you can trust — ask several engines at once and rank results by how many agree (mode="aggregate").
• Read any page as Markdown — fetch_markdown(url) strips menus, ads and cookie banners, keeps the content, counts tokens and chunks it for RAG.
• Give your AI agent the web — ready-made tools for OpenAI, Anthropic, LangChain, LlamaIndex and CrewAI, and an MCP server for Claude, Cursor and VS Code: headless-driver mcp.
• Know when you're blocked — a refusal is reported as blocked or rate_limited, never as "no results".
• Automate a browser when you need one — Selenium or Playwright, screenshots, PDF, structured extraction.

How it keeps working

• Looks like a real browser. Requests carry a genuine Chrome, Edge, Safari or Firefox TLS and HTTP/2 fingerprint (curl_cffi) — the first thing anti-bot systems check, and why plain Python scrapers are refused from cloud servers. The profile rotates after a refusal.
• Asks each engine the way its own website does — DuckDuckGo's POST form, Yahoo's path tokens, Google's Search-App client, regional cookies.
• Backs off by itself. An engine that refuses is stood down and retried later, with longer pauses if it keeps refusing; Chrome is not launched when the address itself is throttled.
• Caches answers in memory, SQLite or Redis, so pipelines and agents never ask the same thing twice.

Features

• 11 engines: Brave, DuckDuckGo (HTML POST form, Lite, JS), Yahoo, Mojeek, Google (basic Search-App endpoint and browser), Startpage, Yandex, Bing
• Search modes first (fallback chain) and aggregate (parallel fan-out, consensus votes, deadline); pages=N pagination; site: enforced
• fetch_markdown(): main-content extraction, absolute links, code and tables kept, token counts, overlapping chunks with heading paths, automatic JavaScript rendering for app-shell pages
• Agent tools: Toolkit → openai_tools(), anthropic_tools(), langchain_tools(), llamaindex_tools(), crewai_tools(), tool-call handlers
• MCP server: search, search_aggregate, fetch_page, extract, screenshot over stdio or HTTP (MCP SDK 1.x and 2.x)
• Result cache: cache="memory", "sqlite:///path.db", "redis://…"
• Typed results: response.typed() dataclasses, response.to_pydantic()
• Honest SearchResponse: per-engine attempts with status, HTTP code, Retry-After, transport and timing; blocked, cooling, cached, skipped
• Per-engine circuit breaker, soft-block detection, regional results
• Transports: impersonate (default), plain http, Selenium or Playwright
• CLI: search, fetch, mcp, extract, engines, doctor, bench, shot, pdf — coloured output, --json for piping
• Tested: 410+ tests, ~90% coverage, and an end-to-end suite run from the installed wheel on Linux, macOS and Windows
• Documentation site with guides and an API reference

Install

pip install "headless-driver[impersonate]"   # recommended: fast browserless search
pip install "headless-driver[playwright]"    # Playwright backend (then: playwright install chromium)
pip install "headless-driver[all]"           # everything
pip install headless-driver                  # Selenium only

Needs Python 3.9+. Search needs no browser at all; browser automation needs an installed Chrome or Chromium (a matching ChromeDriver is downloaded automatically), or Playwright's own build via playwright install chromium.

On Debian/Ubuntu you can install a system browser and driver with:

sudo apt update && sudo apt install -y chromium chromium-driver

Quick start

from headless import Headless

with Headless() as driver:
    driver.get("https://example.com")
    print(driver.title)

Search the web — no browser needed:

from headless import AdvancedSearchScraper

with AdvancedSearchScraper(max_results=5) as scraper:
    response = scraper.search("python headless browser")

    for item in response:                 # iterates like a list
        print(item["title"], item["url"])

    print("answered by", response.engine)
    if response.blocked:                  # every engine refused, not "no results"
        print("blocked:", [str(a) for a in response.refused])

Read a page as Markdown, ready for an LLM:

from headless import fetch_markdown

doc = fetch_markdown("https://docs.python.org/3/library/asyncio.html", chunk_tokens=500)
print(doc.title, doc.tokens)
for chunk in doc.chunks:
    print(chunk.heading, chunk.tokens)

Give an agent web search and page reading:

from headless import Toolkit

toolkit = Toolkit()
response = client.messages.create(model="claude-sonnet-5", tools=toolkit.anthropic_tools(),
                                  messages=messages, max_tokens=2048)
messages.append({"role": "user", "content": toolkit.handle_anthropic_tool_use(response.content)})
# also: openai_tools() / handle_openai_tool_calls(), langchain_tools(),
#       llamaindex_tools(), crewai_tools()

Or connect Claude Desktop, Claude Code, Cursor or VS Code over MCP:

{"mcpServers": {"web": {"command": "headless-driver", "args": ["mcp"]}}}

Ask several engines at once and rank by agreement:

with AdvancedSearchScraper(region="uk-en") as scraper:
    response = scraper.search('site:linkedin.com/in "Jane Doe" Credo Capital',
                              mode="aggregate", deadline=8)
    for hit in response:
        print(hit["votes"], hit["engines"], hit["url"])
    # 3 ['brave', 'duckduckgo', 'mojeek'] https://www.linkedin.com/in/jane-doe

Being told you were blocked is the difference between backing off and recording a false negative. Engines CAPTCHA cloud address ranges, so the same code that works on a laptop can get nothing from ECS — response.blocked, response.rate_limited, response.retry_after and response.cooling say which happened.

Search in parallel:

from headless import ScraperPool

with ScraperPool(size=4) as pool:
    for query, response in pool.map(["python asyncio", "python typing"]):
        print(query, response.engine, len(response))

Playwright, with structured extraction:

from headless.playwright_driver import PlaywrightBrowser

with PlaywrightBrowser(block_resources=True) as browser:
    rows = browser.extract("https://news.ycombinator.com",
                           {"title": ".titleline > a", "link": ".titleline > a@href"},
                           item_selector="tr.athing")
    api = browser.capture_json("https://example.com/app", r"/api/")   # read the site's own JSON
    browser.screenshot("https://example.com", "page.png", full_page=True)

Screenshot and PDF with Selenium:

from headless import ExtendedHeadless

hl = ExtendedHeadless(stealth=True)
driver = hl.get_driver()
driver.get("https://example.com")

hl.screenshot("example.png")
hl.save_pdf("example.pdf")
hl.quit()

Performance

Measured 2026-09-29 from a laptop, three queries against Yahoo, then a five-engine aggregate search. Your numbers will differ; the ratios are the point.

The plain-requests row is why impersonation is the default: a Python TLS handshake behind a Chrome User-Agent is refused on sight. Measure it from your own servers with headless-driver bench.

Command line

headless-driver search "python headless browser" -n 5
headless-driver search 'site:linkedin.com/in "jane doe"' --mode aggregate --region uk-en
headless-driver search "python web frameworks" --pages 3 --cache memory
headless-driver fetch https://example.com > page.md      # clean Markdown
headless-driver mcp                                      # serve tools to AI agents
headless-driver search "selenium stealth" --json | jq -r '.results[].url'
headless-driver search "python" --transport impersonate   # no browser at all
headless-driver engines                    # engines, capabilities, fallback order
headless-driver doctor                     # check chrome, driver, transports, connectivity
headless-driver doctor --engines           # check the engines still parse
headless-driver bench --min-ok-rate 0.5    # which engines answer from this address
headless-driver extract https://example.com -f title=h1 -f link=a@href --json
headless-driver shot https://example.com -o page.png --window 1280x720
headless-driver pdf  https://example.com -o page.pdf --browser playwright

doctor is the quickest way to explain a failing run:

── browser ──────────────────────────────────────────────
 ✓ chrome              Google Chrome 151.0.7922.140
 ✓ chromedriver        ChromeDriver 148.0.7778.179
 ! version match       chrome 151 vs driver 148 - a matching driver will be downloaded
── connectivity ─────────────────────────────────────────
 ✓ html.duckduckgo.com  reachable
 ✓ www.bing.com         reachable
── smoke test ───────────────────────────────────────────
 ✓ launch and navigate  page title 'ok'

 13 passed, 1 warning  +++++++++++++~

Colour switches off automatically when output is not a terminal, so piping gives plain text. NO_COLOR=1 or --no-color disables it; FORCE_COLOR=1 forces it on. Diagnostics go to stderr, keeping --json on stdout clean. Exit status is 0 on success, 1 when nothing could be produced, 2 for a usage error.

See DOCS.md for every command and flag.

Search engines

Searching starts at Brave, which answers from datacentre addresses and honours site: paths. If an engine refuses, has nothing, or stalls, the next is tried — browserless engines first, browser-only ones late, and Bing, which ignores site: paths, last:

brave → duckduckgo → mojeek → yahoo → google_basic → duckduckgo_lite
      → duckduckgo_js → startpage → google → yandex → bing

AdvancedSearchScraper(search_engine="yahoo")                 # start elsewhere
AdvancedSearchScraper(fallback=False)                        # single engine
AdvancedSearchScraper(fallback_engines=["mojeek", "yahoo"])  # custom order
AdvancedSearchScraper(mode="aggregate")                      # fan out by default
scraper.search("query", engine="duckduckgo")                 # force, one-off
scraper.engines_honoring_site()                              # trust these for absence

Each engine declares what it can do, so you do not have to discover it by observation:

scraper.capabilities("yahoo")
# {"js": False, "snippets": True, "honors_site": True, "provider": "bing", "method": "GET", ...}

Add your own engine with register_engine(): a spec can set the HTTP method, form or query builder, cookies, headers, a per-request URL builder, a pre-flight hook, a redirect unwrapper and "no results" markers. Details in DOCS.md.

More

• Documentation site — guides and API reference
• Full reference — every class, argument and CLI flag
• AI agents and the MCP server
• Reading pages as Markdown and caching
• Changelog — what changed in 1.2, 1.1 and 1.0
• Roadmap — what is coming: async API, Puppeteer/Cypress bridges, proxy pools and more
• llms.txt — a compact API summary for LLMs and coding assistants
• Search results — SearchResponse, blocked, per-engine attempts
• Transports — impersonation, plain HTTP, Selenium, Playwright
• Aggregate mode and the circuit breaker
• Playwright — extract(), capture_json(), stealth, tracing
• Deployment — the Dockerfile, and skipping Chrome entirely
• Python API — Headless, ExtendedHeadless, MultiDriverManager, AdvancedSearchScraper, SearchScraper
• Timeouts and troubleshooting
• Running the tests

Support

If this project saves you time, you can support its development:

• Sponsor on GitHub — github.com/nuhmanpk
• Buy me a coffee — buymeacoffee.com/nuhmanpk

Starring the repository helps too.

License

MIT — see LICENSE.

Happy coding 🚀