SERP Maven

Self-hosted SERP API alternative: search results from a real Chrome browser and your own residential proxy.

pip install serpmaven. Formerly gserp and google-browser-scraper: both now install serpmaven, and their old commands, imports and GSERP_* / GBS_* variables keep working.

Quickstart · Output · Poisoned results · HTTP API · MCP · Docker · How it works · Options

Scrape Google search results (SERP) with a real Chrome browser and your own residential proxy, and get clean JSON back. Use it as a CLI, a Python library, a local HTTP API, or an MCP server that gives AI agents live Google search.

Plain HTTP is no longer a reliable way to scrape Google, and a fresh browser on a fresh proxy IP is refused quickly. So the scraper warms its first sticky proxy session with ordinary browsing and starts later ones from the cookies Google already accepts, lets the browser pass Google's checks, and then fetches results pages directly over HTTPS with the browser's own cookies. Captchas and blocks are reported as such, never as "no results".

Features

Use cases

• SEO rank tracking and SERP monitoring for your own keywords
• Keyword and competitor research from live Google results
• Web search for AI agents and LLM apps through MCP
• A self-hosted alternative to paid Google search APIs when you already pay for proxies

Install

Python 3.10 or newer, in a virtual environment:

python3 -m venv serpmaven-env
source serpmaven-env/bin/activate
pip install serpmaven
patchright install chromium
serpmaven doctor

In PowerShell:

python -m venv serpmaven-env
Set-ExecutionPolicy -Scope Process Bypass -Force   # lets this window run Activate.ps1
.\serpmaven-env\Scripts\Activate.ps1
pip install serpmaven
patchright install chromium
serpmaven doctor

doctor ends with ready when the browser starts and looks like a desktop.

sudo apt install xvfb
sudo serpmaven-env/bin/patchright install-deps chromium
export SERPMAVEN_BROWSER_ARGS=--enable-unsafe-swiftshader
xvfb-run -a serpmaven doctor

Quickstart

serpmaven search "best running shoes" --proxy "http://USER-session-{session}:PASS@gate.example.com:7000" -o results.jsonl

The first query takes a few minutes while the first proxy IP warms up on a few ordinary pages; the terminal shows each step. Later queries take seconds, and new proxy IPs start from its cookies. Every run ends with the pages served and the traffic used, warm-up apart from the pages.

With NodeMaven

The session id is handled for you and the credentials come from the environment:

pip install "serpmaven[nodemaven]"
export NODEMAVEN_LOGIN=... NODEMAVEN_PASSWORD=...
printf "best running shoes\nhow to boil eggs\n" > queries.txt
serpmaven search -f queries.txt --nodemaven --country us -o results.jsonl

In PowerShell:

pip install "serpmaven[nodemaven]"
$env:NODEMAVEN_LOGIN = "..."; $env:NODEMAVEN_PASSWORD = "..."
Set-Content queries.txt "best running shoes", "how to boil eggs"
serpmaven search -f queries.txt --nodemaven --country us -o results.jsonl

export NODEMAVEN_API_KEY=...
serpmaven search -f queries.txt --nodemaven-demo --traffic-limit-mb 300 -o results.jsonl

serpmaven search "best running shoes" --no-proxy

From Python

from serpmaven import ProxyTemplate, Scraper, Settings

proxy = ProxyTemplate("http://USER-session-{session}:PASS@gate.example.com:7000")
for page in Scraper(proxy, Settings(pages=2)).run(["best running shoes"]):
    for result in page.get("organic_results", []):
        print(result["position"], result["title"], result["link"])

What you get

One JSON object per results page:

{
  "search_metadata": {"status": "Success", "page_verdict": "ok", "fetched_with": "http"},
  "search_parameters": {"engine": "google", "q": "best running shoes", "page": 1},
  "organic_results": [
    {"position": 1, "title": "...", "link": "https://...", "snippet": "..."}
  ],
  "related_questions": [{"question": "..."}],
  "ai_overview": {"text": "..."},
  "exit": {"session": "...", "queries_on_exit": 1, "bytes": "..."}
}

The field names are the ones common to SERP APIs, so simple consumers can often switch with minimal mapping.

Poisoned results protection

Google does not always block a scraper. Sometimes it answers with a normal-looking results page for a different query: no captcha, no error, just wrong data that slips into your pipeline unnoticed.

The scraper checks results pages against the queries it asked. When a proxy IP starts returning unrelated results, it is dropped within a few pages and the query is asked again on a fresh one.

• No extra requests, no extra traffic: the check runs on the page already downloaded
• On by default

Run it as an API

serpmaven serve --nodemaven --country us --prewarm
curl "http://127.0.0.1:8000/search.json?q=best+running+shoes"

In PowerShell, call curl.exe (plain curl there is another command).

• Same record format as above
• hl (language) and gl (country) per request: /search.json?q=...&hl=de&gl=at
• --prewarm finds a proxy IP Google serves at startup, so the first request does not wait for a warm-up; /health answers 503 warming until then
• Results pages are read one at a time on one held proxy IP; a page's links resolve in the background, so the next request starts while the last one waits for its links
• links=N resolves the links of the first N organic results only (0: none)
• links_wait=S answers after S seconds with the links resolved by then; links still being resolved are null and counted in search_metadata.links_pending
• async=true answers at once with search_metadata.id; GET /searches/<id>.json answers "status": "Processing" until the search and its links are done, then the record. The newest 500 searches are kept there, each for up to an hour; up to 50 async searches wait for the browser at once, more are answered 429
• Set SERPMAVEN_API_KEY to require a key; without one the server only listens on localhost

Use it from an AI agent (MCP)

{
  "mcpServers": {
    "google-search": {
      "command": "serpmaven",
      "args": ["mcp", "--nodemaven", "--country", "us", "--prewarm"],
      "env": {"NODEMAVEN_LOGIN": "...", "NODEMAVEN_PASSWORD": "..."}
    }
  }
}

One tool, google_search(query, pages, hl, gl). It returns the organic results, "People also ask", related searches, and the AI Overview and map places when Google shows them.

Docker

A ready image is published to GitHub Container Registry with every release:

docker run --rm ghcr.io/nodemaven/serpmaven doctor
docker run --rm -p 8000:8000 -e SERPMAVEN_API_KEY=change-me \
  -e SERPMAVEN_PROXY='http://USER-session-{session}:PASS@gate.example.com:7000' \
  ghcr.io/nodemaven/serpmaven serve --host 0.0.0.0

To build it yourself: docker build -t serpmaven .

How it works

1. New identity: a sticky proxy session and a fresh browser profile.
2. Warm-up: the first proxy IP opens a few ordinary pages, staying on each for 20-45 seconds. Later ones start from the Google cookies of a served proxy IP and are warmed up only when no such set is accepted.
3. Search: the first query on a proxy IP goes through the browser. Once Google serves it, the next pages come over HTTPS with the browser's cookies, until Google asks for its check again.
4. Check before parsing: a captcha or a block is reported as one, never parsed as an empty result.
5. Keep or drop: a proxy IP Google answers is reused; one it refuses is dropped and the query retried on a new one. If one proxy IP after another is refused, the run stops instead of spending more traffic.

The defaults come from measurements in proxy-benchmark.

Options

Proxies

Any provider with sticky sessions works through --proxy and a {session} placeholder. A proxy without one is accepted with a warning: if the proxy IP changes on every request, warming it does nothing. SOCKS5 needs an IP-whitelisted endpoint, because Chrome cannot send a SOCKS5 username and password.

Check your machine

serpmaven doctor

Starts the browser on a blank page, sends nothing, and reports what a website would see: whether the browser looks headless, whether WebGL works, the screen size. Google answers some machines much less than others, so run this first.

What this is not

Not a free Google Search API, and not a guarantee that every proxy IP will work. It is a browser-based collector that makes failures explicit, drops proxy IPs Google refuses, and tells you what each page cost.

Status

Early. The cloak engine (--engine cloak) is experimental; patchright is the default.

Google changes its pages often. If results come back empty or wrong, run with --save-html DIR and open an issue with what you see.

You are responsible for using this within Google's terms and the law where you run it.

Adding an engine

An engine joins this tool only after it has been measured on proxy-benchmark. Add it there and run the warm-up ladder against Google:

python scripts/run_ladder.py --engines your-engine

Then open a pull request here with the run file. The ladder runs identities with and without the warm-up in the same hours, so the result shows whether the engine actually gets served, not whether it got lucky once.

License

MIT