Google Search & AI Overview Scraper (stealth-google-search)

A robust, stealthy Python scraper for Google Search results. Built using undetected-chromedriver and parsel, this tool automatically bypasses Google's bot detection, extracts Google AI Overviews, scrapes clean organic search results (titles, genuine snippets, provider domains, dates), and automatically resolves Google's /goto?url=... encrypted redirect links into their original destination URLs.

Table of Contents

• Features
• Prerequisites
• Installation
• Quick Start
• Detailed Parameter Reference
• Response Structure & Schema
• Advanced Usage Examples
  • 1. Scraping Multiple Pages
  • 2. Extracting AI Overview Answers
  • 3. Running in Visible (Non-Headless) Mode
  • 4. Ultra-Fast Scraping (Disabling URL Resolution)
  • 5. Pinning a Specific Chrome Version
• Troubleshooting & FAQs

Features

• 🛡️ Anti-Bot Evasion: Employs undetected-chromedriver with native browser fingerprinting to evade Google CAPTCHA blocks.
• 🤖 Google AI Overview Extraction: Captures the generated AI summary answer and cited sources whenever present.
• 🔗 Automatic Redirect Unmasking: Uses concurrent thread pooling to automatically resolve Google's /goto?url=CAES... redirects into full, direct landing page URLs (e.g. https://en.wikipedia.org/...).
• 🧹 Clean Snippet Parsing: Specifically targets Google's snippet elements while stripping inline scripts, image loaders, and feedback modals.
• 🪟 Stealth Headless Mode: Rather than using detectable --headless=new flags that trigger bot detection, runs off-screen (--window-position=-52000,-52000) so it doesn't disturb your desktop.
• 📄 Multi-Page Pagination: Supports scraping across multiple pages (start=0, start=10, etc.) with URL deduplication.

Prerequisites

1. Python 3.10+ (Python 3.10, 3.11, or 3.12+).
2. Google Chrome Browser: A standard desktop installation of Google Chrome must be installed on your machine.

Installation

Option A: Install from PyPI (Recommended)

pip install stealth-google-search

Option B: Clone the Repository Directly

git clone https://github.com/SEMoazamALi/stealth-google-search.git
cd stealth-google-search

2. Create and Activate a Virtual Environment

On Windows (PowerShell):

python -m venv .venv
.venv\Scripts\Activate.ps1

On Linux / macOS:

python3 -m venv .venv
source .venv/bin/activate

3. Install Dependencies

pip install -r requirements.txt

Note for Python 3.12+ users: setuptools is included in requirements.txt. It is required because Python 3.12 removed distutils, which undetected-chromedriver relies on for version resolution.

Quick Start

Create a script (or run directly from google_search/google_search.py):

from google_search.google_search import get_search

# Run a simple search
response = get_search("Who won the 2022 FIFA World Cup?", pages=1)

# Check if Google provided an AI Overview
if response.get("ai_response"):
    print("🤖 AI Overview:")
    print(response["ai_response"]["text"])
    print("-" * 50)

# Print organic search results
print(f"Found {response['meta']['result_count']} results:\n")
for item in response["results"]:
    print(f"Title:       {item['title']}")
    print(f"URL:         {item['url']}")
    print(f"Provider:    {item.get('provider')}")
    print(f"Description: {item.get('description')}")
    print(f"Date:        {item.get('date')}")
    print("-" * 50)

Detailed Parameter Reference

The primary entry point is the function:

def get_search(
    query: str,
    pages: int = 1,
    version: int | None = None,
    headless: bool = True,
    resolve_urls: bool = True
) -> dict:

Parameter Details

Response Structure & Schema

The function returns a Python dictionary with the following schema:

{
    "query": "Who won the 2022 FIFA World Cup?",
    "ai_response": {
        "text": "The Argentina national football team won the 2022 FIFA World Cup...",
        "sources": [
            {
                "title": "Learn more",
                "url": "https://support.google.com/websearch?p=ai_overviews",
                "provider": "support.google.com",
                "description": "AI responses may include mistakes. Learn more",
                "type": "ai_source"
            }
        ]
    },
    "results": [
        {
            "title": "2022 FIFA World Cup final",
            "url": "https://en.wikipedia.org/wiki/2022_FIFA_World_Cup_final",
            "provider": "en.wikipedia.org",
            "description": "They defeated Poland 3–1 in the round of 16, England 2–1 in the quarter-final, and Morocco 2–0 in the semi-final. The final took place in front of 88,966 ...",
            "date": "18 December 2022",
            "type": "organic"
        },
        {
            "title": "FIFA World Cup Qatar 2022™",
            "url": "https://www.fifa.com/en/tournaments/mens/worldcup/qatar2022",
            "provider": "fifa.com",
            "description": "The FIFA World Cup Qatar 2022 was played from 20 November to 18 December 2022. 32 teams competed across 64 matches in the 22nd edition of the tournament.",
            "date": "18 December 2022",
            "type": "organic"
        }
    ],
    "urls": [
        "https://en.wikipedia.org/wiki/2022_FIFA_World_Cup_final",
        "https://www.fifa.com/en/tournaments/mens/worldcup/qatar2022"
    ],
    "meta": {
        "result_count": 7,
        "url_count": 7,
        "has_ai_response": True,
        "pages_scraped": 1
    }
}

Field Descriptions

• query (str): The query string executed.
• ai_response (dict | None): Contains Google's AI Overview answer. If Google did not generate an AI Overview for the query, this is None.
  • text (str): Clean text of the AI Overview answer.
  • sources (list[dict]): List of sources referenced by the AI answer.
• results (list[dict]): Deduplicated list of organic search results:
  • title (str): Title of the web page.
  • url (str): Original destination URL (unmasked from Google redirect).
  • provider (str): Domain or provider name (e.g., en.wikipedia.org, fifa.com).
  • description (str): Clean search snippet/summary from Google (with scripts and UI noise filtered).
  • date (str | None): Publish or update date if Google displayed one.
  • type (str): Type of result ("organic").
• urls (list[str]): Deduplicated list of all external URLs found across the scraped pages.
• meta (dict):
  • result_count (int): Total number of organic results returned.
  • url_count (int): Total count of unique links.
  • has_ai_response (bool): True if an AI overview was detected and parsed.
  • pages_scraped (int): Number of Google result pages successfully processed.

Advanced Usage Examples

1. Scraping Multiple Pages

To scrape 3 pages of results (up to ~30 results):

from google_search.google_search import get_search

results = get_search("machine learning tutorials", pages=3)
print(f"Scraped {results['meta']['pages_scraped']} pages.")
print(f"Total results: {results['meta']['result_count']}")

2. Extracting AI Overview Answers

Google often provides generative AI summaries for informative or question queries:

from google_search.google_search import get_search

data = get_search("How does photosynthesis work?", pages=1)

if data["meta"]["has_ai_response"]:
    ai = data["ai_response"]
    print("Answer:\n", ai["text"])
    print("\nCited Sources:")
    for s in ai["sources"]:
        print(f"- {s['title']} ({s['url']})")
else:
    print("No AI Overview generated for this query.")

3. Running in Visible (Non-Headless) Mode

If you are debugging or want to observe Chrome typing and scrolling:

from google_search.google_search import get_search

data = get_search("latest tech news", pages=1, headless=False)

4. Ultra-Fast Scraping (Disabling URL Resolution)

If you only need titles and snippet text and do not require resolved destination URLs, pass resolve_urls=False:

from google_search.google_search import get_search

# Saves ~1 second of network time by skipping HTTP 302 redirect checks
data = get_search("fast query", pages=1, resolve_urls=False)

5. Pinning a Specific Chrome Version

If auto-detection fails or you have a standalone Chrome version installed:

from google_search.google_search import get_search

# Forces undetected-chromedriver to download and match Chrome 130 driver
data = get_search("Python tutorials", version=130)

Troubleshooting & FAQs

Q1: ModuleNotFoundError: No module named 'distutils'

• Cause: You are running Python 3.12+ where distutils was removed.
• Fix: Ensure setuptools is installed in your virtual environment:

  pip install setuptools
  

Q2: SessionNotCreatedException: This version of ChromeDriver only supports Chrome version X

• Cause: Your installed Chrome version was updated, but the driver cache is stale or a version was hardcoded.
• Fix: Leave version=None in get_search(...) so it matches automatically. You can also delete the driver cache directory located in %APPDATA%\undetected_chromedriver (Windows) or ~/.local/share/undetected_chromedriver (Linux/Mac).

Q3: OSError: [WinError 6] The handle is invalid on script exit

• Cause: On Windows, when Chrome closes its background communication pipes, Python's garbage collection occasionally warns that the process handle closed early.
• Status: This is harmless and does not affect the scraped data. The script includes a safe deletion hook (_safe_del) to suppress this warning.

Q4: Does this work on cloud servers (AWS / DigitalOcean / Heroku)?

• Note: Google flags datacenter IP ranges very aggressively. If running on a cloud VM, install standard Chrome (google-chrome-stable), xvfb for display emulation, and route traffic through residential proxies to prevent CAPTCHAs.

License

This project is licensed under the MIT License.