Google Search & AI Overview Scraper

A robust, stealthy Python scraper for Google Search results. Built using undetected-chromedriver and parsel, this tool automatically bypasses Google's bot detection, extracts Google AI Overviews, scrapes clean organic search results (titles, genuine snippets, provider domains, dates), and automatically resolves Google's /goto?url=... encrypted redirect links into their original destination URLs.

---

Table of Contents

• Features
• Prerequisites
• Installation
• Quick Start
• Detailed Parameter Reference
• Response Structure & Schema
• Advanced Usage Examples
  • 1. Scraping Multiple Pages
  • 2. Extracting AI Overview Answers
  • 3. Running in Visible (Non-Headless) Mode
  • 4. Ultra-Fast Scraping (Disabling URL Resolution)
  • 5. Pinning a Specific Chrome Version
• Troubleshooting & FAQs

---

Features

• 🛡️ Anti-Bot Evasion: Employs undetected-chromedriver with native browser fingerprinting to evade Google CAPTCHA blocks.
• 🤖 Google AI Overview Extraction: Captures the generated AI summary answer and cited sources whenever present.
• 🔗 Automatic Redirect Unmasking: Uses concurrent thread pooling to automatically resolve Google's /goto?url=CAES... redirects into full, direct landing page URLs (e.g. https://en.wikipedia.org/...).
• 🧹 Clean Snippet Parsing: Specifically targets Google's snippet elements while stripping inline scripts, image loaders, and feedback modals.
• 🪟 Stealth Headless Mode: Rather than using detectable --headless=new flags that trigger bot detection, runs off-screen (--window-position=-52000,-52000) so it doesn't disturb your desktop.
• 📄 Multi-Page Pagination: Supports scraping across multiple pages (start=0, start=10, etc.) with URL deduplication.

---

Prerequisites

1. Python 3.10+ (Python 3.10, 3.11, or 3.12+).
2. Google Chrome Browser: A standard desktop installation of Google Chrome must be installed on your machine.

---

Installation

1. Clone the Repository

git clone https://github.com/your-username/google-search.git
cd google-search

2. Create and Activate a Virtual Environment

On Windows (PowerShell):

python -m venv .venv
.venv\Scripts\Activate.ps1

On Linux / macOS:

python3 -m venv .venv
source .venv/bin/activate

3. Install Dependencies

pip install -r requirements.txt

Note for Python 3.12+ users: setuptools is included in requirements.txt. It is required because Python 3.12 removed distutils, which undetected-chromedriver relies on for version resolution.

---

Quick Start

Create a script (or run directly from google_search/google_search.py):

from google_search.google_search import get_search

# Run a simple search
response = get_search("Who won the 2022 FIFA World Cup?", pages=1)

# Check if Google provided an AI Overview
if response.get("ai_response"):
    print("🤖 AI Overview:")
    print(response["ai_response"]["text"])
    print("-" * 50)

# Print organic search results
print(f"Found {response['meta']['result_count']} results:\n")
for item in response["results"]:
    print(f"Title:       {item['title']}")
    print(f"URL:         {item['url']}")
    print(f"Provider:    {item.get('provider')}")
    print(f"Description: {item.get('description')}")
    print(f"Date:        {item.get('date')}")
    print("-" * 50)

---

Detailed Parameter Reference

The primary entry point is the function:

def get_search(
    query: str,
    pages: int = 1,
    version: int | None = None,
    headless: bool = True,
    resolve_urls: bool = True
) -> dict:

Parameter Details

Parameter	Type	Default	Description
query	str	(Required)	The search query string (e.g., "Python web scraping", "World Cup 2022 winner"). Special characters and spaces are safely encoded.
pages	int	1	Number of search result pages to scrape. • Set to 1 for the first 10 results. • Set to 2 or higher to paginate (&start=10, &start=20, etc.). • If set to 0 or None, it scrapes until no more results are found.
version	int | None	None	The major version of Google Chrome installed on your machine (e.g., 128, 130, 155). • If None (recommended), undetected-chromedriver automatically detects your installed Chrome version and downloads the matching driver. • Pin this only if auto-detection fails or you have multiple browser versions.
headless	bool	True	Browser visibility mode: • True (default): Spawns the browser pushed off-screen (--window-position=-52000,-52000). It operates invisibly without popping up windows or triggering Google's headless detection CAPTCHAs. • False: Launches Chrome in normal visible mode so you can see navigation, scrolling, and debugging in real time.
resolve_urls	bool	True	Redirect resolution setting: • True (default): Concurrent background threads send HTTP requests to Google's /goto?url=... redirect tokens to extract and return the real landing page URL (e.g., https://en.wikipedia.org/...). • False: Skips redirect resolution and leaves the raw Google redirect URLs. Useful if you want the absolute fastest scraping speed and do not need direct external URLs.

---

Response Structure & Schema

The function returns a Python dictionary with the following schema:

{
    "query": "Who won the 2022 FIFA World Cup?",
    "ai_response": {
        "text": "The Argentina national football team won the 2022 FIFA World Cup...",
        "sources": [
            {
                "title": "Learn more",
                "url": "https://support.google.com/websearch?p=ai_overviews",
                "provider": "support.google.com",
                "description": "AI responses may include mistakes. Learn more",
                "type": "ai_source"
            }
        ]
    },
    "results": [
        {
            "title": "2022 FIFA World Cup final",
            "url": "https://en.wikipedia.org/wiki/2022_FIFA_World_Cup_final",
            "provider": "en.wikipedia.org",
            "description": "They defeated Poland 3–1 in the round of 16, England 2–1 in the quarter-final, and Morocco 2–0 in the semi-final. The final took place in front of 88,966 ...",
            "date": "18 December 2022",
            "type": "organic"
        },
        {
            "title": "FIFA World Cup Qatar 2022™",
            "url": "https://www.fifa.com/en/tournaments/mens/worldcup/qatar2022",
            "provider": "fifa.com",
            "description": "The FIFA World Cup Qatar 2022 was played from 20 November to 18 December 2022. 32 teams competed across 64 matches in the 22nd edition of the tournament.",
            "date": "18 December 2022",
            "type": "organic"
        }
    ],
    "urls": [
        "https://en.wikipedia.org/wiki/2022_FIFA_World_Cup_final",
        "https://www.fifa.com/en/tournaments/mens/worldcup/qatar2022"
    ],
    "meta": {
        "result_count": 7,
        "url_count": 7,
        "has_ai_response": True,
        "pages_scraped": 1
    }
}

Field Descriptions

• query (str): The query string executed.
• ai_response (dict | None): Contains Google's AI Overview answer. If Google did not generate an AI Overview for the query, this is None.
  • text (str): Clean text of the AI Overview answer.
  • sources (list[dict]): List of sources referenced by the AI answer.
• results (list[dict]): Deduplicated list of organic search results:
  • title (str): Title of the web page.
  • url (str): Original destination URL (unmasked from Google redirect).
  • provider (str): Domain or provider name (e.g., en.wikipedia.org, fifa.com).
  • description (str): Clean search snippet/summary from Google (with scripts and UI noise filtered).
  • date (str | None): Publish or update date if Google displayed one.
  • type (str): Type of result ("organic").
• urls (list[str]): Deduplicated list of all external URLs found across the scraped pages.
• meta (dict):
  • result_count (int): Total number of organic results returned.
  • url_count (int): Total count of unique links.
  • has_ai_response (bool): True if an AI overview was detected and parsed.
  • pages_scraped (int): Number of Google result pages successfully processed.

---

Advanced Usage Examples

1. Scraping Multiple Pages

To scrape 3 pages of results (up to ~30 results):

from google_search.google_search import get_search

results = get_search("machine learning tutorials", pages=3)
print(f"Scraped {results['meta']['pages_scraped']} pages.")
print(f"Total results: {results['meta']['result_count']}")

2. Extracting AI Overview Answers

Google often provides generative AI summaries for informative or question queries:

from google_search.google_search import get_search

data = get_search("How does photosynthesis work?", pages=1)

if data["meta"]["has_ai_response"]:
    ai = data["ai_response"]
    print("Answer:\n", ai["text"])
    print("\nCited Sources:")
    for s in ai["sources"]:
        print(f"- {s['title']} ({s['url']})")
else:
    print("No AI Overview generated for this query.")

3. Running in Visible (Non-Headless) Mode

If you are debugging or want to observe Chrome typing and scrolling:

from google_search.google_search import get_search

data = get_search("latest tech news", pages=1, headless=False)

4. Ultra-Fast Scraping (Disabling URL Resolution)

If you only need titles and snippet text and do not require resolved destination URLs, pass resolve_urls=False:

from google_search.google_search import get_search

# Saves ~1 second of network time by skipping HTTP 302 redirect checks
data = get_search("fast query", pages=1, resolve_urls=False)

5. Pinning a Specific Chrome Version

If auto-detection fails or you have a standalone Chrome version installed:

from google_search.google_search import get_search

# Forces undetected-chromedriver to download and match Chrome 130 driver
data = get_search("Python tutorials", version=130)

---

Troubleshooting & FAQs

Q1: ModuleNotFoundError: No module named 'distutils'

• Cause: You are running Python 3.12+ where distutils was removed.
• Fix: Ensure setuptools is installed in your virtual environment:

  pip install setuptools
  

Q2: SessionNotCreatedException: This version of ChromeDriver only supports Chrome version X

• Cause: Your installed Chrome version was updated, but the driver cache is stale or a version was hardcoded.
• Fix: Leave version=None in get_search(...) so it matches automatically. You can also delete the driver cache directory located in %APPDATA%\undetected_chromedriver (Windows) or ~/.local/share/undetected_chromedriver (Linux/Mac).

Q3: OSError: [WinError 6] The handle is invalid on script exit

• Cause: On Windows, when Chrome closes its background communication pipes, Python's garbage collection occasionally warns that the process handle closed early.
• Status: This is harmless and does not affect the scraped data. The script includes a safe deletion hook (_safe_del) to suppress this warning.

Q4: Does this work on cloud servers (AWS / DigitalOcean / Heroku)?

• Note: Google flags datacenter IP ranges very aggressively. If running on a cloud VM, install standard Chrome (google-chrome-stable), xvfb for display emulation, and route traffic through residential proxies to prevent CAPTCHAs.

---

License

This project is licensed under the MIT License.