Advanced Keyword Scraper v1.9.2

Google Request Gateway + Reliability Upgrade. Every Google request (SERP and suggestions) goes through one gateway: cache-first, in-flight dedup, <=2 concurrent, global pacing, one total budget, global 429 cooldown, circuit breaker, and a CAPTCHA hard stop. It reduces traffic and stops on abuse signals; it never tries to solve or bypass a CAPTCHA.

keyword-scraper "cotton kurti" --mode research --serp-limit 20 --google-max-requests 100 --debug

Exit codes: 0 ok, 1 error, 3 stopped by Google CAPTCHA/unusual-traffic page (partial results are saved when any exist).

Useful flags: --google-min-interval, --google-jitter, --google-max-requests (TOTAL budget), --google-concurrency (hard-capped at 2), --google-max-attempts, --google-initial-cooldown, --google-max-cooldown, --google-breaker-threshold, --google-cache, --google-cache-ttl-hours, --debug.

Run tests (no pytest needed):

python -m unittest discover -s tests -t . -v

The end-to-end pipeline test needs the normal dependencies (aiohttp, rich); it is skipped otherwise.

Advanced Keyword Scraper v1.8.0

Google SERP safety/persistence upgrade with conservative global pacing, persistent SQLite SERP cache, duplicate suppression, progressive 429 cooldown, CAPTCHA/unusual-traffic stop, Google circuit breaker, per-run request budget, and console telemetry.

This safety layer is designed to reduce request volume and stop on provider abuse signals; it is not an anti-bot bypass mechanism.

Example:

keyword-scraper "cotton kurti" --mode serp --threads 2 --google-min-interval 3 --google-jitter 1.5 --google-max-requests 100

SERP cache defaults to <output>/google_serp_cache.sqlite3 and is valid for 72 hours.

Advanced Keyword Scraper v1.7.0

A local-first keyword research CLI with multi-source discovery, heuristic intelligence, optional SERP analysis, resilient HTTP handling, and export workflows.

Requirements

• Python 3.10 or newer
• Windows, macOS, or Linux

Install

Recommended on Windows:

py -m pip install --upgrade pip
py -m pip install -e .

If py is unavailable, use python instead.

Verify the package

python -m unittest discover -s tests -t . -v

The repository includes a tests/__init__.py bootstrap so tests also work directly from this source-layout checkout before an editable install.

Run keyword research

keyword-scraper "saree" "kurti" --depth 2 --format all

Or without installing the console script:

python -m keyword_scraper.cli "saree" "kurti" --depth 2 --format all

Run SERP research

keyword-scraper "saree" --mode serp --format excel

Run full keyword intelligence research

The research mode combines the existing multi-source discovery engine with optional SERP/PAA enrichment, related searches, local-first keyword clustering, transparent opportunity scoring, and optional provider metrics.

keyword-scraper "cotton kurti" --mode research --depth 2 --serp-limit 20 --format excel

Optional competitor-gap analysis:

keyword-scraper "cotton kurti" --mode research --serp-limit 30 ^
  --target-domain example.com --competitor-domain competitor.com

Optional metrics hydration (no metrics are invented):

keyword-scraper "cotton kurti" --mode research --metrics-file metrics.csv

The metrics CSV can contain:

keyword,search_volume,kd,cpc,competition,competition_level
cotton kurti,12000,28,0.42,0.31,LOW

Research report sheets

The Excel report can include:

• All Keywords
• Opportunity Shortlist
• KPI Summary
• Keyword Clusters
• PAA Questions
• Related Searches
• SERP Analysis
• Competitor Gap
• Intent Summary / n-grams / topic summaries

Opportunity Score is a transparent heuristic. Search volume, KD and CPC are only used when supplied by a provider metrics CSV; the package never fabricates those values.

Workflow modes

These modes create local action-oriented CSV plans from the supplied keywords; they do not claim exact search volume or guaranteed rankings.

keyword-scraper "budget smartphone" --mode youtube --mode-output youtube.csv
keyword-scraper "wireless earbuds" --mode ecommerce --mode-output ecommerce.csv
keyword-scraper "best accounting software" --mode seo --mode-output seo.csv
keyword-scraper "project management software" --mode copywriting --mode-output copywriting.csv

Notes

• SERP providers may change markup, throttle requests, or restrict automated access.
• Opportunity scores are heuristics, not search-engine ranking scores.
• SERP/PAA extraction depends on changing Google markup and may return partial results.
• Clustering is deterministic lexical/intent clustering, not a claim of semantic ML understanding.
• The project does not attempt to bypass provider controls.
• The project does not attempt to bypass provider controls.

Google gateway hardening (v1.9.2)

• Exactly one half-open circuit probe is admitted after breaker cooldown; concurrent probes are blocked.
• Google transport uses a stable per-run User-Agent and does not rotate proxies.
• Proxy rotation, if configured for other providers, is not passed into the Google gateway.
• CAPTCHA/429 remains a hard-stop/cooldown signal; the gateway never changes proxies or User-Agent to retry around it.