Advanced Keyword Scraper v1.9.6

Multi-source SEO / Keyword Intelligence

Version 1.9.6 restores, hardens, and strengthens the multi-source workflow: Google, YouTube Creator suggestions, Bing and Amazon autocomplete can be queried individually or together. Google/YouTube traffic remains behind the safety gateway; Bing/Amazon use their public suggestion endpoints and do not bypass access controls.

New in 1.9.6

• --engine all for Google + YouTube + Bing + Amazon in one keyword run.
• --engines google,youtube,bing,amazon for explicit source selection.
• Source-aware deduplication: the report keeps a combined Source field when the same keyword appears across providers.
• Expanded modifier discovery: geo/local, comparison, content and buyer-intent patterns with --include-modifiers.
• --min-words / --max-words keyword-length filtering.
• --content-brief adds recommended format, title angle and content angle.
• Research workbook now includes Source Coverage.
• More tolerant parsing for Bing OSJSON and Amazon completion payloads.
• Version metadata, CLI output, changelog, and package documentation are synchronized to 1.9.6.

Example:

keyword-scraper "cotton kurti" --engine all --depth 2 --include-modifiers --content-brief --format excel

For only the YouTube creator workflow:

keyword-scraper "cotton kurti" --engine youtube --depth 2

Google SERP safety/persistence upgrade with conservative global pacing, persistent SQLite SERP cache, duplicate suppression, progressive 429 cooldown, CAPTCHA/unusual-traffic stop, Google circuit breaker, per-run request budget, and console telemetry.

This safety layer is designed to reduce request volume and stop on provider abuse signals; it is not an anti-bot bypass mechanism.

Example:

keyword-scraper "cotton kurti" --mode serp --threads 2 --google-min-interval 3 --google-jitter 1.5 --google-max-requests 100

SERP cache defaults to <output>/google_serp_cache.sqlite3 and is valid for 72 hours.

v1.9.6 hardening

• Multi-source aggregation preserves Google, YouTube, Bing, and Amazon provenance for duplicate keywords.
• Source Count and per-provider Found flags are included in keyword outputs.
• Research Opportunity Score uses the observed source count instead of a fixed value.
• Depth-2 expansion preserves provider lineage.
• YouTube Creator and Amazon buyer-intent intelligence are included when their modes are selected.
• Non-Google retry routing is provider-aware: Bing/Amazon requests use their own provider identity when selecting retry/throttling delays. Failed Bing/Amazon responses are not cached as empty results.
• Regression coverage includes multi-provider integration, depth-2 lineage, provider-specific retry routing, export schema, and version consistency.

Historical baseline: v1.7.0

A local-first keyword research CLI with multi-source discovery, heuristic intelligence, optional SERP analysis, resilient HTTP handling, and export workflows.

Requirements

• Python 3.10 or newer
• Windows, macOS, or Linux

Install

Recommended on Windows:

py -m pip install --upgrade pip
py -m pip install -e .

If py is unavailable, use python instead.

Verify the package

python -m unittest discover -s tests -t . -v

The repository includes a tests/__init__.py bootstrap so tests also work directly from this source-layout checkout before an editable install.

Run keyword research

keyword-scraper "saree" "kurti" --depth 2 --format all

Or without installing the console script:

python -m keyword_scraper.cli "saree" "kurti" --depth 2 --format all

Run SERP research

keyword-scraper "saree" --mode serp --format excel

Run full keyword intelligence research

The research mode combines the existing multi-source discovery engine with optional SERP/PAA enrichment, related searches, local-first keyword clustering, transparent opportunity scoring, and optional provider metrics.

keyword-scraper "cotton kurti" --mode research --depth 2 --serp-limit 20 --format excel

Optional competitor-gap analysis:

keyword-scraper "cotton kurti" --mode research --serp-limit 30 ^
  --target-domain example.com --competitor-domain competitor.com

Optional metrics hydration (no metrics are invented):

keyword-scraper "cotton kurti" --mode research --metrics-file metrics.csv

The metrics CSV can contain:

keyword,search_volume,kd,cpc,competition,competition_level
cotton kurti,12000,28,0.42,0.31,LOW

Research report sheets

The Excel report can include:

• All Keywords
• Opportunity Shortlist
• KPI Summary
• Keyword Clusters
• PAA Questions
• Related Searches
• SERP Analysis
• Competitor Gap
• Intent Summary / n-grams / topic summaries

Opportunity Score is a transparent heuristic. Search volume, KD and CPC are only used when supplied by a provider metrics CSV; the package never fabricates those values.

Workflow modes

These modes create local action-oriented CSV plans from the supplied keywords; they do not claim exact search volume or guaranteed rankings.

keyword-scraper "budget smartphone" --mode youtube --mode-output youtube.csv
keyword-scraper "wireless earbuds" --mode ecommerce --mode-output ecommerce.csv
keyword-scraper "best accounting software" --mode seo --mode-output seo.csv
keyword-scraper "project management software" --mode copywriting --mode-output copywriting.csv

Notes

• SERP providers may change markup, throttle requests, or restrict automated access.
• Opportunity scores are heuristics, not search-engine ranking scores.
• SERP/PAA extraction depends on changing Google markup and may return partial results.
• Clustering is deterministic lexical/intent clustering, not a claim of semantic ML understanding.
• The project does not attempt to bypass provider controls.
• The project does not attempt to bypass provider controls.

Google gateway hardening (v1.9.2)

• Exactly one half-open circuit probe is admitted after breaker cooldown; concurrent probes are blocked.
• Google transport uses a stable per-run User-Agent and does not rotate proxies.
• Proxy rotation, if configured for other providers, is not passed into the Google gateway.
• CAPTCHA/429 remains a hard-stop/cooldown signal; the gateway never changes proxies or User-Agent to retry around it.

Multi-source intelligence (v1.9.6)

Keyword output now preserves provider provenance. Duplicate keywords discovered by multiple sources are merged into one row with Source Count, Google Found, YouTube Found, Bing Found, Amazon Found, and Cross-Source Signal fields. Research opportunity scoring uses the actual source count when available.

YouTube workflow output includes creator opportunity and search-intent fields. Ecommerce workflow output includes buyer intent and product-attribute angles.