kazparserbot 0.1.2

Keyword-driven web scraping pipeline (Serper + OpenAI).

kazparserbot

Keyword-driven scraping pipeline using Serper (search + scrape) and OpenAI.

Install

pip install kazparserbot

Required environment variables

Put these in your environment or in a local .env file (see .env.template):

• SERPER_API_KEY (required)
• OPENAI_API_KEY (recommended)

Library usage (recommended)

Basic usage (sync)

from kazparserbot import KazParserBot

bot = KazParserBot.from_env()

result = bot.scrape_keywords_sync(
    ["keyword1", "keyword2"],
    google_results_to_get_per_query=10,
    top_results_to_get=5,
    queries_to_generate_per_keyword=3,
)

With image retrieval enabled

This downloads images to an imgs/ folder in the current working directory and returns image metadata under the __imgs__ key.

from kazparserbot import KazParserBot

bot = KazParserBot.from_env()

result = bot.scrape_keywords_sync(
    ["keyword1", "keyword2"],
    collect_imgs_and_context=True,
)

imgs = result.get("__imgs__", [])

Public API

kazparserbot.KazParserBot

Create an instance:

• KazParserBot.from_env(...)
  • dotenv_path: path to a .env file (optional)
  • load_dotenv_first: whether to load .env before reading env vars (default True)
  • gpt_concurrency: max in-flight OpenAI requests (default 100)
  • serper_concurrency: max in-flight Serper requests (default 100)
  • http_timeout_s: per-request HTTP timeout in seconds for Serper + image fetching (default 30.0)

Core methods:

• await bot.scrape_keywords(keywords, *, google_results_to_get_per_query=10, top_results_to_get=5, queries_to_generate_per_keyword=3, collect_imgs_and_context=False)

  • keywords: list of keywords to process
  • google_results_to_get_per_query: how many Serper search results to fetch per query
  • top_results_to_get: how many results to keep per language (RU + KK); internally the model chooses top_results_to_get * 2
  • queries_to_generate_per_keyword: number of RU queries and KK queries to generate per keyword
  • collect_imgs_and_context: if True, downloads images and extracts nearby text context
  • Returns dict[str, list[dict]]. When image retrieval is enabled, images are stored under __imgs__ as a list of dicts containing:
    • url, img_url, context_text_before, context_text_after, file_path

• bot.scrape_keywords_sync(keywords, **kwargs)

  • Synchronous wrapper around scrape_keywords(...) (same kwargs)

• bot.scrape_from_files(*, keywords_json, output_json, google_results_to_get_per_query=10, top_results_to_get=5, queries_to_generate_per_keyword=3, collect_imgs_and_context=False)

  • keywords_json: path to a JSON file containing a list of keywords
  • output_json: path to write results JSON

kazparserbot.Settings

• Settings.from_env(dotenv_path=None, load_dotenv_first=True)
  • Validates SERPER_API_KEY is present and reads OPENAI_API_KEY if set.

CLI usage

After installing:

kazparserbot keywords.json output.json \
  --google_results_to_get_per_query=10 \
  --top_results_to_get=5 \
  --queries_to_generate_per_keyword=3 \
  --collect_imgs_and_context=True

The legacy entrypoint is still available:

python scrap_by_keywords.py keywords.json output.json --collect_imgs_and_context=True