googlenewsscraper

Getting Started

Installation

$ pip install GoogleNewsScraper

Reference

Importing

from GoogleNewsScraper import GoogleNewsScraper

Instantiating Scraper

GoogleNewsScraper(driver)

Constructor Parameters

Possible values:

• 'chrome': The driver will default to use this package's chrome driver
• A path to some driver (FireFox, for instance) stored on the user's system

Methods

This method is both public and private, though it really should only be used by the class

locate_html_element(self, driver, element, selector, wait_seconds)

To configure the 'selector' param:

First install selenium

$ pip install selenium

Then import By

from selenium.webdriver.common.by import By

Possible values:

• By.ID
• By.CLASS_NAME
• By.CSS_SELECTOR
• By.LINK_TEXT
• By.NAME
• By.PARTIAL_LINK_TEXT
• By.TAG_NAME
• By.XPATH

GoogleNewsScraper(...args).search(search_text, date_range, pages, pagination_pause_per_page, cb) -> list or None

• Example using 'cb' paramater:

def handle_page_data(page_data: list):
  # Do something with page_data

GoogleNewsScraper(...args).search(...args, cb=handle_page_data)

NOTE:

• If no argument is provided for 'cb,' the scrape method will return a two-dimensional list
• Each list will contain an object of news article data for every news article on that page

Example of the data that every article-object will contain:

• 'id': A unique id for every article data object
• 'description': The preview description of the news article
• 'title': The title of the news article
• 'source': The source of news article (New York Times, for instance)
• 'image_url': The url of the preview news article image
• 'url': A link to the news article
• 'date_time': A datetime string that represents the date of when the article was published