Scrapes Google News article data
Project description
googlenewsscraper
Getting Started
Installation
pip install GoogleNewsScraper
Reference
Importing
from GoogleNewsScraper import GoogleNewsScraper
Instantiating Scraper
GoogleNewsScraper(driver)
Constructor Parameters
Name | Type | Required |
---|---|---|
driver | web driver | no |
Possible values:
'chrome'
: The driver will default to use this package's chrome driver- A path to some driver (FireFox, for instance) stored on the user's system
Methods
This method is both public and private, though it really should only be used by the class
locate_html_element(self, driver, element, selector, wait_seconds)
Name | Type | Required | Description |
---|---|---|---|
driver | web driver | yes | A web driver (Chrome, FireFox, etc) |
element | string | yes | Id or class selector of an HTML element |
selector | Module import | yes | see below |
wait_seconds | int | no | Waits a certain number of seconds in order to locate an HTML element |
To configure the 'selector' param:
First install selenium
pip install selenium
Then import By
from selenium.webdriver.common.by import By
Possible values:
By.ID
By.CLASS_NAME
By.CSS_SELECTOR
By.LINK_TEXT
By.NAME
By.PARTIAL_LINK_TEXT
By.TAG_NAME
By.XPATH
GoogleNewsScraper(...args).search(search_text, date_range, pages, pagination_pause_per_page, cb)
Name | Type | Required | Description |
---|---|---|---|
search_text | str | yes | A series of word(s) that will be inputted into the Google search engine |
date_range | str | no | Filters article by date. Possible values: Past hours, Past 24 hours, Past week, Past month, Past year, Archives |
pages | str or int | no | Number of pages that should be scraped (defaults to 'max') |
pagination_pause_per_page | int | no | Waits a certain amount of seconds before a new page is scraped (defaults to 2). Time may have to be increased if Google prevents you from scraping all pages. |
cb | function | no | Will return all article data on a single page for every page scraped (defaults to False) |
- Example using 'cb' paramater:
def handle_page_data(page_data: list):
# Do something with page_data
GoogleNewsScraper(...args).search(...args, handle_page_data)
NOTE:
- If no argument is provided fro 'cb,' the scrape method will return a two-dimensional list
- Each list will contain an object of news article data for every news article on that page
Example of what type of data that a single article-object will contain:
'description'
: The preview description of the news article'title'
: The title of the news article'source'
: The source of news article (New York Times, for instance)'image_url'
: The url of the preview news article image'article_link'
: A link to the news article'time_published_ago'
: A datetime string that represents the date of when the article was published
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
Close
Hashes for GoogleNewsScraper-0.0.6-py3-none-any.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 705da3d7bd05d5e9065317540a1c5a869e215d42bbf8118963baad767effd906 |
|
MD5 | 781577bdc3ffd4bdc11a2103c70258ed |
|
BLAKE2b-256 | fcdfed63cc690efa230d506923a0211c478af630f941b1fa1541b0c70a8d53e4 |