Skip to main content

News scraping and reporting.

Project description

newsscraper provides a framework for scraping web news with Selenium and Beautiful Soup. newsscraper will take care of remembering which news items where already read and creates results in form of json, csv, or html files.

A minimal scraper for newsscraper that fetches the newest questions from https://stackoverflow.com/questions can look like this:

# content of stackoverflow.py
import newsscraper
import sys

with newsscraper.Scraper(sys.argv) as scraper:
   driver = scraper.get_chrome()
   driver.get('https://stackoverflow.com/questions')
   for question in driver.find_elements_by_xpath('//a[@class="question-hyperlink"]'):
       scraper.add(question.get_attribute('href'), question.text)

Additional configuration can be provided with arguments:

python3 stackoverflow.py --headless --verbose report=html --out=$(date '+%Y-%m-%d %H:%M:%S').html

Run python3 stackoverflow.py -h for a list of all arguments.

Features

  • Remember already added news items
  • Create reports in json, csv, html, or a custom format
  • Merge multiple json reports
  • Custom command-line arguments

Roadmap

  • Proxy support
  • Sort items by date in HTML5 report
  • Tags in HTML5 report
  • Custom report templates
  • RSS reports
  • python2 support

Installation

pip3 install newsscraper

If you want to use the selenium drivers you have to download the corresponding third party drivers in the ./assets/ subdirectory to your script. newsscraper will also automatically load all add-ons you place in ./assets/.

Changes

  • 0.1.0 initial version

License

This project is licensed under the MIT License - see the LICENSE for details.

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Files for newsscraper, version 0.1.4
Filename, size File type Python version Upload date Hashes
Filename, size newsscraper-0.1.4-py3-none-any.whl (8.3 kB) File type Wheel Python version py3 Upload date Hashes View hashes
Filename, size newsscraper-0.1.4.tar.gz (7.0 kB) File type Source Python version None Upload date Hashes View hashes

Supported by

Elastic Elastic Search Pingdom Pingdom Monitoring Google Google BigQuery Sentry Sentry Error logging AWS AWS Cloud computing DataDog DataDog Monitoring Fastly Fastly CDN SignalFx SignalFx Supporter DigiCert DigiCert EV certificate StatusPage StatusPage Status page