Skip to main content

News scraping and reporting.

Project description

newsscraper provides a framework for scraping web news with Selenium and Beautiful Soup. newsscraper will take care of remembering which news items where already read and creates results in form of json, csv, or html files.

A minimal scraper for newsscraper that fetches the newest questions from https://stackoverflow.com/questions can look like this:

# content of stackoverflow.py
import newsscraper
import sys

with newsscraper.Scraper(sys.argv) as scraper:
   driver = scraper.get_chrome()
   driver.get('https://stackoverflow.com/questions')
   for question in driver.find_elements_by_xpath('//a[@class="question-hyperlink"]'):
       scraper.add(question.get_attribute('href'), question.text)

Additional configuration can be provided with arguments:

python3 stackoverflow.py --headless --verbose report=html --out=$(date '+%Y-%m-%d %H:%M:%S').html

Run python3 stackoverflow.py -h for a list of all arguments.

Features

  • Remember already added news items

  • Create reports in json, csv, html, or a custom format

  • Merge multiple json reports

  • Custom command-line arguments

Roadmap

  • Proxy support

  • Sort items by date in HTML5 report

  • Tags in HTML5 report

  • Custom report templates

  • RSS reports

  • python2 support

Installation

pip3 install newsscraper

If you want to use the selenium drivers you have to download the corresponding third party drivers in the ./assets/ subdirectory to your script. newsscraper will also automatically load all add-ons you place in ./assets/.

Changes

  • 0.1.0 initial version

License

This project is licensed under the MIT License - see the LICENSE for details.

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

newsscraper-0.1.4.tar.gz (7.0 kB view details)

Uploaded Source

Built Distribution

newsscraper-0.1.4-py3-none-any.whl (8.3 kB view details)

Uploaded Python 3

File details

Details for the file newsscraper-0.1.4.tar.gz.

File metadata

  • Download URL: newsscraper-0.1.4.tar.gz
  • Upload date:
  • Size: 7.0 kB
  • Tags: Source
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/2.0.0 pkginfo/1.5.0.1 requests/2.22.0 setuptools/42.0.1 requests-toolbelt/0.9.1 tqdm/4.39.0 CPython/3.6.9

File hashes

Hashes for newsscraper-0.1.4.tar.gz
Algorithm Hash digest
SHA256 edd5f47eb4e93fe6a770328c61b06a724a6046fb3e8d187ccafb4b4dee6bc50d
MD5 7d4fb349bc47162562a9d7e1cb48a087
BLAKE2b-256 2dbaf89a9372544fe08416c24c00543bc1067e958c7972e1227c3665b7368a47

See more details on using hashes here.

File details

Details for the file newsscraper-0.1.4-py3-none-any.whl.

File metadata

  • Download URL: newsscraper-0.1.4-py3-none-any.whl
  • Upload date:
  • Size: 8.3 kB
  • Tags: Python 3
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/2.0.0 pkginfo/1.5.0.1 requests/2.22.0 setuptools/42.0.1 requests-toolbelt/0.9.1 tqdm/4.39.0 CPython/3.6.9

File hashes

Hashes for newsscraper-0.1.4-py3-none-any.whl
Algorithm Hash digest
SHA256 cba12f61fc2ce78d5f12638c8bb701fb326028826a076bb2676bfeb372ace434
MD5 c6ecfab0e2c3cab2d9d54ba5b9c9cbda
BLAKE2b-256 95efc652f907f5c818973403efdebdf4c1269f06492cf3834b26fa0611550d12

See more details on using hashes here.

Supported by

AWS AWS Cloud computing and Security Sponsor Datadog Datadog Monitoring Fastly Fastly CDN Google Google Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Sentry Sentry Error logging StatusPage StatusPage Status page