News scraping and reporting.
Project description
newsscraper provides a framework for scraping web news with Selenium and Beautiful Soup. newsscraper will take care of remembering which news items where already read and creates results in form of json, csv, or html files.
A minimal scraper for newsscraper that fetches the newest questions from https://stackoverflow.com/questions can look like this:
# content of stackoverflow.py
import newsscraper
import sys
with newsscraper.Scraper(sys.argv) as scraper:
driver = scraper.get_chrome()
driver.get('https://stackoverflow.com/questions')
for question in driver.find_elements_by_xpath('//a[@class="question-hyperlink"]'):
scraper.add(question.get_attribute('href'), question.text)
Additional configuration can be provided with arguments:
python3 stackoverflow.py --headless --verbose report=html --out=$(date '+%Y-%m-%d %H:%M:%S').html
Run python3 stackoverflow.py -h for a list of all arguments.
Features
Remember already added news items
Create reports in json, csv, html, or a custom format
Merge multiple json reports
Custom command-line arguments
Roadmap
Proxy support
Sort items by date in HTML5 report
Tags in HTML5 report
Custom report templates
RSS reports
python2 support
Installation
pip3 install newsscraper
If you want to use the selenium drivers you have to download the corresponding third party drivers in the ./assets/ subdirectory to your script. newsscraper will also automatically load all add-ons you place in ./assets/.
Changes
0.1.0 initial version
License
This project is licensed under the MIT License - see the LICENSE for details.
Project details
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
File details
Details for the file newsscraper-0.1.4.tar.gz
.
File metadata
- Download URL: newsscraper-0.1.4.tar.gz
- Upload date:
- Size: 7.0 kB
- Tags: Source
- Uploaded using Trusted Publishing? No
- Uploaded via: twine/2.0.0 pkginfo/1.5.0.1 requests/2.22.0 setuptools/42.0.1 requests-toolbelt/0.9.1 tqdm/4.39.0 CPython/3.6.9
File hashes
Algorithm | Hash digest | |
---|---|---|
SHA256 | edd5f47eb4e93fe6a770328c61b06a724a6046fb3e8d187ccafb4b4dee6bc50d |
|
MD5 | 7d4fb349bc47162562a9d7e1cb48a087 |
|
BLAKE2b-256 | 2dbaf89a9372544fe08416c24c00543bc1067e958c7972e1227c3665b7368a47 |
File details
Details for the file newsscraper-0.1.4-py3-none-any.whl
.
File metadata
- Download URL: newsscraper-0.1.4-py3-none-any.whl
- Upload date:
- Size: 8.3 kB
- Tags: Python 3
- Uploaded using Trusted Publishing? No
- Uploaded via: twine/2.0.0 pkginfo/1.5.0.1 requests/2.22.0 setuptools/42.0.1 requests-toolbelt/0.9.1 tqdm/4.39.0 CPython/3.6.9
File hashes
Algorithm | Hash digest | |
---|---|---|
SHA256 | cba12f61fc2ce78d5f12638c8bb701fb326028826a076bb2676bfeb372ace434 |
|
MD5 | c6ecfab0e2c3cab2d9d54ba5b9c9cbda |
|
BLAKE2b-256 | 95efc652f907f5c818973403efdebdf4c1269f06492cf3834b26fa0611550d12 |