DigitalSpot 0.0.1

The beginnings of a custom web crawler

This software is in Alpha!!!

Sister project to Rover

For some reason, Mozilla decided not to be helpful in hiding the WebDriver flag. Notice the deleted comment. So, we are using Chromium

RPi

sudo apt-get update
sudo apt-get install python-pip chromium-browser chromium-chromedriver xvfb
pip3 install -r requirements.txt
python3 main.py

Sites To Consider Indexing

• https://apnews.com/
• https://www.reuters.com/
• https://www.foxnews.com/
• https://www.nbcnews.com/
• https://us.cnn.com/
• https://www.nytimes.com/
• https://www.huffpost.com/
• https://www.usatoday.com/
• https://www.politico.com/
• https://www.npr.org/sections/news/
• https://thehill.com/
• https://www.texastribune.org/
• https://hillreporter.com/
• https://www.rawstory.com/
• https://www.washingtonpost.com/
• https://www.thegatewaypundit.com/
• https://www.oann.com/