Skip to main content
This is a pre-production deployment of Warehouse. Changes made here affect the production instance of PyPI (pypi.python.org).
Help us improve Python packaging - Donate today!

Flexible tool for scraping for certain certain DOM elements, and then emailing if new ones are added.

Project Description

alertscraper

alertscraper badge

travis badge

General purpose flexible tool for scraping a given URL for a certain type of items, and then email if new items are added. Useful for monitoring ad or auction websites. Could also be useful for setting up email alerts on your own site.

WARNING

  • Check the Terms of Service of the site before you use this tool! For some sites, using this tool may violate their terms of service, and should not be used.

Limitations

  • This code ONLY scrapes based on the initial HTTP request. Websites that function as single-page apps will not work. This could be supported in the future using JSON, or integrating with something heavier weight like Selenium.

Usage

Installation

Assuming Python’s pip is installed (for Debian-based systems, this can be installed with sudo apt-get install python-pip), alertscraper can be installed directly from PyPI:

pip install alertscraper

Python versions 3.3+ (and 2.6+) are supported and tested against.

Quick start

alertscraper is based on URLs, and maintains a history file for each URL that you scrape so it knows when something is new.

Start by navigating in your web-browser to the website you want to scrape, and then copying and pasting the URL. Then, inspect the page source of the site and see if you can figure out the DOM path to the relevant element. In this case, it was a li element with the class name result so the combined thing becomes li.result.

alertscraper 'https://some-site.org/?query=guitar&maxprice=550' li.result

This will download the given URL and list the text content of each item specified. This lets you know your query is correct.

Now we want to save this to a database file, that is, say that “I’ve seen everything currently posted and am only now interested in new stuff”.

alertscraper 'https://some-site.org/?query=guitar&maxprice=550' li.result --file=guitars.txt

Notice that it prints out again all the links it found. If we were to run the command again, it would not print them out since it will have stored them as “already seen”.

Finally, lets run the command to email us everything that has not yet been seen.

alertscraper 'https://some-site.org/?query=guitar&maxprice=550' li.result --file=guitars.txt --email=myemail@gmail.com

This only runs once. If you want it to run continually, I’d recommend putting it in a cronjob. Eventually I may add a daemon mode, but this is good for now.

Happy scraping!

Contributing

New features, tests, and bug fixes are welcome!

Release History

Release History

This version
History Node

0.1.7

History Node

0.1.6

History Node

0.1.5

History Node

0.1.4

History Node

0.1.3

History Node

0.1.2

Download Files

Download Files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

File Name & Checksum SHA256 Checksum Help Version File Type Upload Date
alertscraper-0.1.7-py2.py3-none-any.whl (8.3 kB) Copy SHA256 Checksum SHA256 3.5 Wheel Nov 7, 2017
alertscraper-0.1.7.tar.gz (20.1 kB) Copy SHA256 Checksum SHA256 Source Nov 7, 2017

Supported By

WebFaction WebFaction Technical Writing Elastic Elastic Search Pingdom Pingdom Monitoring Dyn Dyn DNS Sentry Sentry Error Logging CloudAMQP CloudAMQP RabbitMQ Heroku Heroku PaaS Kabu Creative Kabu Creative UX & Design Fastly Fastly CDN DigiCert DigiCert EV Certificate Rackspace Rackspace Cloud Servers DreamHost DreamHost Log Hosting