Skip to main content

A concurrent flood web scraper.

Project description

Concurrent Flood Scraper

It’s probably exactly what you think it is, based off the name

GET a page. scrape for urls, filter those according to some regex. Put all those in a master queue. Scrape page for any data you want. Repeat…

There’s a small demo in the wikipedia_demo. There you can see how easy it is to set up to fit your web scraping needs!

Specifics

  1. Create a child class of concurrentfloodscraper.Scraper and implement the scrape_page(self, text) method. text is the raw html. In this method you do the specific scraping required. Note that only urls that match the class url_filter_regex will be added to the master queue.
  2. Annotate your Scraper subclass with concurrentfloodscraper.Route. The single parameter is a regex; URL’s that match the regex will be parsed with that scraper.
  3. Repeat steps 1 and 2 for as many different types of pages you expect to be scraping from.
  4. Create an instance of concurrentfloodscraper.ConcurrentFloodScraper, pass it the root URL, the number of threads to use, and a page limit. Page limit defaults to None, which means ‘go forever’.
  5. Start the ConcurrentFloodScraper instance, and enjoy the magic!

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Files for concurrentfloodscraper, version 1.0.1
Filename, size File type Python version Upload date Hashes
Filename, size concurrentfloodscraper-1.0.1-py3-none-any.whl (9.8 kB) File type Wheel Python version py3 Upload date Hashes View

Supported by

AWS AWS Cloud computing Datadog Datadog Monitoring DigiCert DigiCert EV certificate Facebook / Instagram Facebook / Instagram PSF Sponsor Fastly Fastly CDN Google Google Object Storage and Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Salesforce Salesforce PSF Sponsor Sentry Sentry Error logging StatusPage StatusPage Status page