Skip to main content
Join the official 2020 Python Developers SurveyStart the survey!

A concurrent flood web scraper.

Project description

Concurrent Flood Scraper

It’s probably exactly what you think it is, based off the name

GET a page. scrape for urls, filter those according to some regex. Put all those in a master queue. Scrape page for any data you want. Repeat…

There’s a small demo in the wikipedia_demo. There you can see how easy it is to set up to fit your web scraping needs!


  1. Create a child class of concurrentfloodscraper.Scraper and implement the scrape_page(self, text) method. text is the raw html. In this method you do the specific scraping required. Note that only urls that match the class url_filter_regex will be added to the master queue.
  2. Annotate your Scraper subclass with concurrentfloodscraper.Route. The single parameter is a regex; URL’s that match the regex will be parsed with that scraper.
  3. Repeat steps 1 and 2 for as many different types of pages you expect to be scraping from.
  4. Create an instance of concurrentfloodscraper.ConcurrentFloodScraper, pass it the root URL, the number of threads to use, and a page limit. Page limit defaults to None, which means ‘go forever’.
  5. Start the ConcurrentFloodScraper instance, and enjoy the magic!

Project details

Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Files for concurrentfloodscraper, version 1.0.1
Filename, size File type Python version Upload date Hashes
Filename, size concurrentfloodscraper-1.0.1-py3-none-any.whl (9.8 kB) File type Wheel Python version py3 Upload date Hashes View

Supported by

Pingdom Pingdom Monitoring Google Google Object Storage and Download Analytics Sentry Sentry Error logging AWS AWS Cloud computing DataDog DataDog Monitoring Fastly Fastly CDN DigiCert DigiCert EV certificate StatusPage StatusPage Status page