Skip to main content
Join the official 2020 Python Developers SurveyStart the survey!

Easy multythread web scraper

Project description

MulTithreaded SCRAPER

Hello, welcome you here. This is the mt_scraper library documentation for python version 3.


This is a project of a multithreaded site scraper. Multithreading operation speeds up data collection from Web several times (more than 10 on a normal old work laptop). To use it, you need to redefine the parse method for your needs and enjoy the benefits of multithreading (with all its implementation in Python)

Collecting data in the JSON file, which stores a list of objects (dictionaries) with the collected data.


Simple application

Main Library Usage Scenario

import mt_scraper

scraper = mt_scraper.Scraper () ()

As you can see there are only three lines of code

What happens when this happens

With this application, you get a data scraper from the pages of the list:

url_components_list = [

The last two pages were added to demonstrate the two most common errors when retrieving data from the Internet, these are HTTP 404 - Not Found, and the URL Error: Name: or service not known.

The real URL is obtained by substituting this list into a template:

url_template = '{}'

Data is accumulated in the file:

out_filename = 'out.json'

The work is conducted in 5 threads and a task queue of 5 units is created (this has a value, for example, when canceling an operation from the keyboard, the queue length indicates how many tasks were sent for execution):

threads_num = 5
queue_len = 5

The following is used as a parser function:

def parse (self, num, url_component, html):
    '''You must override this method.
    Must return a dictionary or None if parsing the page
    parser = MyDummyHTMLParser ()
    parser.feed (html)
    obj = parser.obj
    obj ['url_component'] = url_component
    return parser.obj

DummyParser is a simple version of HTML parser, it is remarkable only because it uses only one standard library and does not require any additional modules. File

from html.parser import HTMLParser

class MyDummyHTMLParser (HTMLParser):
    def __init __ (self):
        super () .__ init __ ()
        self.a_tag = False
        self.h1_tag = False
        self.p_tag = False
        self.obj = {}

    def handle_starttag (self, tag, attrs):
        if tag == 'h1':
            self.h1_tag = True
        elif tag == 'p':
            self.p_tag = True
        elif tag == 'a':
            self.a_tag = True
            for (attr, value,) in attrs:
                if attr == 'href':
                    self.obj ['link'] = value

    def handle_endtag (self, tag):
        if tag == 'h1':
            self.h1_tag = False
        elif tag == 'p':
            self.p_tag = False
        elif tag == 'a':
            self.a_tag = False

    def handle_data (self, data):
        if self.h1_tag:
            self.obj ['header'] = data
        elif self.p_tag and not self.a_tag:
            self.obj ['article'] = data

This approach is used only to demonstrate the capabilities of multithreading, in real projects it is recommended to use the lxml or BS libraries, a more advanced application will be shown below in the section "Advanced Application"

Project details

Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Files for mt-scraper, version 0.4.0
Filename, size File type Python version Upload date Hashes
Filename, size mt_scraper-0.4.0-py3-none-any.whl (7.5 kB) File type Wheel Python version py3 Upload date Hashes View
Filename, size mt_scraper-0.4.0.tar.gz (6.9 kB) File type Source Python version None Upload date Hashes View

Supported by

Pingdom Pingdom Monitoring Google Google Object Storage and Download Analytics Sentry Sentry Error logging AWS AWS Cloud computing DataDog DataDog Monitoring Fastly Fastly CDN DigiCert DigiCert EV certificate StatusPage StatusPage Status page