ghettobird

A Python framework/tool designed for web scraping. (This readme and the tool itself are still under construction.)

Installation

Use the package manager pip to install foobar.

pip install ghettobird

Goals

The primary goal of this project is to simplify scraping applications into a single dictionary/JSON object (with a few auxilary functions sprinkled in).

This allows us to:

• Reduce boilerplate code
• Increase code readability
• Group fragile pieces of code for easy maintenance
• Reduce inconsistancies in error handling

Usage

Usage examples feature this website. It is a static HTML page, but a JS-heavy sample will be added soon.

from ghettobird import fly

Example One: Grabbing a single element

If we wanted to grab a page header from our sample page and we expect only one element to be returned, we could use the following "flightpath":

itinerary = {
    "url": "http://ghettobird.sample.s3-website.us-east-2.amazonaws.com",
    "flightpath": {
        "header": "//*[@class='page-header']",
    },
}

We would be returned with dictionary that follows the blueprint we laid out, but with the data being populated with:

{'header': 'Jobs in St. Louis, Missouri'}

Example Two: Grabbing a list

If we wanted to grab every single job title from our sample page, the following flightpath would be appropriate. Notice the brackets that surround our xpath. This allows us to return multiple values from elements.

itinerary = {
    "url": "http://ghettobird.sample.s3-website.us-east-2.amazonaws.com",
    "flightpath": {
        "titles": ["//h4[@class='title']"],
    },
}

The result:

{'titles': ['Senior Software Dev',
            'Agile Coach',
            'Software Engineer',
            'Junior Software Dev',
            'Ping Pong Player']}

Example Three: Transformer functions

By default, elements that are found with a given xpath have their text values returned unless specified otherwise. However, if we need to perform some sort of transformation on the element or get an HREF rather than text, "transformer" functions will be necessary.

from ghettobird import fly, transformer