License		Version
Travis CI		Coverage
Wheel		Implementation
Status		Downloads
Supported versions

ezscrape

ezscrape provides a boilerplate for simple scraping tasks.

It provide generic access to scraping functionality without exposing the user directly the the underlying libraries used (e.g. requests, selenium) when using the scraping functionality. The used scraper is chosen based on the specified config parameters and will prefer the most flexibe / least resource intensive ones if possible.

The exceptions of the underlying modules are handled and converted into the status of the result object.

Setup Requirements

Setup Chrome and Webdriver

For some websites selenium will be used for scraping (e.g. requests doesn't support javascript)

For that purpose the following environment variables need to be set otherwise an exception will be raised when the code is using selenium.

1. The Chrome Executable

   CHROME_EXEC_PATH

2. The Chrome Webdriver Executable

   CHROME_WEBDRIVER_PATH

Usage

1. The basic concept of a simple scrape is
2. Create the Scrape Config with a Url
3. Optional - set additional parameters on the scrape config
4. Scrape with the given Config
5. Check the Return Object if the Scrape was succesfull
6. Get the HTML from the Return Object

Scrape a simple HTML Page

import ezscrape.scraping.scraper as scraper
from ezscrape.scraping.core import ScrapeConfig
from ezscrape.scraping.core import ScrapeStatus

result = scraper.scrape_url(ScrapeConfig('http://www.website.com'))

if result.status == ScrapeStatus.SUCCESS:
    html = result.first_page.html
else:
    print(result.error_msg)

Scrape a Page with Multiple Pages

import ezscrape.scraping.scraper as scraper
from ezscrape.scraping.core import ScrapeConfig
from ezscrape.scraping.core import WaitForXpathElem
from ezscrape.scraping.core import ScrapeStatus

config = ScrapeConfig('http://www.website.com')
# Add contition to wait until Element with "title='id'" is loaded"
config.wait_for_elem_list.append(WaitForXpathElem(R'''//a[@title='id']'''))

result = scraper.scrape_url(config)

for page in result:
    if page.status == ScrapeStatus.SUCCESS:
        html = page.html
    else:
        print(result.error_msg)

Scrape a Page and wait until an Element is Loaded

import ezscrape.scraping.scraper as scraper
from ezscrape.scraping.core import ScrapeConfig
from ezscrape.scraping.core import WaitForXpathElem
from ezscrape.scraping.core import ScrapeStatus


config = ScrapeConfig('http://www.website.com')
# Add contition to wait until Element with "title='id'" is loaded"
config.wait_for_elem_list.append(WaitForXpathElem(R'''//a[@title='id']'''))

result = scraper.scrape_url(config)

if result.status == ScrapeStatus.SUCCESS:
    html = result.first_page.html
else:
    print(result.error_msg)

Scrape Config

ezscrape.scraping.core.ScrapeConfig

The url is specified when creating the object.

from ezscrape.scraping.core import ScrapeConfig

config = ScrapeConfig('http://some-url.com') 

Additional parameters can be specified

Scrape Status

The following statuses are supported in ezscrape.scraping.core.ScrapeStatus

For non Success cases, additional error details are given in the ScrapeResult object

Scrape Result

The following attributes are available in ezscrape.scraping.core.ScrapeResults

Scrape Page

The following attributes are available in ezscrape.scraping.core.ScrapePage

Contributing

Pull requests are welcome. For major changes, please open an issue first to discuss what you would like to change.

Please make sure to update tests as appropriate.

License

GPLv3

[1]: # Scrape Result [2]: # Scrape Result