Skip to main content

Python library for scraping with Selenium.

Project description

as-scraper

PyPI - Python Version PyPI - Downloads

Python library for scraping using Selenium

If you are looking for the library implemented inside airflow, go to https://github.com/Avila-Systems/as-scraper-airflow.

Installation

The as-scraper library uses Geckodriver (Firefox) for scraping with the Selenium library. In order to use it, you need to have an Geckodriver dependency. Check the selenium documentation for details about how to install the Firefox browser driver.

Usage

Creating a simple scraper

Lets say that we want to scrap yellowpages.com. Our target data would be the popular cities that we can find in the sitemap url.

Our output data will have two columns: name of the city and url which is linked to the city. For example, for Houston, we would want the following output:

name url
Houston https://www.yellowpages.com/houston-tx

Declaring our Scraper Class

So first we create a scraper that extends from the Scraper class, and define the COLUMNS variable to ['name', 'url'].

Create the scrapers/yellowpages.py file and type the following code into it:

from as_scraper.scraper import Scraper


class YellowPagesScraper(Scraper):
    COLUMNS = ['name', 'url']

Deciding wether to load javascript or not

Now, there are two execution options when running scrapers. We can either load javascript which uses the Selenium library, or not load javascript and use the requests library for http requests.

For this example, let's go ahead and use the Selenium library. To configure this, simply add the following variable to your scraper:

from as_scraper.scraper import Scraper


class YellowPagesScraper(Scraper):
    COLUMNS = ['name', 'url']
    LOAD_JAVASCRIPT = True

Defining the scrape_handler

And the magic comes in the next step. We will define the scrape_handler method in our class, which will have the responsibility to scrape a given url and extract the data from it.

All scrapers must define the scrape_handler method.

from typing import Optional
from selenium.webdriver import Firefox
from selenium.webdriver.common.by import By
import pandas as pd
from as_scraper.scraper import Scraper


class YellowPagesScraper(Scraper):
    COLUMNS = ['name', 'url']
    LOAD_JAVASCRIPT = True

    def scrape_handler(self, url: str, html: Optional[str] = None, driver: Optional[Firefox] = None, **kwargs) -> pd.DataFrame:
        rows = []
        div_tag = driver.find_element(By.CLASS_NAME, "row-content")
        div_tag = div_tag.find_element(By.CLASS_NAME, "row")
        section_tags = div_tag.find_elements(By.TAG_NAME, "section")
        for section_tag in section_tags:
            a_tags = section_tag.find_elements(By.TAG_NAME, "a")
            for a_tag in a_tags:
                city_name = a_tag.text
                city_url = a_tag.get_attribute("href")
                rows.append({"name": city_name, "url": city_url})
        df = pd.DataFrame(rows, columns=self.COLUMNS)
        return df

Execution

Finally, to execute the scraper you must call the *execute method.

from typing import Optional
from selenium.webdriver import Firefox
from selenium.webdriver.common.by import By
import pandas as pd
from as_scraper.scraper import Scraper

class YellowPagesScraper(Scraper):
    COLUMNS = ['name', 'url']
    LOAD_JAVASCRIPT = True

    def scrape_handler(self, url: str, html: Optional[str] = None, driver: Optional[Firefox] = None, **kwargs) -> pd.DataFrame:
        rows = []
        div_tag = driver.find_element(By.CLASS_NAME, "row-content")
        div_tag = div_tag.find_element(By.CLASS_NAME, "row")
        section_tags = div_tag.find_elements(By.TAG_NAME, "section")
        for section_tag in section_tags:
            a_tags = section_tag.find_elements(By.TAG_NAME, "a")
            for a_tag in a_tags:
                city_name = a_tag.text
                city_url = a_tag.get_attribute("href")
                rows.append({"name": city_name, "url": city_url})
        df = pd.DataFrame(rows, columns=self.COLUMNS)
        return df

if __name__ == '__main__':
    urls = ['https://www.yellowpages.com/sitemap']
    scraper = YellowPagesScraper(urls)
    results, errors = scraper.execute()
    print(results)
    print(errors)

Now go ahead and run python scrapers/yellowpages.py. Have fun!

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

as-scraper-2.2.1.tar.gz (9.1 kB view details)

Uploaded Source

Built Distribution

as_scraper-2.2.1-py3-none-any.whl (9.4 kB view details)

Uploaded Python 3

File details

Details for the file as-scraper-2.2.1.tar.gz.

File metadata

  • Download URL: as-scraper-2.2.1.tar.gz
  • Upload date:
  • Size: 9.1 kB
  • Tags: Source
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/4.0.1 CPython/3.10.8

File hashes

Hashes for as-scraper-2.2.1.tar.gz
Algorithm Hash digest
SHA256 3582c1ad867606aed5127820f5d9376a74916153da2ba36f47255044e408f959
MD5 fb8fced739d95b39b2d52d3dbb4f0e08
BLAKE2b-256 e89112151bfdf0d9674c8c15ea335c62765a93bb6a6c7473b721ee2d9eb78e92

See more details on using hashes here.

File details

Details for the file as_scraper-2.2.1-py3-none-any.whl.

File metadata

  • Download URL: as_scraper-2.2.1-py3-none-any.whl
  • Upload date:
  • Size: 9.4 kB
  • Tags: Python 3
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/4.0.1 CPython/3.10.8

File hashes

Hashes for as_scraper-2.2.1-py3-none-any.whl
Algorithm Hash digest
SHA256 0d9afe6b8c922b6e4d7cf4dcc441305dced161b4c0b0b0f675574e6664bd6e6f
MD5 bc7ba943b199e503df792f1a92f1373a
BLAKE2b-256 0024309360a196bb5dc069fab07be9561f06468b4308b4ca45626d8de9c35a97

See more details on using hashes here.

Supported by

AWS AWS Cloud computing and Security Sponsor Datadog Datadog Monitoring Fastly Fastly CDN Google Google Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Sentry Sentry Error logging StatusPage StatusPage Status page