web-scraper-client 0.1.3

Client library for https://github.com/snackbeard/web-scraper-api

Client Library

The client library makes it easy to access the API by providing a builder class.
Example:

    options = DriverOptions(
        user_agent='Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/125.0.0.0 Safari/537.36',
        options=['--headless', '--disable-gpu', '--no-sandbox', '--disable-dev-shm-usage']
    )

    page_source = WebScraperBuilder(
                        url='https://my-domain.com',
                        api_key='c1f24ee0-1a77-4719-a33b-408069dfc15f')
                    .wait_for(seconds=5,
                          by=ApiInstructionIdentificatorType.CSS_SELECTOR,
                          wait_for=ApiInstructionElementType.ELEMENT_CLICKALBE,
                          element_id='button.sc-aXZVg.sc-lcIPJg.bjommA.jlhbaU.acceptAll',
                          ignore_error=True)
                    .click(ignore_error=True)
                    .wait_for(seconds=10,
                          by=ApiInstructionIdentificatorType.CSS_SELECTOR,
                          wait_for=ApiInstructionElementType.ELEMENT_PRESENCE,
                          element_id='div.category-event-items')
                    .scroll(block=ApiInstructionBlockType.END)
                    .get(page_url='https://www.page-to-scrape.com', options=options)

In the first .wait instruction a cookie dialog overlaps with the actual content we want to scrape. So to accept it we have to wait for the .acceptAll button to be clickable and then click it. Cookies won't appear again after accepting them once so if we scrape the site again an error would appear. So ignore_error is set to true in both instructions. Then we wait for an list to be present and scroll to the end of it to load all its content.

The following actions are currently supported

• wait - simply waits
  • seconds seconds to wait
• wait_for - wait for an element
  • seconds - seconds to wait
  • by - find it either by css selector or by element_id
  • wait_for - wait until the element is either present or clickable
  • element_id - id/selector of the element
  • ignore_error - if an element was not found ignore it and continue with the next instruction
• find - find and element
  • by - find it either by css selector or by element id
  • element_id - id/selector of the element
  • ignore_error - ...
• click - click an element after finding it or waiting for it
  • ignore_error - ...
• scroll - scroll to an element after finding it or waiting for it
  • block - scroll to either end or start
  • _ignore_error - ...
• get
  • page_url - webpage to scrape
  • content - page_source (html) or xhr (json)
  • xhr_name - document name (has to be provided if content is xhr)
  • options - chromedriver options

These instructions should be able to handle most use-cases