pcdt-scraper

A PyChromeDevTools based WebScraper and selenium like syntax.

Introduction

Sometimes website blocks your requests or aiohttp web request but don't block chrome web request.

For this solution, here is "pcdt-scraper".

Compatability

Python 3.6+ is required.

Installation

pip install pcdt-scraper

or

pip3 install pcdt-scraper

Usage:

1. First run chromium or chrome remote instance

chromium --remote-debugging-port=9222 --remote-allow-origins=*

or You can run as headless mode.

chromium --remote-debugging-port=9222 --remote-allow-origins=* --headless

2. Then run python code

from pcdt_scraper import WebScraper

scraper = WebScraper()
url = "https://www.example.com/"
try:
    # Navigate to a page
    if scraper.get(url):

        # Get page content
        content = scraper.get_page_content()

        # find element by class name
        text = scraper.find_element_by_class_name('class_name').text()
        print(text)

except Exception as e:
    print(f"An error occurred: {str(e)}")

finally:
    scraper.close()