pw-simple-scraper 0.1.2

A simple and light Playwright-based scraper

pw-simple-scraper

A lightweight, easy-to-use web scraper built with Python and Playwright

한국어 보러가기

Overview

• pw-simple-scraper scrapes desired elements from a web page.
• Provide a URL + CSS selector, and it will return the matching elements as a list of strings.
• The result is wrapped in a ScrapeResult object. You can access the extracted values via .result (List[str]).

Installation

# 1. Install Playwright
pip install playwright

# 2-1. Install Chromium (macOS / Windows)
python -m playwright install chromium

# 2-2. Install Chromium (Linux)
python -m playwright install --with-deps chromium

# 3. Install pw-simple-scraper
pip install pw-simple-scraper

• Since this scraper is built on top of Playwright, both the Playwright library and the Chromium browser are required.

Usage

from pw-simple_scraper import scrape_context, scrape_attrs

# Extract text
res = scrape_context("https://example.com", "h3")
print(res.result)   # ['h3-type-content1', 'h3-type-content2', ...]
print(res.count)    # n (number of scraped elements)

# Extract links by Attribute (herf ...)
links = scrape_attr("https://example.com", "a", "herf")
print(links.result) # ['https://www.iana.org/domains/example', ...]

# Apply timeout option (default: 30 seconds)
scrape_context("https://example.com", "something", timeout=10) # 10 seconds
links = scrape_attr("https://example.com", "a", "herf", timeout=20) # 20 seconds

Result is a ScrapeResult object

@dataclass
class ScrapeResult:
    url: str
    selector: str
    result: List[str]       # Extracted values
    count: int              # Number of values
    fetched_at: datetime    # Execution timestamp (UTC)

FAQ

• Installed but browser fails to launch

  • You must install the browser with python -m playwright install chromium (Be mindful of the Linux --with-deps option.)

• RuntimeError: All strategies failed

  • This may happen if the selector doesn’t exist or the page loads slowly. Double-check your selector and try increasing the timeout.

• Scraping inside iframe

  • Planned for future support.

• xpath support

  • Planned for future support.

• robot.txt support

  • Will be added as a configurable option in the future.