philiprehberger-web-scraper

Lightweight web scraper with rate limiting and CSS selectors.

Installation

pip install philiprehberger-web-scraper

Usage

from philiprehberger_web_scraper import Scraper

scraper = Scraper(rate_limit=2.0, retry_attempts=3)

# Fetch a single page
page = scraper.get("https://example.com")
titles = page.select_all("h2.title")
link = page.select_one("a.next")
all_links = page.links()

# Extract data
for el in page.select_all(".product"):
    print(el.select_one(".name").text)
    print(el.select_one("a").attr("href"))

# Crawl multiple pages
for page in scraper.crawl("https://example.com/blog", max_pages=20):
    for article in page.select_all("article"):
        print(article.select_one("h2").text)

# Export
Scraper.export_csv(data, "output.csv")
Scraper.export_json(data, "output.json")

Retry with Backoff

Transient HTTP errors (429 and 503) are retried automatically with exponential backoff. Configure the number of attempts and base delay:

from philiprehberger_web_scraper import Scraper

scraper = Scraper(retry_attempts=5, retry_delay=2.0)
page = scraper.get("https://example.com/api")

Response Caching

Cache fetched pages to disk so repeated requests for the same URL skip the network entirely:

from philiprehberger_web_scraper import Scraper, ResponseCache

cache = ResponseCache(cache_dir=".scraper_cache")
scraper = Scraper(cache=cache)

page = scraper.get("https://example.com")  # fetches from network
page = scraper.get("https://example.com")  # served from disk cache

cache.clear()  # remove all cached responses

Cache TTL

Expire cached entries after a number of seconds. Stale files are deleted on the next read so the cache directory does not grow unbounded:

from philiprehberger_web_scraper import ResponseCache

cache = ResponseCache(cache_dir=".scraper_cache", ttl=3600)  # 1 hour

Table Extraction

Pull an HTML table into a list of dicts using extract_table(). Use extract_tables() to pull every matching table on the page:

from philiprehberger_web_scraper import Scraper, extract_table, extract_tables

scraper = Scraper()
page = scraper.get("https://example.com/data")

# First matching table
rows = extract_table(page, "table#prices")
# [{"Product": "Widget", "Price": "$9.99"}, ...]

# All tables on the page
all_tables = extract_tables(page, "table")
# [[{...}, ...], [{...}, ...]]

Following Paginated Links

Use follow_links() to crawl paginated content by following a CSS-selected link on each page:

from philiprehberger_web_scraper import Scraper

scraper = Scraper()
for page in scraper.follow_links("https://example.com/page/1", "a.next-page", max_pages=10):
    for item in page.select_all(".result"):
        print(item.text)

Proxy Rotation

Distribute requests across multiple proxies by passing a list of proxy URLs:

from philiprehberger_web_scraper import Scraper

scraper = Scraper(proxies=[
    "http://proxy1:8080",
    "http://proxy2:8080",
    "http://proxy3:8080",
])
page = scraper.get("https://example.com")  # uses proxy1
page = scraper.get("https://example.com/2")  # uses proxy2

Rotating User Agents

Rotate the User-Agent header round-robin across requests:

from philiprehberger_web_scraper import Scraper

scraper = Scraper(user_agents=[
    "Mozilla/5.0 (X11; Linux x86_64) Firefox/130.0",
    "Mozilla/5.0 (Macintosh; Intel Mac OS X 14_5) Safari/17.5",
    "Mozilla/5.0 (Windows NT 10.0; Win64; x64) Chrome/127.0.0.0",
])
page = scraper.get("https://example.com")  # uses agent #1
page = scraper.get("https://example.com/2")  # uses agent #2

Respecting robots.txt

Enable respect_robots to honor each origin's robots.txt. The file is fetched once per origin and cached; disallowed URLs raise RobotsDisallowedError from get() (and are skipped automatically during crawl()/follow_links()):

from philiprehberger_web_scraper import Scraper, RobotsDisallowedError

scraper = Scraper(respect_robots=True)
try:
    page = scraper.get("https://example.com/private")
except RobotsDisallowedError as e:
    print(f"blocked: {e.url}")

Meta Tags

Read <meta> tags (including Open Graph og:* properties) from a page:

from philiprehberger_web_scraper import Scraper

scraper = Scraper()
page = scraper.get("https://example.com")

page.meta("description")   # "Example description"
page.meta("og:title")      # "Example — Open Graph title"
page.meta_tags()           # {"description": "...", "og:title": "...", ...}

Structured Data (JSON-LD)

Parse every <script type="application/ld+json"> block on a page. A block holding a JSON array contributes its items; invalid blocks are skipped:

from philiprehberger_web_scraper import Scraper

page = Scraper().get("https://example.com/product")
for item in page.json_ld():
    if isinstance(item, dict) and item.get("@type") == "Product":
        print(item["name"])

Selecting Text

Get the stripped text of every element matching a selector:

page.select_text("h2.title")  # ["First post", "Second post", ...]

Sitemaps

Read every page URL from an XML sitemap. Sitemap indexes are followed up to max_depth levels, URLs are deduplicated in order, and every fetch goes through get() (rate limiting, retries, robots.txt, caching, proxies, User-Agent rotation). A root document that fails, is malformed, or is not a sitemap raises ValueError; broken nested sitemaps are skipped:

from philiprehberger_web_scraper import Scraper

scraper = Scraper(rate_limit=1.0)
urls = scraper.sitemap_urls("https://example.com/sitemap.xml", max_depth=2)
for page in (scraper.get(u) for u in urls[:10]):
    print(page.title)

API

Development

pip install -e .
python -m pytest tests/ -v

Support

If you find this project useful:

⭐ Star the repo

🐛 Report issues

💡 Suggest features

❤️ Sponsor development

🌐 All Open Source Projects

💻 GitHub Profile

🔗 LinkedIn Profile

License

MIT