scrapy-py 0.1.0

A simple web scraper library

🕷️ WebScraper

A Python library for automating repetitive web scraping tasks — built with batteries included.

Features

Module	What it does
Scraper	HTTP engine with retries, rate limiting, rotating User-Agents
Parser	HTML extraction — text, links, tables, images, meta tags
Pipeline	One-call scrape → parse → save for URL lists and paginated sites
Scheduler	Schedule recurring jobs (every N seconds/minutes/hours)
Storage	Save results as JSON or CSV, append, deduplicate

---

Installation

pip install requests beautifulsoup4 lxml
# Then copy the webscraper/ folder into your project

---

Quick Start

1. Fetch a page

from webscraper import Scraper, Parser

scraper = Scraper(delay=1.0)          # 1 second between requests
response = scraper.get("https://example.com")

parser = Parser(response.text)
print(parser.title())                  # → "Example Domain"
print(parser.links())                  # → ["https://..."]

2. Extract structured data

data = parser.extract({
    "title": "h1",
    "price": ".product-price",
    "images": {"selector": "img", "attr": "src", "all": True},
})
# → {"title": "Product Name", "price": "$9.99", "images": [...]}

3. Scrape multiple pages automatically

from webscraper import Pipeline, Storage

pipeline = Pipeline(
    scraper=Scraper(delay=1.5),
    storage=Storage("output"),
)

# Paginated scraping
results = pipeline.run_paginated(
    url_template="https://site.com/products?page={page}",
    start_page=1,
    end_page=10,
    extract_rules={
        "name":  "h2.product-title",
        "price": ".price",
        "link":  {"selector": "a.product-link", "attr": "href"},
    },
    output_file="products.json",
)

4. Schedule recurring scrapes

from webscraper import Scheduler

scheduler = Scheduler()

@scheduler.every(minutes=15, max_runs=96)   # every 15 min for 24h
def monitor_prices():
    scraper = Scraper()
    resp = scraper.get("https://example.com/prices")
    parser = Parser(resp.text)
    # ... extract and save

scheduler.run()   # blocking
# or: scheduler.run(blocking=False)  # background thread

---

API Reference

Scraper

Scraper(
    delay=1.0,           # seconds between requests
    delay_jitter=0.5,    # adds random 0–0.5s on top
    retries=3,           # auto-retry on 5xx / timeout
    timeout=30,          # request timeout
    headers={},          # custom headers
    proxies={},          # {"http": "...", "https": "..."}
    rotate_user_agents=True,
    verify_ssl=True,
)

Method	Description
.get(url)	GET request
.post(url, data, json)	POST request
.fetch_many(urls)	Fetch list of URLs
.crawl(start_url, max_pages)	Follow links from start URL

Parser

Parser(html_string)

Method	Description
.title()	<title> text
.text(selector, all=False)	Text from CSS selector
.attr(selector, attribute, all=False)	Attribute value
.links(base_url)	All <a href> URLs
.links_with_text()	[{"url": ..., "text": ...}]
.images(base_url)	[{"src": ..., "alt": ...}]
.table(index=0)	Parse HTML table → list of dicts
.all_tables()	All tables
.meta()	All meta tags
.og_data()	Open Graph tags
.body_text()	All visible text
.extract(rules)	Multi-field extraction
.extract() rules	{"field": "selector"} or {"field": {"selector": "...", "attr": "...", "all": bool}}

Pipeline

Pipeline(scraper=None, storage=None)

Method	Description
.run(urls, extract_rules, ...)	Scrape a list of URLs
.run_paginated(url_template, start, end, ...)	Scrape paginated URLs
.results	Last run's results

Scheduler

Scheduler()

Method	Description
.every(seconds, minutes, hours)	Decorator to schedule a function
.add_job(func, interval_seconds, ...)	Register job programmatically
.run(blocking=True)	Start scheduler
.stop()	Stop scheduler
.cancel(name)	Cancel a job
.status()	List all jobs and their state

Storage

Storage(output_dir="output")

Method	Description
.save_json(data, filename)	Save as JSON
.load_json(filename)	Load JSON
.append_json(new_data, filename)	Append to JSON array
.save_csv(data, filename)	Save as CSV
.load_csv(filename)	Load CSV
.append_csv(new_data, filename)	Append to CSV
.deduplicate(data, key)	Remove duplicates by field
.list_files()	List output directory files

---

License

MIT