web-scraper-tool 0.2.0

A web scraper tool to extract data from e-commerce websites

Web Scraper Tool

Secure, configurable web scraper for catalogue-style pages built with requests, BeautifulSoup, and optional Selenium.

Highlights

• ✅ HTTPS-first fetching with optional headless browser fallback
• ✅ Robots.txt enforcement and allowed-domain guardrails
• ✅ Retry-aware HTTP session with rate limiting
• ✅ Structured parsing + safe JSON/CSV persistence
• ✅ Environment-driven configuration (.env support)

Prerequisites

• Python 3.10+
• Chrome/Chromium installed if using browser mode
• Matching ChromeDriver on PATH for Selenium runs

Installation

pip install -r requirements.txt
# Optional dev tooling
pip install -r requirements-dev.txt

Quick Start (CLI)

python -m web_scraper_tool.cli https://example.com/catalog \
  --allow-domain example.com \
  --output results.json \
  --format json

Key flags:

• --use-browser – use headless Chrome for dynamic content.
• --allow-domain example.com – whitelist target domains.
• --format csv – persist CSV instead of JSON when writing to disk.
• --respect-robots / --ignore-robots – override robots policy per run.
• --verbose – emit debug logging.

Programmatic Usage

from web_scraper_tool import ScraperConfig, WebScraper
from web_scraper_tool.storage import save_products_to_json

config = ScraperConfig(
    allowed_domains=["example.com"],
    respect_robots=True,
)

scraper = WebScraper("https://example.com/catalog", config=config)
result = scraper.scrape()

for product in result.products:
    print(product.name, product.price)

save_products_to_json(result.products, "output/products.json")

Configuration

All settings may come from environment variables (see .env.example).

Variable	Description	Default
SCRAPER_TIMEOUT	HTTP timeout (seconds)	10
SCRAPER_MAX_RETRIES	Retry attempts for transient errors	3
SCRAPER_BACKOFF	Exponential backoff factor	0.5
SCRAPER_DELAY	Delay between requests (seconds)	1
SCRAPER_REQUIRE_HTTPS	Require HTTPS URLs	1
SCRAPER_RESPECT_ROBOTS	Enforce robots.txt	1
SCRAPER_ALLOWED_DOMAINS	Comma-separated domain whitelist	(unset)
SCRAPER_HEADLESS	Run headless browser	1
SCRAPER_USE_BROWSER	Prefer Selenium over HTTP	0

Settings are ingested via python-dotenv when a .env file is present.

Security Checklist

• Enforce HTTPS (SCRAPER_REQUIRE_HTTPS=1) unless explicit permission allows HTTP.
• Maintain a domain allowlist to prevent SSRF-style misuse.
• Honor robots.txt (default) unless you have legal clearance to opt out.
• Rate limit (SCRAPER_DELAY) to avoid detection/bans.
• Keep credentials and proxies in environment variables.

Running Tests

pytest

Docker Usage

Build the image (once):

docker build -t web-scraper-tool:latest .

Run the CLI inside the container:

docker run --rm \
  --env-file .env \
  --env SCRAPER_USE_BROWSER=0 \
  web-scraper-tool:latest \
  https://example.com/catalog \
  --allow-domain example.com \
  --output /app/output/results.json

Retrieve generated files by mounting a host volume:

docker run --rm \
  --env-file .env \
  -v "$(pwd)/output:/app/output" \
  --env SCRAPER_USE_BROWSER=0 \
  web-scraper-tool:latest \
  https://example.com/catalog \
  --allow-domain example.com \
  --output /app/output/results.json

Local Demo Catalog

To avoid rate limits and showcase the scraper safely, a static catalogue lives in demo/catalog.html.

1. Serve the page locally:

   cd demo
   python -m http.server 8000
   

2. In another terminal, run the scraper (native Python):

   python -m web_scraper_tool.cli "http://localhost:8000/catalog.html" \
     --allow-domain localhost \
     --require-http \
     --output output/demo.json
   

3. Or via Docker (Linux hosts) using host networking:

   docker run --rm \
     --network host \
     -v "$(pwd)/output:/app/output" \
     --env SCRAPER_USE_BROWSER=0 \
     web-scraper-tool:latest \
     "http://localhost:8000/catalog.html" \
     --allow-domain localhost \
     --require-http \
     --output /app/output/demo.json
   

Tip: The Docker image omits Chromium, so set SCRAPER_USE_BROWSER=0 (the default in native runs) to stay on the requests-based path. When tuning SCRAPER_DELAY, provide a numeric value (seconds) so the rate limiter remains happy.

Note: Real-world e-commerce sites often block automated scraping with CAPTCHAs or 503 errors. Demo with the provided catalogue or any site that explicitly permits scraping.

Troubleshooting

• ChromeDriver must match Chrome version for Selenium mode.
• Use --verbose to surface HTTP status codes and retry behaviour.
• Respect the target website's terms of service.

License

MIT