Rubbernecker

A web scraping engine built with Python and Selenium that crawls web pages, stores raw HTML, and parses structured data. Supports configurable page actions and depth-based crawling.

Installation

Prerequisites

Python 3.12+

Google Chrome

macOS:

brew install --cask google-chrome

Fedora/RHEL (including WSL 2):

sudo dnf install -y fedora-workstation-repositories
sudo dnf config-manager setopt google-chrome.enabled=1
sudo dnf install -y google-chrome-stable

Ubuntu/Debian:

wget -q -O - https://dl.google.com/linux/linux_signing_key.pub | sudo apt-key add -
echo "deb [arch=amd64] http://dl.google.com/linux/chrome/deb/ stable main" | sudo tee /etc/apt/sources.list.d/google-chrome.list
sudo apt update
sudo apt install -y google-chrome-stable

Setup

make install

Or manually:

uv sync

Quick Start

See QUICKSTART.md for a step-by-step tutorial.

Stealth Crawling

Targets with bot protection (Cloudflare/Imperva-class) can be crawled with --driver stealth, which masks common automation fingerprints (navigator.webdriver, chromedriver's cdc_* variables, and a JavaScript evasion pack). This is best-effort only — see docs/stealth.md for what it does, its limits, and the fallback path if a target still blocks you.

uv run rubbernecker crawl tmp/urls.txt tmp/output.avro --driver stealth

Commands

Documentation

• Action Scripts — Automate page interactions during crawls
• Stealth Crawling — Best-effort evasion of bot detection
• Output Formats — Avro schemas for all command outputs
• Development — Testing, linting, and build commands

License

Apache-2.0