Scrapy Scraper

Web crawler and scraper based on Scrapy and Playwright's headless browser.

To use the headless browser specify -p option. Browsers, unlike other standard web request libraries, have the ability to render JavaScript encoded HTML content.

Future plans:

• check if Playwright's Chromium headless browser is installed,
• add option to stop on rate limiting.

Resources:

• docs.scrapy.org - docs
• playwright.dev - docs
• scrapy/scrapy - GitHub
• scrapy-plugins/scrapy-playwright - GitHub

Tested on Kali Linux v2024.2 (64-bit).

Made for educational purposes. I hope it will help!

Table of Contents

• How to Install
  • Install Playwright and Chromium
  • Standard Install
  • Build and Install From the Source
• How to Run
• Usage
• Images

How to Install

Install Playwright and Chromium

pip3 install --upgrade playwright

playwright install chromium

Make sure each time you upgrade your Playwright dependency to re-install Chromium; otherwise, you might get an error using the headless browser.

Standard Install

pip3 install --upgrade scrapy-scraper

Build and Install From the Source

git clone https://github.com/ivan-sincek/scrapy-scraper && cd scrapy-scraper

python3 -m pip install --upgrade build

python3 -m build

python3 -m pip install dist/scrapy-scraper-4.0-py3-none-any.whl

How to Run

Example, start in-scope crawling from https://example.com/home, download in-scope JavaScript files, and extract links:

scrapy-scraper -u https://example.com/home -o results.json -a random -s 2 -rs -d downloads

Example, start in-scope crawling from URLs specified in urls.txt, take a screenshot of only the start URLs, and extract links:

scrapy-scraper -u urls.txt -o results.json -a random -s 2 -rs -p -ss screenshots

Usage

Scrapy Scraper v4.0 ( github.com/ivan-sincek/scrapy-scraper )

Usage:   scrapy-scraper -u urls                     -o out          [-d downloads] [-ss screenshots]
Example: scrapy-scraper -u https://example.com/home -o results.json [-d downloads] [-ss screenshots]

DESCRIPTION
    Probe, crawl, scrape, and screenshot websites
URLS
    File containing URLs or a single URL to start collecting from
    -u, --urls = urls.txt | https://example.com/home | etc.
WHITELIST
    File containing whitelisted domain names to limit the scope
    Specify 'off' to disable domain whitelisting
    Default: limit the scope to domain names extracted from the starting URLs
    -w, --whitelist = whitelist.txt | off | etc.
PLAYWRIGHT
    Use Playwright's headless browser
    -p, --playwright
PLAYWRIGHT WAIT
    Wait time in seconds before fetching the page content
    -pw, --playwright-wait = 0.5 | 2 | 4 | etc.
CONCURRENT REQUESTS
    Number of concurrent requests
    Default: 30
    -cr, --concurrent-requests = 30 | 45 | etc.
CONCURRENT REQUESTS PER DOMAIN
    Number of concurrent requests per domain
    Default: 10
    -crd, --concurrent-requests-domain = 10 | 15 | etc.
SLEEP
    Sleep time in seconds between two consecutive requests to the same domain
    -s, --sleep = 1.5 | 3 | etc.
RANDOM SLEEP
    Randomize the sleep time between requests to vary between '0.5 * sleep' and '1.5 * sleep'
    -rs, --random-sleep
AUTO THROTTLE
    Automatically throttle concurrent requests based on load and latency
    Sleep time is still respected
    -at, --auto-throttle = 0.5 | 10 | 15 | 30 | etc.
RETRIES
    Number of retries per URL
    Default: 2
    -rt, --retries = 0 | 4 | etc.
RECURSION
    Recursion depth limit
    Specify 'off' to disable crawling
    Specify '0' for no limit
    Default: 1
    -r, --recursion = off | 0 | 5 | etc.
REQUEST TIMEOUT
    Request timeout in seconds
    Default: 60
    -t, --request-timeout = 30 | 90 | etc.
HEADER
    Specify any number of extra HTTP request headers
    -H, --header = "Authorization: Bearer ey..." | etc.
COOKIE
    Specify any number of extra HTTP cookies
    -b, --cookie = PHPSESSIONID=3301 | etc.
USER AGENT
    User agent to use
    Default: Scrapy Scraper/4.0
    -a, --user-agent = random[-all] | curl/3.30.1 | etc.
PROXY
    Web proxy to use
    -x, --proxy = http://127.0.0.1:8080 | etc.
DOWNLOADS
    Output directory for downloaded JavaScript files
    Automatically beautifies the files
    -d, --downloads = downloads | etc.
SCREENSHOTS
    Output directory for screenshots
    -ss, --screenshots = screenshots | etc.
OUT
    Output file
    -o, --out = results.json | etc.
DEBUG
    Enable debug output
    -dbg, --debug

Images

Figure 1 - Scraping