scrape-toolkit 0.1.0

Resilient web scraper and crawler CLI

web-scraper-toolkit

You give it a URL and tell it which data you want off the page. It gives you back a table (JSON, CSV or SQLite), following links along the way if you ask it to.

It is built for the real web: sites defend themselves against bots, so the tool goes slow, retries when it gets throttled, rotates User-Agents and, when it has to, opens a real browser. And it respects robots.txt by default.

Installation

pip install scrape-toolkit

That gives you the scrape command, and it is enough for most sites. If you also want browser mode (for sites that build the page with JavaScript), install the browser extra. It is kept separate on purpose: it downloads a whole Chromium, and you will not always need it.

pip install "scrape-toolkit[browser]"
playwright install chromium

From source

If you are going to work on the tool itself:

python3 -m venv .venv
.venv/bin/pip install -e ".[dev]"

Your first scrape

There is a ready-made example pointing at books.toscrape.com, a site built precisely for practising scraping. Save this as scraper.example.toml (it also ships in the repo):

[crawl]
seeds = ["https://books.toscrape.com/"]
max_depth = 1
max_pages = 25
same_domain = true
follow_selector = "h3 > a"
extract_pattern = "/catalogue/[^/]+/index\\.html$"

[fetch]
fetcher = "http"
delay = 1.0

[selectors]
title = "//h1/text()"
price = ".price_color"
availability = ".availability"
description = "#product_description ~ p"

[export]
formats = ["json", "csv", "sqlite"]
path = "output/books"

Run it as is:

scrape run scraper.example.toml

You will see a log line for every page it visits and, at the end, a table with what it wrote. Have a look at output/books.json:

{
  "title": "A Light in the Attic",
  "price": "£51.77",
  "availability": "In stock (22 available)",
  "description": "It's hard to imagine a world without A Light in the Attic...",
  "url": "https://books.toscrape.com/catalogue/a-light-in-the-attic_1000/index.html"
}

That is 20 books, also written to output/books.csv and output/books.db. That is all the tool does: turn pages into rows.

The concepts you need

Seed (seeds): the URL it starts from. You can give it several.

Crawling (max_depth): on top of scraping the seed, follow the links it finds there and scrape those pages too. The depth is how many hops it is allowed: 0 means "the seeds only", 1 means "the seeds and what they link to", and so on. Careful, it grows fast.

Selector: how you point at where the data lives inside the HTML. There are two languages for that, and the tool accepts both:

• CSS, the same one you use to style a page: .price_color means "the element with class price_color".
• XPath, more powerful, good for things CSS cannot do (like reading an attribute): //h1/text() means "the text of the h1".

The rule that tells them apart: if the selector starts with /, ./ or (, it is read as XPath. Otherwise, as CSS. You do not have to declare anything.

Following is not the same as extracting. This is the point that confuses people the most, and the one that makes a scraper return garbage. In the example, the books.toscrape.com home page is a list of 20 books: it is the page you reach each book from, but it is not a book itself. If you ask the tool to extract data from every page it visits, you get a row with title = "All products" and the price of the first book in the grid, mixed in among the real books. That is why there are two separate options:

• follow_selector: which links to follow (in the example, h3 > a, the links to each book).
• extract_pattern: which URLs to extract data from, as a regular expression. Pages that do not match are still visited — that is how you reach the ones that do — but they do not pollute the output.

Assets ([assets]): optionally, on top of the data, download each page's images or PDFs to disk.

Scraping your own site

1. Copy the example and change the seed.

cp scraper.example.toml my-site.toml

Put your URL in seeds.

2. Get the selectors from the browser. Open the page, F12, right-click the value you want → Copy → Copy selector. That gives you a CSS selector that works.

But do not paste it as is: Chrome tends to give you something like div:nth-child(3) > div > span, which breaks the moment the site moves a row around. Look at the HTML and find a class that describes the data (.price, .product-title). Those hold up.

3. Try a single page before you set off crawling. That way you check your selectors without hammering the site with hundreds of requests. Set seeds to the URL of one detail page (a product, an article: whatever it is you want to extract), and use:

max_depth = 0        # do not follow any link
max_pages = 1
# extract_pattern = ...   ← comment it out while you are testing

.venv/bin/scrape run my-site.toml

Look at the JSON. If a field comes out null, that selector did not match — and when none of them match, the tool tells you so in the log ("no selector matched on ..."). Once the data looks right, put the real seed back, raise max_depth and uncomment extract_pattern.

4. If junk rows show up, add extract_pattern. Work out what the URLs of the pages you actually want have in common (almost always a shared prefix, like /product/) and write the regex that matches them.

When the site blocks you

It will happen. Most defences are already handled; the table says what the tool does on its own and what you have to touch yourself.

You see this	What is going on	What to do
403	The site spotted you as a bot	Try --fetcher browser. The error message already suggests it
429	You are asking too fast	It retries on its own, honouring the wait the site asks for. If it persists, raise delay
503	Site down, or Cloudflare-style protection	It retries on its own, waiting longer each time. If it persists, --fetcher browser
The data comes out null	The selector does not match	Check the selector. If you can see it in the browser but not here, the page loads it with JavaScript → --fetcher browser
It visits nothing	robots.txt does not let you in	That is on purpose. Read the disclaimer before overriding it

If a site blocks you by IP, proxies takes a list and rotates through it.

CLI usage

scrape run config.toml                  # run with the config
scrape run config.toml --max-depth 3    # override the depth
scrape run config.toml --fetcher browser  # use the headless browser
scrape run config.toml --out data/output  # change where it writes
scrape run config.toml --log-file scrape.log  # save the log to a file
scrape run config.toml --ignore-robots  # see the disclaimer

Config reference

[crawl]
seeds = ["https://books.toscrape.com/"]  # where it starts (one or many)
max_depth = 1              # link hops allowed. 0 = the seeds only
max_pages = 25             # hard page cap: your safety net
same_domain = true         # do not wander off to other domains
follow_selector = "h3 > a" # which links to follow
extract_pattern = "/catalogue/[^/]+/index\\.html$"  # regex: which URLs to extract from

[fetch]
fetcher = "http"           # "http" (fast) or "browser" (handles JS)
delay = 1.0                # seconds to wait between requests
timeout = 20.0             # how long to wait for a response
max_retries = 3            # retries before giving up
respect_robots = true      # honour robots.txt
proxies = []               # list of proxies, rotated through
# user_agents = [...]      # omit it and a list of real browsers is used

[selectors]                # field_name = selector (CSS or XPath)
title = "//h1/text()"
price = ".price_color"

[export]
formats = ["json", "csv", "sqlite"]   # one or more
path = "output/books"                 # no extension: each format adds its own

[assets]                   # optional: download images/PDFs to disk
selectors = ["#product_gallery img"]
dir = "output/assets"

How it defends against blocks

• It goes slow: waits delay between requests. If robots.txt asks for a longer Crawl-delay, the site's number wins.
• It retries with a brain: on a 429 or 503 it honours the Retry-After header the server sends; if there is none, it waits longer each time (exponential backoff with jitter). A 404 is not retried: insisting will not make it appear.
• It blends in: rotates User-Agents from real browsers on every request.
• It rotates proxies: round-robin, sharing cookies across all of them.
• It keeps the session: cookies persist for the whole crawl.
• A real browser: fetcher = "browser" drives Chromium via Playwright. The two fetchers are genuinely interchangeable: robots.txt and assets travel over the chosen transport too, with its cookies and its proxies.

Tests

.venv/bin/pytest

HTTP responses are mocked, so the tests never hit a real site.

Legal disclaimer

This tool is for responsible scraping of sites that allow themselves to be scraped.

• robots.txt is respected by default. The --ignore-robots flag exists for cases where you have explicit permission from the site owner; using it is a deliberate decision and the responsibility lies with whoever runs the tool.
• It is your responsibility to review the site's Terms of Service and the applicable law (intellectual property, personal data, unauthorised access) before scraping.
• Set a reasonable delay. An aggressive scraper is indistinguishable from a denial-of-service attack.
• Do not scrape personal data or content behind a login without a legal basis for doing so.

Out of scope

• CAPTCHA solving: deliberately not included (ToS grey area).
• Concurrent crawling: the crawl is sequential. With delays between requests and a Crawl-delay to honour, concurrency buys little and complicates rate limiting.