crawlr

An AI-powered, self-healing web scraper for e-commerce price & stock monitoring.

Website · PyPI · pip install crawlr

Crawlr uses an LLM (or a free offline heuristic) only to generate and repair CSS selectors, then extracts pages deterministically with those cached selectors. When a site changes its layout and the selectors break, Crawlr detects it and regenerates them automatically — so your monitoring never silently dies.

• Runs free & offline — no API key required; add an OpenAI/Anthropic key for higher accuracy.
• Self-healing — survives redesigns that break traditional scrapers.
• Zero-config — crawlr watch <url> auto-detects the right schema (product, jobs, real estate, news).
• Price & stock alerts — webhook (signed), Slack, Discord, Teams, Telegram, ntfy, or email, with throttling.
• General-purpose — e-commerce ships in; add new verticals as simple YAML.

Install

pipx install crawlr        # or: pip install crawlr

Quickstart

crawlr watch "https://store.com/product/123" --target 25   # schema auto-detected
crawlr watchlist                                           # current price, movement, stock
crawlr monitor --daemon                                    # keep checking in the background
crawlr serve                                               # optional local dashboard

Omit --schema and Crawlr detects the page type for you (product, product list, jobs, real estate, news). Pass --schema <name> to override.

Deploy with Docker

docker run -p 8000:8000 -v crawlr-data:/data ghcr.io/ardfaiyaz/crawlr   # dashboard on :8000

Or run the dashboard and a background monitor together:

docker compose up -d      # then open http://localhost:8000

Build locally with docker build -t crawlr .. Data (SQLite DB, selector cache, snapshots) persists in the /data volume.

Alert triggers

Pick one per watch with --trigger (or in the dashboard):

For richer logic, run crawlr init to create an editable crawlr.rules.yaml that maps circumstances to actions (and overrides per-watch triggers):

default_action: ignore
rules:
  - when: price_drops_below
    amount: 25
    action: alert
  - when: back_in_stock
    action: alert

Every option, explained

crawlr watch <url> [options] — start tracking a product page or a store's product list.

Triggers (--trigger ...), in plain English:

The commands you'll actually use:

The <id> numbers come from crawlr watchlist or crawlr sites.

Recipes

# Track a competitor's price, alert on a 10%+ drop, check hourly
crawlr watch "https://store.com/p/123" --trigger price_drop --interval 60

# Get pinged when something is back in stock
crawlr watch "https://store.com/p/123" --restock

# Compare the same product across three shops
crawlr compare "https://a.com/p" "https://b.com/p" "https://c.com/p"

# Monitor a job board (schema auto-detected)
crawlr watch "https://boards.example/jobs" --interval 720

# Free phone alerts via ntfy.sh
export CRAWLR_ALERT_NTFY="https://ntfy.sh/my-crawlr-alerts"
crawlr monitor --daemon

# Pause / resume / remove a watch
crawlr pause 3 && crawlr resume 3 && crawlr unwatch 3

# Pipe machine-readable output into other tools
crawlr watchlist --json | jq '.[] | {title, price}'

Custom schemas (no code)

Add new verticals by dropping a YAML file into your schema directory:

name: jobs
item_selector: ".job-card"
fields:
  - name: title
    description: the job title
    type: text
    required: true
  - name: salary
    description: annual salary
    type: number

Then crawlr scrape <url> --schema jobs. List schemas with crawlr schemas.

Configuration

Set via environment variables or a .env file:

Examples

Ready-to-run recipes live in examples/ — watch competitors, use a custom schema, and call the hosted API from Python.

Development

pip install -e '.[dev]'
pytest          # offline test suite
ruff check .    # lint
crawlr eval     # extraction accuracy gate

Contributing

Contributions are welcome — see CONTRIBUTING.md. Great first contributions: new vertical presets, golden eval cases for sites that extract wrong, and docs/examples.

Demo

A short GIF is the best intro. Generate one with VHS: vhs docs/demo.tape → docs/demo.gif (see docs/DEMO.md).

Does it work on big marketplaces (Lazada, Shopee, Amazon)?

Short answer: it can, but they're the hardest sites to scrape — expect extra setup, and don't count on 100% reliability.

Crawlr shines on sites with clean HTML or embedded product data (schema.org JSON-LD). Big marketplaces are deliberately hostile to bots, in three ways — here's how to give Crawlr its best shot:

1. They render prices with JavaScript. The price often isn't in the raw HTML; it loads after the page runs its scripts. Install the browser engine so Crawlr can render it:

   pip install "crawlr[js]"
   playwright install chromium
   

   Then pass --js (e.g. crawlr watch "<url>" --js) or let Crawlr auto-escalate when it detects a JS-only shell.

2. They have anti-bot protection (Cloudflare, CAPTCHAs, rate limits). Crawlr detects when it's blocked and skips that run instead of saving a bogus price — but getting through reliably usually needs rotating proxies:

   export CRAWLR_PROXIES="http://user:pass@host:port,http://host2:port"
   

3. robots.txt — many marketplaces disallow scraping these pages, and Crawlr honors robots.txt by default, so it may refuse to fetch. Only override this for sites you're permitted to scrape (see Legal & responsible use below):

   export CRAWLR_RESPECT_ROBOTS=false
   

Rule of thumb:

Tip: prove the whole flow works on an easy site first, then point it at the hard one:

crawlr watch "https://books.toscrape.com/catalogue/a-light-in-the-attic_1000/index.html" --target 60
crawlr watchlist

Legal & responsible use

Crawlr is a general-purpose scraping tool, and you are responsible for how you use it. Before monitoring any site, make sure you comply with:

• the target site's Terms of Service and robots.txt — Crawlr honors robots.txt by default, so keep CRAWLR_RESPECT_ROBOTS=true unless you own the site;
• applicable laws such as the CFAA (US) and GDPR/CCPA whenever personal data is involved — prefer public, non-personal data and avoid collecting PII;
• reasonable rate limits — use sensible intervals and delays so you don't overload a site.

Crawlr is meant for legitimate uses such as tracking your own listings, price research, and public-data monitoring. Please don't use it to bypass authentication, paywalls, or anti-abuse protections. The maintainers provide the software "as is" and are not responsible for how it is used (see LICENSE).

License

Crawlr is released under the MIT License — free to use, modify, and distribute, including commercially. See LICENSE.

Building a commercial or hosted product on top of Crawlr? An open-core model works well: keep the CLI and library MIT-licensed (this repo) and offer any hosted service, team features, or premium connectors as a separate paid layer.