scrapingbee-cli 1.4.1

Command-line client for the ScrapingBee API: scrape pages (single or batch), crawl sites, check usage/credits, and use Google Search, Fast Search, Amazon, Walmart, YouTube, and ChatGPT from the terminal.

ScrapingBee CLI

Command-line client for the ScrapingBee API: scrape URLs (single or batch), crawl sites, check usage and credits, and use Google, Fast Search, Amazon, Walmart, YouTube, and ChatGPT from the terminal.

Requirements

• Python 3.10+

Setup: Install (below), then authenticate (Configuration). You need a ScrapingBee API key before any command will work.

Installation

Recommended — install with uv (no virtual environment needed):

curl -LsSf https://astral.sh/uv/install.sh | sh
uv tool install scrapingbee-cli

Alternative — install with pip in a virtual environment:

pip install scrapingbee-cli

From source: clone the repo and run pip install -e . in the project root.

Configuration

You need a ScrapingBee API key:

1. scrapingbee auth – Validate and save the key to config (use --api-key KEY for non-interactive; --show to print config path).
2. Environment – export SCRAPINGBEE_API_KEY=your_key
3. .env file – In the current directory or ~/.config/scrapingbee-cli/.env

Remove the stored key with scrapingbee logout. Get your API key from the ScrapingBee dashboard.

Usage

scrapingbee [command] [arguments] [options]

• scrapingbee --help – List all commands.
• scrapingbee [command] --help – Options and parameters for that command.

Options are per-command. Each command has its own set of options — run scrapingbee [command] --help to see them. Common options across batch-capable commands include --output-file, --output-dir, --input-file, --input-column, --concurrency, --output-format, --overwrite, --retries, --backoff, --resume, --update-csv, --no-progress, --extract-field, --fields, --smart-extract, --deduplicate, --sample, --post-process, --on-complete, --scraping-config, and --verbose. For details, see the documentation.

Parameter values: Choice parameters accept both hyphens and underscores interchangeably (e.g. --sort-by price-low and --sort-by price_low both work).

Commands

Command	Description
usage	Check credits and max concurrency
auth / logout	Save or remove API key
docs	Print docs URL; --open to open in browser
scrape [url]	Scrape a URL (HTML, JS, screenshot, extract)
crawl	Crawl sites following links, with AI extraction and save-pattern filtering
google / fast-search	Search SERP APIs
amazon-product / amazon-search	Amazon product and search
walmart-search / walmart-product	Walmart search and product
youtube-search / youtube-metadata	YouTube search and video metadata
chatgpt	ChatGPT API (--search true for web-enhanced responses)
export	Merge batch/crawl output to ndjson, txt, or csv (with --flatten, --columns)
schedule	Schedule commands via cron (--name, --list, --stop)
tutorial	Interactive step-by-step guide to CLI features (--chapter N, --reset, --list, --output-dir)

Batch mode: Commands that take a single input support --input-file (one line per input, or .csv with --input-column) and --output-dir. Use --output-format csv or --output-format ndjson to stream all results to a single file (or stdout) instead of individual files. Add --deduplicate to remove duplicate URLs, --sample N to test on a subset, or --post-process 'jq .title' to transform each result. Use --resume to skip already-completed items after interruption. Run bare scrapingbee --resume to discover incomplete batches in the current directory.

Parameters and options: Use space-separated values (e.g. --render-js false), not --option=value. For full parameter lists, response formats, and credit costs, see scrapingbee [command] --help and the ScrapingBee API documentation.

Key features

• AI extraction: --ai-extract-rules '{"price": "product price", "title": "product name"}' pulls structured data from any page using natural language — no CSS selectors needed. Works with scrape, crawl, and batch mode.
• CSS/XPath extraction: --extract-rules '{"title": "h1", "price": ".price"}' for consistent, cheaper production scraping. Find selectors in browser DevTools.
• Pipelines: Chain commands with --extract-field — e.g. google QUERY --extract-field organic_results.url > urls.txt then scrape --input-file urls.txt. Use --fields to filter JSON output keys; supports dot notation (e.g. --fields product.title,product.price).
• Smart Extract: --smart-extract extracts data from any format (JSON, HTML, XML, CSV, Markdown) using a path expression. Auto-detects format. Supports slicing, regex filtering, and JSON schema output.
• Update CSV: --update-csv fetches fresh data and updates the input CSV in-place. Ideal for daily price tracking, inventory monitoring, or any dataset that needs periodic refresh.
• Crawl with filtering: --include-pattern, --exclude-pattern control which links to follow. --save-pattern only saves pages matching a regex (others are visited for link discovery but not saved).
• Output formats: --output-format accepts ndjson (streams results as JSON lines) or csv (writes a single CSV) — these are the only valid values. Default (no flag) writes individual files per item into --output-dir.
• CSV input: --input-file products.csv --input-column url reads URLs from a CSV column.
• Export: scrapingbee export --input-dir batch/ --format csv --flatten --columns "title,price" merges batch output with nested JSON flattening and column selection.
• Scheduling: scrapingbee schedule --every 1d --name prices scrape --input-file products.csv --update-csv registers a cron job. Use --list, --stop NAME, or --stop all.
• Deduplication & sampling: --deduplicate removes duplicate URLs; --sample 100 processes only 100 random items.
• RAG chunking: scrape --chunk-size 500 --chunk-overlap 50 --return-page-markdown true outputs NDJSON chunks ready for vector DB ingestion.
• Scraping configurations: --scraping-config "My-Config" applies a pre-saved configuration from your ScrapingBee dashboard. Inline options override config settings. Create configurations in the request builder. Running scrapingbee --scraping-config NAME (without a subcommand) auto-routes to scrape.

Examples

scrapingbee usage
scrapingbee scrape "https://example.com" --output-file page.html
scrapingbee scrape "https://example.com/product" --ai-extract-rules '{"title": "product name", "price": "price"}'
scrapingbee google "pizza new york" --extract-field organic_results.url > urls.txt
scrapingbee scrape --input-file urls.txt --output-dir pages --deduplicate
scrapingbee crawl "https://store.com" --output-dir products --save-pattern "/product/" --ai-extract-rules '{"name": "name", "price": "price"}' --max-pages 200 --concurrency 200
scrapingbee export --input-dir products --format csv --flatten --columns "name,price" --output-file products.csv
scrapingbee scrape --input-file products.csv --input-column url --update-csv --ai-extract-rules '{"price": "current price"}'
scrapingbee schedule --every 1d --name price-tracker scrape --input-file products.csv --input-column url --update-csv --ai-extract-rules '{"price": "price"}'
scrapingbee schedule --list

# Smart Extract — pull fields from any format with a path expression
scrapingbee google "pizza new york" --smart-extract 'organic_results[0:3].title'
scrapingbee scrape "https://example.com" --smart-extract '...a[href=/mailto/].text'
scrapingbee scrape "https://example.com" --smart-extract '{"titles": "...h1", "links": "...href[0:5]"}'

Security

The --post-process, --on-complete, and schedule commands execute arbitrary shell commands on your machine. These features are disabled by default and require explicit human setup to enable.

For advanced features setup, see the Security section in our CLI documentation.

Do not enable these features in AI agent environments where commands may be constructed from scraped web content. ScrapingBee is not responsible for any damages caused by shell execution features. Use at your own discretion.

More information

• CLI Documentation – Full CLI reference with pipelines, parameters, and examples.
• Advanced usage examples – Shell piping, command chaining, batch workflows, monitoring scripts, NDJSON streaming, screenshots, Google search patterns, LLM chunking, and more.
• ScrapingBee API documentation – Parameters, response formats, credit costs, and best practices.
• Claude / AI agents: This repo includes a Claude Skill and Claude Plugin for agent use with file-based output and security rules.

Testing

Pytest is configured in pyproject.toml ([tool.pytest.ini_options]). From the project root:

1. Install the package with dev dependencies

pip install -e ".[dev]"

2. Run tests

Command	What runs
pytest tests/unit	Unit tests only (no API key needed)
pytest -m "not integration"	All except integration (no API key needed)
pytest	Full suite (integration tests require SCRAPINGBEE_API_KEY)
python tests/run_e2e_tests.py	E2E tests (182 tests, requires SCRAPINGBEE_API_KEY)
python tests/run_e2e_tests.py --filter GG	E2E tests filtered by prefix

Integration tests call the live ScrapingBee API and are marked with @pytest.mark.integration.