It's a command-line tool to extract HTML elements using an XPath query or CSS3 selector.
Project description
scrape cli
It's a command-line tool to extract HTML elements using an XPath query or CSS3 selector.
It's based on the great and simple scraping tool written by Jeroen Janssens.
Installation
You can install scrape-cli using several methods:
Using pipx (recommended for CLI tools)
pipx install scrape-cli
Using uv (modern Python package manager)
# Install as a global CLI tool (recommended)
uv tool install scrape-cli
# Or install with uv pip
uv pip install scrape-cli
# Or run temporarily without installing
uvx --from scrape-cli scrape --help
Using pip
pip install scrape-cli
Or install from source:
git clone https://github.com/aborruso/scrape-cli
cd scrape-cli
pip install -e .
Requirements
- Python >=3.6
- requests
- lxml
- cssselect
How does it work?
Using the Test HTML File
In the resources directory you'll find a test.html file that you can use to test various scraping scenarios.
Note: You can also test directly from the URL without cloning the repository:
scrape -e "h1" https://raw.githubusercontent.com/aborruso/scrape-cli/refs/heads/master/resources/test.html
Here are some examples:
- Extract all table data:
# CSS
scrape -e "table.data-table td" resources/test.html
# XPath
scrape -e "//table[contains(@class, 'data-table')]//td" resources/test.html
- Get all list items:
# CSS
scrape -e "ul.items-list li" resources/test.html
# XPath
scrape -e "//ul[contains(@class, 'items-list')]/li" resources/test.html
- Extract specific attributes:
# CSS
scrape -e "a.external-link" -a href resources/test.html
# XPath
scrape -e "//a[contains(@class, 'external-link')]/@href" resources/test.html
- Check if an element exists:
# CSS
scrape -e "#main-title" --check-existence resources/test.html
# XPath
scrape -e "//h1[@id='main-title']" --check-existence resources/test.html
- Extract nested elements:
# CSS
scrape -e ".nested-elements p" resources/test.html
# XPath
scrape -e "//div[contains(@class, 'nested-elements')]//p" resources/test.html
- Get elements with specific attributes:
# CSS
scrape -e "[data-test]" resources/test.html
# XPath
scrape -e "//*[@data-test]" resources/test.html
- Additional XPath examples:
# Get all links with href attribute
scrape -e "//a[@href]" resources/test.html
# Get checked input elements
scrape -e "//input[@checked]" resources/test.html
# Get elements with multiple classes
scrape -e "//div[contains(@class, 'class1') and contains(@class, 'class2')]" resources/test.html
# Get text content of specific element
scrape -e "//h1[@id='main-title']/text()" resources/test.html
General Usage Examples
A CSS selector query like this
curl -L 'https://en.wikipedia.org/wiki/List_of_sovereign_states' -s \
| scrape -be 'table.wikitable > tbody > tr > td > b > a'
Note: When using both -b and -e options together, they must be specified in the order -be (body first, then expression). Using -eb will not work correctly.
or an XPATH query like this one:
curl -L 'https://en.wikipedia.org/wiki/List_of_sovereign_states' -s \
| scrape -be "//table[contains(@class, 'wikitable')]/tbody/tr/td/b/a"
gives you back:
<html>
<head>
</head>
<body>
<a href="/wiki/Afghanistan" title="Afghanistan">
Afghanistan
</a>
<a href="/wiki/Albania" title="Albania">
Albania
</a>
<a href="/wiki/Algeria" title="Algeria">
Algeria
</a>
<a href="/wiki/Andorra" title="Andorra">
Andorra
</a>
<a href="/wiki/Angola" title="Angola">
Angola
</a>
<a href="/wiki/Antigua_and_Barbuda" title="Antigua and Barbuda">
Antigua and Barbuda
</a>
<a href="/wiki/Argentina" title="Argentina">
Argentina
</a>
<a href="/wiki/Armenia" title="Armenia">
Armenia
</a>
...
...
</body>
</html>
Text Extraction
You can extract only the text content (without HTML tags) using the -t option, which is particularly useful for LLMs and text processing:
# Extract all text content from a page
curl -L 'https://en.wikipedia.org/wiki/List_of_sovereign_states' -s \
| scrape -t
# Extract text from specific elements
curl -L 'https://en.wikipedia.org/wiki/List_of_sovereign_states' -s \
| scrape -te 'table.wikitable td'
# Extract text from headings only
scrape -te 'h1, h2, h3' resources/test.html
The -t option automatically excludes text from <script> and <style> tags and cleans up whitespace for better readability.
JSON Output Integration
You can integrate scrape-cli with xq (part of yq) to convert HTML output to structured JSON:
# Extract and convert to JSON (requires -b for complete HTML)
scrape -be "a.external-link" resources/test.html | xq .
Output:
{
"html": {
"body": {
"a": {
"@href": "https://example.com",
"@class": "external-link",
"#text": "Example Link"
}
}
}
}
Table extraction example:
scrape -be "table.data-table td" resources/test.html | xq .
Output:
{
"html": {
"body": {
"td": [
"1",
"John Doe",
"john@example.com",
"2",
"Jane Smith",
"jane@example.com"
]
}
}
}
Note: The -b flag is mandatory to produce valid HTML with <html>, <head> and <body> tags.
Useful for JSON-based pipelines, APIs, databases, and processing with jq/DuckDB.
Some notes on the commands:
-eto set the query-bto add<html>,<head>and<body>tags to the HTML output-tto extract only text content (useful for LLMs and text processing)
License
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
Filter files by name, interpreter, ABI, and platform.
If you're not sure about the file name format, learn more about wheel file names.
Copy a direct link to the current filters
File details
Details for the file scrape_cli-1.2.2.tar.gz.
File metadata
- Download URL: scrape_cli-1.2.2.tar.gz
- Upload date:
- Size: 9.3 kB
- Tags: Source
- Uploaded using Trusted Publishing? No
- Uploaded via: twine/6.2.0 CPython/3.12.7
File hashes
| Algorithm | Hash digest | |
|---|---|---|
| SHA256 |
fec6ac4a3f3424d86469ae1ec213b79fcabf5ba176d1c3caf971cfa3829e48cf
|
|
| MD5 |
f7c999d27fd7b2c6dbf15934f545d397
|
|
| BLAKE2b-256 |
ad301698f1484b11caa219ecf3305b83aa313b30bf413b43fa0c96e4ae01345e
|
File details
Details for the file scrape_cli-1.2.2-py3-none-any.whl.
File metadata
- Download URL: scrape_cli-1.2.2-py3-none-any.whl
- Upload date:
- Size: 7.9 kB
- Tags: Python 3
- Uploaded using Trusted Publishing? No
- Uploaded via: twine/6.2.0 CPython/3.12.7
File hashes
| Algorithm | Hash digest | |
|---|---|---|
| SHA256 |
6b585a8da22473f7609b1900be420ed0ce27a84af1f6c94e9390967dbbf529b3
|
|
| MD5 |
ff6f9f2af7c618c789c3d02d3c42dc45
|
|
| BLAKE2b-256 |
43403168246edcf2f3d361eddd3312ae964e939ba43f7775d1ddc7af86887ce0
|
