Skip to main content

PyPI version Python Versions Ask DeepWiki

scrape cli

It's a command-line tool to extract HTML elements using an XPath query or CSS3 selector.

It's based on the great and simple scraping tool written by Jeroen Janssens.

Installation

You can install scrape-cli using several methods:

pipx install scrape-cli

Using uv (modern Python package manager)

# Install as a global CLI tool (recommended)
uv tool install scrape-cli

# Or install with uv pip
uv pip install scrape-cli

# Or run temporarily without installing
uvx --from scrape-cli scrape --help

Using pip

pip install scrape-cli

Or install from source:

git clone https://github.com/aborruso/scrape-cli
cd scrape-cli
pip install -e .

Requirements

  • Python >=3.8
  • requests
  • lxml
  • cssselect
  • xmltodict

How does it work?

Using the Test HTML File

In the resources directory you'll find a test.html file that you can use to test various scraping scenarios.

Note: You can also test directly from the URL without cloning the repository:

scrape -e "h1" https://raw.githubusercontent.com/aborruso/scrape-cli/refs/heads/master/resources/test.html

Here are some examples:

  1. Extract all table data:
# CSS
scrape -e "table.data-table td" resources/test.html
# XPath
scrape -e "//table[contains(@class, 'data-table')]//td" resources/test.html
  1. Get all list items:
# CSS
scrape -e "ul.items-list li" resources/test.html
# XPath
scrape -e "//ul[contains(@class, 'items-list')]/li" resources/test.html
  1. Extract specific attributes:
# CSS
scrape -e "a.external-link" -a href resources/test.html
# XPath
scrape -e "//a[contains(@class, 'external-link')]/@href" resources/test.html
  1. Check if an element exists:
# CSS
scrape -e "#main-title" --check-existence resources/test.html
# XPath
scrape -e "//h1[@id='main-title']" --check-existence resources/test.html
  1. Extract nested elements:
# CSS
scrape -e ".nested-elements p" resources/test.html
# XPath
scrape -e "//div[contains(@class, 'nested-elements')]//p" resources/test.html
  1. Get elements with specific attributes:
# CSS
scrape -e "[data-test]" resources/test.html
# XPath
scrape -e "//*[@data-test]" resources/test.html
  1. Additional XPath examples:
# Get all links with href attribute
scrape -e "//a[@href]" resources/test.html

# Get checked input elements
scrape -e "//input[@checked]" resources/test.html

# Get elements with multiple classes
scrape -e "//div[contains(@class, 'class1') and contains(@class, 'class2')]" resources/test.html

# Get text content of specific element
scrape -e "//h1[@id='main-title']/text()" resources/test.html

General Usage Examples

A CSS selector query like this

curl -L 'https://en.wikipedia.org/wiki/List_of_sovereign_states' -s \
| scrape -be 'table.wikitable > tbody > tr > td > b > a'

Note: When using both -b and -e options together, they must be specified in the order -be (body first, then expression). Using -eb will not work correctly.

or an XPATH query like this one:

curl -L 'https://en.wikipedia.org/wiki/List_of_sovereign_states' -s \
| scrape -be "//table[contains(@class, 'wikitable')]/tbody/tr/td/b/a"

gives you back:

<html>
 <head>
 </head>
 <body>
  <a href="/wiki/Afghanistan" title="Afghanistan">
   Afghanistan
  </a>
  <a href="/wiki/Albania" title="Albania">
   Albania
  </a>
  <a href="/wiki/Algeria" title="Algeria">
   Algeria
  </a>
  <a href="/wiki/Andorra" title="Andorra">
   Andorra
  </a>
  <a href="/wiki/Angola" title="Angola">
   Angola
  </a>
  <a href="/wiki/Antigua_and_Barbuda" title="Antigua and Barbuda">
   Antigua and Barbuda
  </a>
  <a href="/wiki/Argentina" title="Argentina">
   Argentina
  </a>
  <a href="/wiki/Armenia" title="Armenia">
   Armenia
  </a>
...
...
 </body>
</html>

Text Extraction

You can extract only the text content (without HTML tags) using the -t option, which is particularly useful for LLMs and text processing:

# Extract all text content from a page
curl -L 'https://en.wikipedia.org/wiki/List_of_sovereign_states' -s \
| scrape -t

# Extract text from specific elements
curl -L 'https://en.wikipedia.org/wiki/List_of_sovereign_states' -s \
| scrape -te 'table.wikitable td'

# Extract text from headings only
scrape -te 'h1, h2, h3' resources/test.html

The -t option automatically excludes text from <script> and <style> tags and cleans up whitespace for better readability.

JSON Output

Use the -j/--json flag to get structured JSON natively, with no external tools:

scrape -je "a.external-link" resources/test.html

Output:

{
  "html": {
    "body": {
      "a": {
        "@href": "https://example.com",
        "@class": "external-link",
        "#text": "Example Link"
      }
    }
  }
}

Table extraction example:

scrape -je "table.data-table td" resources/test.html

Output:

{
  "html": {
    "body": {
      "td": [
        "Italy",
        "Rome",
        "59",
        "France",
        "Paris",
        "68"
      ]
    }
  }
}

-j automatically wraps the result in <html>/<body> before conversion, so you don't need to pass -b. For elements, the output is the same you would get from scrape -be ... | xq . (the underlying converter is xmltodict, the same library used by xq). Expressions that return only strings, such as //a/@href or //p/text(), give the string in body when there is one result and an array when there are more.

-j is mutually exclusive with -t, -x (--check-existence) and -a (--argument).

-j exits with an error when a matched element has an attribute name that is not valid XML, such as @click or :bind.

Useful for JSON-based pipelines, APIs, databases, and processing with jq/DuckDB.

Some notes on the commands:

  • -e to set the query
  • -b to add <html>, <head> and <body> tags to the HTML output
  • -j to output structured JSON (built-in)
  • -t to extract only text content (useful for LLMs and text processing)

License

MIT

Metadata

Release files for scrape-cli 1.3.3

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for scrape-cli 1.3.3
File Size Uploaded Provenance Actions
scrape_cli-1.3.3.tar.gz 14.8 kB Without provenance Details for scrape_cli-1.3.3.tar.gz

Built distribution (wheel)

Table of built distributions (wheels) for scrape-cli 1.3.3
File Interpreter ABI Platform Actions
scrape_cli-1.3.3-py3-none-any.whl Python 3 none any Details for scrape_cli-1.3.3-py3-none-any.whl

Total release size: 24.7 kB

Release files / scrape_cli-1.3.3.tar.gz

Download URL scrape_cli-1.3.3.tar.gz
Size 14.8 kB
Tags Source
SHA-256 checksum
How to use checksums
9165214943cf44dcf38e8ec21023ec9cb4ec36bdef98fd50377de137ed33bcbf
BLAKE2b-256 checksum
How to use checksums
6f1ae35aa88a73c9f98db33eaad7c20b00003546925d68f959cb8f7b3562ff31
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via twine/6.2.0 CPython/3.12.7

Release files / scrape_cli-1.3.3-py3-none-any.whl

Download URL scrape_cli-1.3.3-py3-none-any.whl
Size 9.9 kB
Tags Python 3
SHA-256 checksum
How to use checksums
1c5df677ac147efd32f56c917c75f0fbdd4763bf4ca60cf1d5cd0aa19843621b
BLAKE2b-256 checksum
How to use checksums
be660ae85fa8610fe15bee34a7d925f8a03549a757f4b2d2890ece9b41ee1fda
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via twine/6.2.0 CPython/3.12.7

Security for scrape-cli 1.3.3

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

No provenance found

None of the files in this release were uploaded with provenance metadata.

A note on package security

PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.

To learn more about protecting yourself from malicious actors, visit our package security documentation.

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page