Skip to main content

PyPI version Python Versions Ask DeepWiki

scrape cli

It's a command-line tool to extract HTML elements using an XPath query or CSS3 selector.

It's based on the great and simple scraping tool written by Jeroen Janssens.

Installation

You can install scrape-cli using several methods:

pipx install scrape-cli

Using uv (modern Python package manager)

# Install as a global CLI tool (recommended)
uv tool install scrape-cli

# Or install with uv pip
uv pip install scrape-cli

# Or run temporarily without installing
uvx --from scrape-cli scrape --help

Using pip

pip install scrape-cli

Or install from source:

git clone https://github.com/aborruso/scrape-cli
cd scrape-cli
pip install -e .

Requirements

  • Python >=3.8
  • requests
  • lxml
  • cssselect
  • xmltodict

How does it work?

Using the Test HTML File

In the resources directory you'll find a test.html file that you can use to test various scraping scenarios.

Note: You can also test directly from the URL without cloning the repository:

scrape -e "h1" https://raw.githubusercontent.com/aborruso/scrape-cli/refs/heads/master/resources/test.html

Here are some examples:

  1. Extract all table data:
# CSS
scrape -e "table.data-table td" resources/test.html
# XPath
scrape -e "//table[contains(@class, 'data-table')]//td" resources/test.html
  1. Get all list items:
# CSS
scrape -e "ul.items-list li" resources/test.html
# XPath
scrape -e "//ul[contains(@class, 'items-list')]/li" resources/test.html
  1. Extract specific attributes:
# CSS
scrape -e "a.external-link" -a href resources/test.html
# XPath
scrape -e "//a[contains(@class, 'external-link')]/@href" resources/test.html
  1. Check if an element exists:
# CSS
scrape -e "#main-title" --check-existence resources/test.html
# XPath
scrape -e "//h1[@id='main-title']" --check-existence resources/test.html
  1. Extract nested elements:
# CSS
scrape -e ".nested-elements p" resources/test.html
# XPath
scrape -e "//div[contains(@class, 'nested-elements')]//p" resources/test.html
  1. Get elements with specific attributes:
# CSS
scrape -e "[data-test]" resources/test.html
# XPath
scrape -e "//*[@data-test]" resources/test.html
  1. Additional XPath examples:
# Get all links with href attribute
scrape -e "//a[@href]" resources/test.html

# Get checked input elements
scrape -e "//input[@checked]" resources/test.html

# Get elements with multiple classes
scrape -e "//div[contains(@class, 'class1') and contains(@class, 'class2')]" resources/test.html

# Get text content of specific element
scrape -e "//h1[@id='main-title']/text()" resources/test.html

General Usage Examples

A CSS selector query like this

curl -L 'https://en.wikipedia.org/wiki/List_of_sovereign_states' -s \
| scrape -be 'table.wikitable > tbody > tr > td > b > a'

Note: When using both -b and -e options together, they must be specified in the order -be (body first, then expression). Using -eb will not work correctly.

or an XPATH query like this one:

curl -L 'https://en.wikipedia.org/wiki/List_of_sovereign_states' -s \
| scrape -be "//table[contains(@class, 'wikitable')]/tbody/tr/td/b/a"

gives you back:

<html>
 <head>
 </head>
 <body>
  <a href="/wiki/Afghanistan" title="Afghanistan">
   Afghanistan
  </a>
  <a href="/wiki/Albania" title="Albania">
   Albania
  </a>
  <a href="/wiki/Algeria" title="Algeria">
   Algeria
  </a>
  <a href="/wiki/Andorra" title="Andorra">
   Andorra
  </a>
  <a href="/wiki/Angola" title="Angola">
   Angola
  </a>
  <a href="/wiki/Antigua_and_Barbuda" title="Antigua and Barbuda">
   Antigua and Barbuda
  </a>
  <a href="/wiki/Argentina" title="Argentina">
   Argentina
  </a>
  <a href="/wiki/Armenia" title="Armenia">
   Armenia
  </a>
...
...
 </body>
</html>

Text Extraction

You can extract only the text content (without HTML tags) using the -t option, which is particularly useful for LLMs and text processing:

# Extract all text content from a page
curl -L 'https://en.wikipedia.org/wiki/List_of_sovereign_states' -s \
| scrape -t

# Extract text from specific elements
curl -L 'https://en.wikipedia.org/wiki/List_of_sovereign_states' -s \
| scrape -te 'table.wikitable td'

# Extract text from headings only
scrape -te 'h1, h2, h3' resources/test.html

The -t option automatically excludes text from <script> and <style> tags and cleans up whitespace for better readability.

JSON Output

Use the -j/--json flag to get structured JSON natively, with no external tools:

scrape -je "a.external-link" resources/test.html

Output:

{
  "html": {
    "body": {
      "a": {
        "@href": "https://example.com",
        "@class": "external-link",
        "#text": "Example Link"
      }
    }
  }
}

Table extraction example:

scrape -je "table.data-table td" resources/test.html

Output:

{
  "html": {
    "body": {
      "td": [
        "Italy",
        "Rome",
        "59",
        "France",
        "Paris",
        "68"
      ]
    }
  }
}

-j automatically wraps the result in <html>/<body> before conversion, so you don't need to pass -b. For elements, the output is the same you would get from scrape -be ... | xq . (the underlying converter is xmltodict, the same library used by xq). Expressions that return only strings, such as //a/@href or //p/text(), give the string in body when there is one result and an array when there are more.

Use --json-array when a script needs one shape for any number of results. The output is a flat list with no html/body envelope, one item per result, in the order of the expressions. Strings stay strings and elements become objects with the tag as key, so scrape --json-array -e "//a/@href" -e "b" gives ["x", "y", {"b": "tre"}], and no results give []. It implies -j and has the same limits.

With mixed expressions, each result keeps its own form and its place in the order of the expressions. XPath function values are strings, so count() gives "2", not 2:

scrape --json-array -e "h1" -e "//a/@href" -e "count(//a)" page.html
# [{"h1": "News"}, "/one", "/two", "2"]

The list shape makes jq pipelines simple, with the same code for zero, one or many results:

# Count the results
scrape --json-array -e "a" page.html | jq 'length'

# Loop over attribute values; with no results, the loop does not run
scrape --json-array -e "//a/@href" page.html | jq -r '.[]' | while read -r href; do
  echo "got $href"
done

# Read fields of each element, as TSV
scrape --json-array -e "a" page.html | jq -r '.[].a | [.["@href"], .["#text"]] | @tsv'

# Filter elements by attribute
scrape --json-array -e "a" page.html | jq -c '.[] | select(.a["@class"] == "ext")'

# With mixed expressions, keep only the strings
scrape --json-array -e "h1" -e "//a/@href" page.html | jq -c 'map(strings)'

# Turn a count() into a number
scrape --json-array -e "count(//a)" page.html | jq '.[0] | tonumber'

-j is mutually exclusive with -t, -x (--check-existence) and -a (--argument).

-j exits with an error when a matched element has an attribute name that is not valid XML, such as @click or :bind.

Useful for JSON-based pipelines, APIs, databases, and processing with jq/DuckDB.

Some notes on the commands:

  • -e to set the query
  • -b to add <html>, <head> and <body> tags to the HTML output
  • -j to output structured JSON (built-in)
  • -t to extract only text content (useful for LLMs and text processing)

Several fields per element

Separate -e expressions lose the link between fields: if an article has no date, every following date ends up on the wrong article. Use --row to select the elements, and one --field NAME=EXPR for each value, relative to the row:

scrape --row "article" --field "title=h2" --field "url=.//a/@href" --field "date=time/@datetime" page.html
[
  {"title": "First story", "url": "/one", "date": "2026-01-01"},
  {"title": "Second story", "url": "/two", "date": null}
]
  • Every object has the same keys, and every value is a string or null. null means no match, "" a match with no text.
  • A field gives only its first match. Elements give their cleaned text, and XPath functions a string.
  • Field expressions are XPath or CSS, as with -e. Write attributes as .//a/@href: a/@href looks among direct children only, and //a is refused because it would read the whole document. In functions, write string(.//h2).
  • --absolute-urls and --base-url apply to href and src fields.
  • --row cannot be combined with -e, -t, -x, -a, -b, -j or --json-array.

JSON Lines

Use --jsonl to get one JSON value per line: the objects of --row, or otherwise the items of --json-array. With no results the output is empty. It cannot be combined with -t, -x, -a, -b or -j.

scrape --jsonl --row "article" --field "title=h2" --field "date=time/@datetime" page.html
# {"title": "First story", "date": "2026-01-01"}
# {"title": "Second story", "date": null}

# Articles with no date, with jq
scrape --jsonl --row "article" --field "title=h2" --field "date=time/@datetime" page.html \
  | jq -r 'select(.date == null) | .title'

# Query with DuckDB
scrape --jsonl --row "article" --field "title=h2" --field "date=time/@datetime" page.html \
  | duckdb -c "select title, date from read_json('/dev/stdin', format='newline_delimited')"

The whole page is read and the output built in memory: --jsonl changes the format, not the memory use.

Absolute URLs

Links and images often have relative values. Use --absolute-urls to get absolute href and src values, in every output mode except -t:

scrape --absolute-urls -e "//a/@href" https://www.iana.org/help/example-domains

With stdin or a local file there is no page URL, so pass it with --base-url, which implies --absolute-urls:

curl -s https://www.iana.org/help/example-domains \
  | scrape --base-url https://www.iana.org/help/example-domains -e "//a/@href"
  • The base is the first <base href> of the page, resolved against the page URL, as in browsers. With a URL as input, the page URL is the final one after redirects.
  • With no absolute base, the command exits with code 1.
  • Selection uses the original values, so a[href^="/"] still matches.
  • Other attributes, such as srcset, and XPath functions, such as string(//a/@href), are not changed. Use //a/@href.
  • --absolute-urls and --base-url cannot be combined with -r, and are ignored with -t and -x.

License

MIT

Metadata

Release files for scrape-cli 1.3.7

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for scrape-cli 1.3.7
File Size Uploaded Provenance Actions
scrape_cli-1.3.7.tar.gz 23.5 kB Without provenance Details for scrape_cli-1.3.7.tar.gz

Built distribution (wheel)

Table of built distributions (wheels) for scrape-cli 1.3.7
File Interpreter ABI Platform Actions
scrape_cli-1.3.7-py3-none-any.whl Python 3 none any Details for scrape_cli-1.3.7-py3-none-any.whl

Total release size: 37.6 kB

Release files / scrape_cli-1.3.7.tar.gz

Download URL scrape_cli-1.3.7.tar.gz
Size 23.5 kB
Tags Source
SHA-256 checksum
How to use checksums
c2a8f2f5c39b56985583535b31cab8b33387277fd61b75d04ce8a943fa5dbc15
BLAKE2b-256 checksum
How to use checksums
72e14cfffb8b25d73a2205371079343887f7eae4ff5be98c753ecdffcde120b5
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via twine/6.2.0 CPython/3.12.7

Release files / scrape_cli-1.3.7-py3-none-any.whl

Download URL scrape_cli-1.3.7-py3-none-any.whl
Size 14.1 kB
Tags Python 3
SHA-256 checksum
How to use checksums
ccb2c6cb4f68e6a175ad406c0b15dfd3742becfb8bb39506573fdac2fa0553d5
BLAKE2b-256 checksum
How to use checksums
4526ed475415ab2a79004cd5a19bfc144bb1b777e15c8ae2f942d5ab6d8571d2
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via twine/6.2.0 CPython/3.12.7

Security for scrape-cli 1.3.7

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

No provenance found

None of the files in this release were uploaded with provenance metadata.

A note on package security

PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.

To learn more about protecting yourself from malicious actors, visit our package security documentation.

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page