whsk
whsk (pronounced "whisk") is a command line utility for web scraper authors.
It provides a set of utilities for inspecting HTML responses, and applying selectors against them.
Installation
It is recommended you install whsk with uvx or pipx.
uvx whsk is the fastest way to get running with whsk
It currently consists of two utilities:
whsk shell
whsk shell fetches a page, automatically parsing HTML, XML, or JSON responses.
It then opens an ipython shell allowing you to interact with the raw and parsed response.
When the command runs it will print a table of the variables it has loaded (which will depend on the type of page and particular flags passed):
$ uvx whsk shell https://example.com
variables
┌──────────┬───────────────────────┐
│ url │ https://example.com │
│ resp │ <Response [200 OK]> │
│ root │ lxml.html.HtmlElement │
└──────────┴───────────────────────┘
In [1]:
The In[1]: is an ipython prompt, the variables in the table area available for inspection & usage.
If you pass a selector from the command line, that first query will be made for you:
$ uvx whsk shell https://example.com --xpath //p
variables
┌──────────┬───────────────────────┐
│ url │ https://example.com │
│ resp │ <Response [200 OK]> │
│ root │ lxml.html.HtmlElement │
│ selector │ //p │
│ selected │ 2 elements │
└──────────┴───────────────────────┘
In [1]:
Options
Usage: whsk shell [OPTIONS] URL
Launch an interactive Python shell for scraping
╭─ Arguments ──────────────────────────────────────────────────────────────────────────╮
│ * url TEXT URL to scrape [default: None] [required] │
╰──────────────────────────────────────────────────────────────────────────────────────╯
╭─ Options ────────────────────────────────────────────────────────────────────────────╮
│ --ua TEXT User agent to make requests with │
│ --postdata -p TEXT POST data (will make a POST instead of GET) │
│ --header -h TEXT Additional headers in format 'Name: Value' │
│ --css -c TEXT css selector │
│ --xpath -x TEXT xpath selector │
│ --help Show this message and exit. │
╰──────────────────────────────────────────────────────────────────────────────────────╯
whsk query
whsk query takes the same command line options as whsk shell but instead of opening a shell
will output the results of the --css or --xpath selection, and then exit immediately.
As such, you must provide one of the two selector parameters.
This can be used for rapid testing of queries without opening the shell each time.
Options
Usage: whsk query [OPTIONS] URL
Run a one-off query against the URL
╭─ Arguments ─────────────────────────────────────────────────────────────────────────────────────────╮
│ * url TEXT URL to scrape [default: None] [required] │
╰─────────────────────────────────────────────────────────────────────────────────────────────────────╯
╭─ Options ───────────────────────────────────────────────────────────────────────────────────────────╮
│ --ua TEXT User agent to make requests with │
│ --postdata -p TEXT POST data (will make a POST instead of GET) │
│ --header -h TEXT Additional headers in format 'Name: Value' │
│ --css -c TEXT css selector │
│ --xpath -x TEXT xpath selector │
│ --help Show this message and exit. │
╰─────────────────────────────────────────────────────────────────────────────────────────────────────╯
Common Parameters
--ua
This parameter is provided as a shortcut to set common browser "User-Agent" headers.
It must be one of:
- linux.chrome
- linux.firefox
- mac.chrome
- mac.firefox
- mac.safari
- win.chrome
- win.edge
- win.firefox
These will use the values in user_agents.py, a relatively recent snapshot of a real user agent for the browser in question.
If you need to set a custom user agent, use --header 'user-agent: whatever you need'
Metadata
Release files for whsk 0.3.1
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | Provenance | Actions |
|---|---|---|---|---|
| whsk-0.3.1.tar.gz | 227.9 kB | Without provenance | Details for whsk-0.3.1.tar.gz |
Built distribution (wheel)
| File | Interpreter | ABI | Platform | Actions Reset |
|---|---|---|---|---|
| whsk-0.3.1-py3-none-any.whl | Python 3 | none | any | Details for whsk-0.3.1-py3-none-any.whl |
Total release size: 233.3 kB
Release files / whsk-0.3.1.tar.gz
| Download URL | whsk-0.3.1.tar.gz |
|---|---|
| Size | 227.9 kB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
45d279a00501b7310b92d8279595f5070687fbbd2626148765b99527a9711727
|
|
BLAKE2b-256 checksum How to use checksums |
457f073b3c29cffb1363f88a6526d07281ca01cb33c66f1dadbe2f299a5777ac
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
uv/0.7.22
|
Release files / whsk-0.3.1-py3-none-any.whl
| Download URL | whsk-0.3.1-py3-none-any.whl |
|---|---|
| Size | 5.4 kB |
| Tags | Python 3 |
|
SHA-256 checksum How to use checksums |
820c80fd4a097d4c4fe11fac53065ca1240bb66eab783db08631a926b57bb835
|
|
BLAKE2b-256 checksum How to use checksums |
516f0c3742c276466761b67550fa4e9718af7f70123b65c533f91392e1a31759
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
uv/0.7.22
|
Security for whsk 0.3.1
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
No provenance found
None of the files in this release were uploaded with provenance metadata.
A note on package security
PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.
To learn more about protecting yourself from malicious actors, visit our package security documentation.