Small scraping utilities for CLI and Python workflows.
Verified details
These details have been verified by PyPIProject links
GitHub Statistics
Maintainers
Project description
scrape-smith
No-dependency Python utilities for scraping workflows.
The package provides a scrape command with tools for table extraction, list
extraction, body content JSONL conversion, combined three-in-one extraction,
and polite file downloads.
Installation
scrape-smith requires Python 3.10 or newer.
python -m pip install scrape-smith
Quickstart
scrape tables <html-file-or-url>
scrape tables page.html
scrape tables https://example.com/page.html -o result.csv
scrape tables page.html --format json -o tables.json
Without -o, output is written to a safe source-based filename:
page.html -> page-tables.csv
https://yeiichi.github.io/claim-class-model -> claim-class-model-tables.csv
If the default filename already exists, scrape avoids overwriting it by adding
a numeric suffix, such as page-tables-2.csv.
On success, scrape reports the written file to stdout:
Wrote 1 table to page-tables.csv
Use --quiet to suppress the success report.
Extract lists from an HTML page:
scrape lists page.html
scrape lists https://example.com/page.html
The lists command reads a local HTML file or HTTP(S) URL and extracts ul,
ol, and dl elements. Ordered and unordered lists produce a single item
column in CSV. Definition lists produce two columns, term and description.
Multiple lists are separated by blank rows.
Without -o, CSV output is written to a safe source-based filename:
page.html -> page-lists.csv
Convert body content from one HTML page to JSONL:
scrape content page.html
scrape content https://example.com/page.html
The content command reads a local HTML file or HTTP(S) URL, extracts visible
text from the <body>, ignores metadata-like elements such as <script>, and
writes one JSON object per line for headings (h1–h6), paragraphs (p), links (a), list items (li), table cells (td, th), block quotes (blockquote), figure captions (figcaption), definition terms and descriptions (dt, dd), table captions (caption), labels (label), and buttons (button):
{"h1": "Title"}
{"p": "Hello docs."}
{"a": "docs", "href": "/docs"}
Without -o, JSONL output is written to a safe source-based filename:
page.html -> page-content.jsonl
Extract all three in source order into one JSONL file:
scrape three page.html
scrape three https://example.com/page.html
The three command runs content, table, and list extraction in a single pass
and writes every record to one JSONL file in document order. Each line is a
{"type": ..., "data": ...} object where data is a JSON string:
{"type": "content", "data": "{\"tag\": \"h1\", \"text\": \"Title\"}"}
{"type": "list", "data": "{\"tag\": \"ul\", \"items\": [\"Alpha\", \"Beta\"]}"}
{"type": "table", "data": "{\"caption\": null, \"headers\": [\"Name\"], \"rows\": [[\"Ada\"]]}"}
Without -o, output is written to a safe source-based filename:
page.html -> page-extract.jsonl
Download target files from a URL list:
scrape download /path/to/urls.txt
The URL list should contain one URL per line. Blank lines and lines starting
with # are ignored. scrape-smith downloads CSV, PDF, DOCX, XLSX, and PPTX files
sequentially, waits between requests by default, and skips non-target URLs.
Without -o, files are saved under a collision-safe directory based on the URL
list filename:
urls.txt -> urls-downloads/
Downloaded files keep their original filenames when the URL or response headers
provide one. If no filename is available, scrape-smith falls back to a fixed-width
epoch filename such as 1700000000123.pdf.
Warning: treat downloaded files as untrusted. Scan them before opening, and do not open files blindly; documents and spreadsheets can contain harmful content.
Command Line
The command-line entry point is scrape.
scrape tables <html-file-or-url> [--format csv|json] [-o OUTPUT] [--index N] [--quiet]
scrape lists <html-file-or-url> [--format csv|json] [-o OUTPUT] [--quiet]
scrape content <html-file-or-url> [-o OUTPUT] [--quiet]
scrape three <html-file-or-url> [-o OUTPUT] [--quiet]
scrape download <url-list-path> [-o OUTPUT_DIR] [--delay SECONDS]
Options:
--format: output format, eithercsvorjson; defaults tocsv-o,--output: output file path; defaults to a safe source-based filename--index: write one table by zero-based index-q,--quiet: suppress success reports on stdout
Download options:
-o,--output-dir: output directory; defaults to a safe URL-list-based name--delay: seconds to wait between requests; defaults to1
Errors and validation messages are written to stderr. If --index is out of
range, the command exits with status code 2.
Python API
Python APIs live under scrape_smith.tools.
from scrape_smith.tools.tables import extract_tables
tables = extract_tables("page.html")
Each extracted table has caption, headers, and rows fields. JSON output
uses the same shape.
from scrape_smith.tools.downloads import download_files
summary = download_files("urls.txt", delay_seconds=1)
print(summary.downloaded_count)
from scrape_smith.tools.lists import extract_lists
lists = extract_lists("page.html")
from scrape_smith.tools.extract import extract_all
records = extract_all("page.html")
from scrape_smith.tools.content import extract_content_records
records = extract_content_records("page.html")
Documentation
Full documentation is available at https://scrape-smith.readthedocs.io/.
Project details
Verified details
These details have been verified by PyPIProject links
GitHub Statistics
Maintainers
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
Filter files by name, interpreter, ABI, and platform.
If you're not sure about the file name format, learn more about wheel file names.
Copy a direct link to the current filters
File details
Details for the file scrape_smith-0.3.0.tar.gz.
File metadata
- Download URL: scrape_smith-0.3.0.tar.gz
- Upload date:
- Size: 12.2 kB
- Tags: Source
- Uploaded using Trusted Publishing? Yes
- Uploaded via: twine/6.1.0 CPython/3.13.12
File hashes
| Algorithm | Hash digest | |
|---|---|---|
| SHA256 |
8a34ec17d204188d56df1dc6e84754edea1fe530f204ce77a12e31e1d1e358b1
|
|
| MD5 |
f2b9bfb79b8867c75aa99bff5be91608
|
|
| BLAKE2b-256 |
b10bdd7dac948c000b003fbfb3f72b6ec30f4e89238345077115be3cd95e1343
|
Provenance
The following attestation bundles were made for scrape_smith-0.3.0.tar.gz:
Publisher:
publish-pypi.yml on yeiichi/scrape-smith
-
Statement:
-
Statement type:
https://in-toto.io/Statement/v1 -
Predicate type:
https://docs.pypi.org/attestations/publish/v1 -
Subject name:
scrape_smith-0.3.0.tar.gz -
Subject digest:
8a34ec17d204188d56df1dc6e84754edea1fe530f204ce77a12e31e1d1e358b1 - Sigstore transparency entry: 2136142632
- Sigstore integration time:
-
Permalink:
yeiichi/scrape-smith@dd7efff35dcf616673c6311d715389ff28413fcf -
Branch / Tag:
refs/heads/main - Owner: https://github.com/yeiichi
-
Access:
public
-
Token Issuer:
https://token.actions.githubusercontent.com -
Runner Environment:
github-hosted -
Publication workflow:
publish-pypi.yml@dd7efff35dcf616673c6311d715389ff28413fcf -
Trigger Event:
push
-
Statement type:
File details
Details for the file scrape_smith-0.3.0-py3-none-any.whl.
File metadata
- Download URL: scrape_smith-0.3.0-py3-none-any.whl
- Upload date:
- Size: 17.4 kB
- Tags: Python 3
- Uploaded using Trusted Publishing? Yes
- Uploaded via: twine/6.1.0 CPython/3.13.12
File hashes
| Algorithm | Hash digest | |
|---|---|---|
| SHA256 |
0518c25601f5cdc387d280967f79683a68d5cc6cada92beb17c1b62df73d8eb3
|
|
| MD5 |
33638b086c43511618f33776581ca92a
|
|
| BLAKE2b-256 |
fc34d9f89c4a3e3ce6c4801d5c7319eac74d06cb683e3f6964d6d28d74c5caec
|
Provenance
The following attestation bundles were made for scrape_smith-0.3.0-py3-none-any.whl:
Publisher:
publish-pypi.yml on yeiichi/scrape-smith
-
Statement:
-
Statement type:
https://in-toto.io/Statement/v1 -
Predicate type:
https://docs.pypi.org/attestations/publish/v1 -
Subject name:
scrape_smith-0.3.0-py3-none-any.whl -
Subject digest:
0518c25601f5cdc387d280967f79683a68d5cc6cada92beb17c1b62df73d8eb3 - Sigstore transparency entry: 2136142647
- Sigstore integration time:
-
Permalink:
yeiichi/scrape-smith@dd7efff35dcf616673c6311d715389ff28413fcf -
Branch / Tag:
refs/heads/main - Owner: https://github.com/yeiichi
-
Access:
public
-
Token Issuer:
https://token.actions.githubusercontent.com -
Runner Environment:
github-hosted -
Publication workflow:
publish-pypi.yml@dd7efff35dcf616673c6311d715389ff28413fcf -
Trigger Event:
push
-
Statement type:
