Python bindings around rust-scraper/scraper with PyO3
Project description
scraper-rs
Python bindings for the Rust scraper crate via PyO3. It gives you a lightweight Document/Element API with CSS selectors, XPath (via sxd_html/sxd_xpath), handy helpers, and zero Python-side parsing work.
Quick start
from scraper_rs import Document, first, select, select_first, xpath
html = """
<html><body>
<div class="item" data-id="1"><a href="/a">First</a></div>
<div class="item" data-id="2"><a href="/b">Second</a></div>
</body></html>
"""
doc = Document(html)
print(doc.text) # "First Second"
items = doc.select(".item")
print(items[0].attr("data-id")) # "1"
print(items[0].to_dict()) # {"tag": "div", "text": "First", "html": "<a...>", ...}
first_link = doc.select_first("a[href]") # alias: doc.find(...)
print(first_link.text, first_link.attr("href")) # First / /a
links_within_first = first_link.select("a[href]")
print([link.attr("href") for link in links_within_first]) # ["/a"]
# XPath (element results only)
xpath_items = doc.xpath("//div[@class='item']/a")
print([link.text for link in xpath_items]) # ["First", "Second"]
print(doc.xpath_first("//div[@data-id='1']/a").attr("href")) # "/a"
# Functional helpers
links = select(html, "a[href]")
print([link.attr("href") for link in links]) # ["/a", "/b"]
print(first(html, "a[href]").text) # First
print(select_first(html, "a[href]").text) # First
print([link.text for link in xpath(html, "//div[@class='item']/a")]) # ["First", "Second"]
For a runnable sample, see examples/demo.py.
Async usage
The scraper_rs.asyncio module wraps the top-level helpers so you can keep the event loop responsive. parse yields to the loop (the Document stays in the current thread), while select/xpath run in a thread pool:
import asyncio
from scraper_rs import asyncio as scraping_async
html = "<div class='item'><a href='/a'>First</a></div>"
async def main():
doc = await scraping_async.parse(html)
links = await scraping_async.select(html, "a[href]")
print(doc.select_first(".item").text) # First
print([link.attr("href") for link in links]) # ["/a"]
asyncio.run(main())
All async functions accept the same keyword arguments as their sync counterparts (max_size_bytes, truncate_on_limit, etc.).
Large documents and memory safety
To avoid runaway allocations, parsing defaults to a 1 GiB cap. Pass max_size_bytes to override:
from scraper_rs import Document, select
doc = Document(html, max_size_bytes=5_000_000) # 5 MB guard
links = select(html, "a[href]", max_size_bytes=5_000_000)
If you want to parse a limited portion of an oversized HTML document instead of rejecting it entirely, use truncate_on_limit=True:
# Parse only the first 100KB of a large HTML document
doc = Document(large_html, max_size_bytes=100_000, truncate_on_limit=True)
links = doc.select("a[href]") # Will only find links in the first 100KB
# Also works with top-level functions
items = select(large_html, ".item", max_size_bytes=100_000, truncate_on_limit=True)
Note: Truncation happens at valid UTF-8 character boundaries to prevent encoding errors.
API highlights
Document(html: str)/Document.from_html(html)parses once and keeps the DOM..select(css)→list[Element],.select_first(css)/.find(css)→ firstElement | None,.css(css)is an alias..xpath(expr)/.xpath_first(expr)evaluate XPath expressions that return element nodes..textreturns normalized text;.htmlreturns the original input.scraper_rs.asyncioexposes asyncparse/select/xpathwrappers to keep the event loop responsive.Elementexposes.tag,.text,.html,.attrsplus helpers.attr(name),.get(name, default),.to_dict().- Elements support nested CSS and XPath selection via
.select(css),.select_first(css),.find(css),.css(css),.xpath(expr),.xpath_first(expr). - Top-level helpers mirror the class methods:
parse(html),select(html, css),select_first(html, css)/first(html, css),xpath(html, expr),xpath_first(html, expr). max_size_byteslets you fail fast on oversized HTML; defaults to a 1 GiB limit.truncate_on_limitallows parsing a truncated version (limited tomax_size_bytes) of oversized HTML instead of raising an error.- Call
doc.close()(orwith Document(html) as doc: ...) to free parsed DOM resources when you're done.
Installation
Built wheels target abi3 (CPython 3.10+). To build locally:
# Install maturin (uv is used in this repo, but pip works too)
pip install maturin
# Build a wheel
maturin build --release --compatibility linux
# Install the generated wheel
pip install target/wheels/scraper_rs-*.whl
If you have just installed, the repo includes helpers: just build (local wheel), just install-wheel (install the built wheel), and just build_manylinux (via the official maturin Docker image).
Projects Using scraper-rs
- silkworm - Async web scraping framework on top of Rust
Development
Requirements: Rust toolchain, Python 3.10+, maturin, and pytest for tests.
- Run tests:
just testoruv run pytest tests/test_scraper.py - Format/typing: Rust and Python are small; no formatters are enforced yet.
- The PyO3 module name is
scraper_rs; the Rust crate is built ascdylib.
Contributions and issues are welcome. If you add public API, please extend tests/test_scraper.py and the example script accordingly.
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
Filter files by name, interpreter, ABI, and platform.
If you're not sure about the file name format, learn more about wheel file names.
Copy a direct link to the current filters
File details
Details for the file scraper_rust-0.2.11.tar.gz.
File metadata
- Download URL: scraper_rust-0.2.11.tar.gz
- Upload date:
- Size: 32.0 kB
- Tags: Source
- Uploaded using Trusted Publishing? No
- Uploaded via: maturin/1.10.2
File hashes
| Algorithm | Hash digest | |
|---|---|---|
| SHA256 |
28ff87cd0b5801a726c2e5abe67a95dbeade471c10090b6ecfbd482b2b4ae758
|
|
| MD5 |
ece0d5c44b78acc96994e63c7403f96c
|
|
| BLAKE2b-256 |
90eff99289892e176bc336d8bac197d0f06848a28ba3918a77dd609b881f9ea4
|
File details
Details for the file scraper_rust-0.2.11-cp310-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.
File metadata
- Download URL: scraper_rust-0.2.11-cp310-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
- Upload date:
- Size: 739.7 kB
- Tags: CPython 3.10+, manylinux: glibc 2.17+ x86-64
- Uploaded using Trusted Publishing? No
- Uploaded via: maturin/1.10.2
File hashes
| Algorithm | Hash digest | |
|---|---|---|
| SHA256 |
0e6378dc34c98a1841fb38f32c1e1ad56dd1ced12833c2ae6b59d22df647d70a
|
|
| MD5 |
850275caff85e19e22b4b332a4bda594
|
|
| BLAKE2b-256 |
10a91c9b298f1b5493ec69758a18f497e7eab27ad432cfe1e4157afdb8d97d44
|
