scraper-rs

Python bindings for high-performance Rust HTML parsing via PyO3. It provides a lightweight Document/Element API with CSS selectors backed by rustedbytes-tl, XPath (via xee-xpath), handy helpers, and zero Python-side parsing work.

Install

pip install scraper-rust

# or using uv
uv add scraper-rust

Quick start

from scraper_rs import Document, first, prettify, select, select_first, xpath

html = """
<html><body>
  <div class="item" data-id="1"><a href="/a">First</a></div>
  <div class="item" data-id="2"><a href="/b">Second</a></div>
</body></html>
"""

doc = Document(html)
print(doc.text)  # "First Second"

items = doc.select(".item")
print(items[0].attr("data-id"))  # "1"
print(items[0].to_dict())  # {"tag": "div", "text": "First", "html": "<a...>", ...}

first_link = doc.select_first("a[href]")  # alias: doc.find(...)
print(first_link.text, first_link.attr("href"))  # First /a
links_within_first_item = items[0].select("a[href]")
print([link.attr("href") for link in links_within_first_item])  # ["/a"]

# XPath (element results only)
xpath_items = doc.xpath("//div[@class='item']/a")
print([link.text for link in xpath_items])  # ["First", "Second"]
print(doc.xpath_first("//div[@data-id='1']/a").attr("href"))  # "/a"

# Functional helpers
links = select(html, "a[href]")
print([link.attr("href") for link in links])  # ["/a", "/b"]
print(first(html, "a[href]").text)  # First
print(select_first(html, "a[href]").text)  # First
print(
    [link.text for link in xpath(html, "//div[@class='item']/a")]
)  # ["First", "Second"]
print(prettify(html))

For runnable samples, see examples/demo.py and examples/demo_prettify_url.py. Quick URL prettify demo:

python examples/demo_prettify_url.py https://example.com --max-lines 80

Async usage

The scraper_rs.asyncio module exposes an async-first surface for coroutine code. AsyncDocument stores shareable HTML/text state instead of a thread-affine sync Document, and all selectors are awaitable for consistent async calling style:

import asyncio
from scraper_rs import asyncio as scraping_async

html = "<div class='item'><a href='/a'>First</a></div>"


async def main():
    async with await scraping_async.parse(html) as doc:
        items = await doc.select(".item")
        first_link = await items[0].select_first("a[href]")
        print(first_link.text)  # First

    links = await scraping_async.select(html, "a[href]")
    print([link.attr("href") for link in links])  # ["/a"]


asyncio.run(main())

All async functions accept the same keyword arguments as their sync counterparts (max_size_bytes, truncate_on_limit, etc.). CPU-bound async parsing and selection work is offloaded through the Rust async core onto Rayon worker threads so concurrent coroutines do not run that work on the event-loop thread. AsyncDocument supports async with for automatic cleanup in coroutine code, and AsyncElement / AsyncDocument both expose async .prettify() helpers.

Large documents and memory safety

To avoid runaway allocations, parsing defaults to a 1 GiB cap. Pass max_size_bytes to override:

from scraper_rs import Document, select

doc = Document(html, max_size_bytes=5_000_000)  # 5 MB guard
links = select(html, "a[href]", max_size_bytes=5_000_000)

If you want to parse a limited portion of an oversized HTML document instead of rejecting it entirely, use truncate_on_limit=True:

# Parse only the first 100KB of a large HTML document
doc = Document(large_html, max_size_bytes=100_000, truncate_on_limit=True)
links = doc.select("a[href]")  # Will only find links in the first 100KB

# Also works with top-level functions
items = select(large_html, ".item", max_size_bytes=100_000, truncate_on_limit=True)

Note: Truncation happens at valid UTF-8 character boundaries to prevent encoding errors.

API highlights

Document(html: str) / Document.from_html(html) parse HTML for CSS and keep the DOM; XPath parsing is initialized lazily on first XPath query.
.select(css) → list[Element], .select_first(css) / .find(css) → first Element | None, .css(css) is an alias.
.xpath(expr) / .xpath_first(expr) evaluate XPath expressions that return element nodes.
.prettify() renders the current DOM as an indented string for readable output/debugging.
.text returns normalized text; Document.html is the original input HTML; Element.html is inner HTML.
scraper_rs.asyncio exposes async parse/select/xpath wrappers plus awaitable AsyncDocument / AsyncElement methods.
Element exposes .tag, .text, .html, .attrs plus helpers .attr(name), .get(name, default), .to_dict().
Elements support nested CSS and XPath selection via .select(css), .select_first(css), .find(css), .css(css), .xpath(expr), .xpath_first(expr).
Elements also expose .prettify() to format element HTML with indentation.
Top-level helpers mirror the class methods: parse(html), prettify(html), select(html, css), select_first(html, css) / first(html, css), xpath(html, expr), xpath_first(html, expr).
Parse-tree helpers parse_document(html) and parse_fragment(html) return rustedbytes-tl backed structural dictionaries for callers that need node data.
CSS selection uses the native rustedbytes-tl selector subset: tag, id, class, *, attribute selectors, comma selectors, descendants, and direct children.
max_size_bytes lets you fail fast on oversized HTML; defaults to a 1 GiB limit.
truncate_on_limit allows parsing a truncated version (limited to max_size_bytes) of oversized HTML instead of raising an error.
Call doc.close() (or with Document(html) as doc: ...) to free parsed DOM resources when you're done.
In async workflows, use async with await scraper_rs.asyncio.parse(html) as doc: ... for automatic AsyncDocument cleanup.

Installation

Built wheels target abi3 (CPython 3.10+). To build locally:

# Install maturin (uv is used in this repo, but pip works too)
pip install maturin

# Build a wheel
maturin build --release --compatibility linux

# Install the generated wheel
pip install target/wheels/scraper_rust-*.whl

If you have just installed, the repo includes helpers: just build (local wheel), just install-wheel (install the built wheel), and just build_manylinux (via the official maturin Docker image).

Projects Using scraper-rs

silkworm - Async web scraping framework on top of Rust
silkworm-mcp - An MCP server for silkworm

Development

Requirements: Rust toolchain, Python 3.10+, uv, maturin, pytest, and pytest-asyncio for tests.

Run tests: just test or uv run pytest tests/
Format code: just fmt (or cargo fmt --all and uv run ruff format)
Lint Rust: just lint (or cargo clippy --all-targets --all-features -- -D warnings)
The PyO3 module name is scraper_rs; the Rust crate is built as cdylib.

Contributions and issues are welcome. If you add public API, please extend the relevant tests, type stubs, docs, and examples accordingly.

Release files for scraper-rust 0.10.0

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for scraper-rust 0.10.0
File	Size	Uploaded
scraper_rust-0.10.0.tar.gz	79.2 kB	Sep 25, 2026	Details

Built distributions (wheels)

Table of built distributions (wheels) for scraper-rust 0.10.0
File	Interpreter	ABI	Platform	Reset
scraper_rust-0.10.0-cp310-abi3-win_amd64.whl 2.1 MB Sep 25, 2026	CPython 3.10	abi3	Windows x86-64	Details
scraper_rust-0.10.0-cp310-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl 2.3 MB Sep 25, 2026	CPython 3.10	abi3	Linux glibc 2.17+ x86-64	Details
scraper_rust-0.10.0-cp310-abi3-manylinux_2_17_aarch64.manylinux2014_aarch64.whl 2.2 MB Sep 25, 2026	CPython 3.10	abi3	Linux glibc 2.17+ ARM64	Details
scraper_rust-0.10.0-cp310-abi3-macosx_11_0_arm64.whl 2.0 MB Sep 25, 2026	CPython 3.10	abi3	macOS 11.0+ ARM64	Details

Total release size: 8.7 MB

Release files / scraper_rust-0.10.0.tar.gz

Download URL	scraper_rust-0.10.0.tar.gz
Size	79.2 kB
Tags	Source
SHA-256 checksum How to use checksums	`a92a75aa88b21d191e5286b83bc7a6ccf554ca7a02697eaff3d6641500c59db6`
BLAKE2b-256 checksum How to use checksums	`10e8545710dfce9d7bf8c7adedb23f00d92f792cb36ac5ca8d027936a78703db`
Upload date	Sep 25, 2026
Uploaded using Trusted Publishing? What is trusted publishing?	Yes
Uploaded via	`twine/7.0.0 CPython/3.13.14`

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

Identity

Publishing platform: GitHub Actions
Publishing repository: github.com/RustedBytes/scraper-rs
Publishing commit: github.com/RustedBytes/scraper-rs/tree/d30989ea4a1f185f8c9d82e5ff102c7f1944f09e

Workflow

Publishing configuration: github.com/RustedBytes/scraper-rs/blob/d30989ea4a1f185f8c9d82e5ff102c7f1944f09e/.github/workflows/release.yml
Publishing logs: github.com/RustedBytes/scraper-rs/actions/runs/36147786582/attempts/1

Artifact

Subject: scraper_rust-0.10.0.tar.gz
SHA-256 checksum: a92a75aa88b21d191e5286b83bc7a6ccf554ca7a02697eaff3d6641500c59db6 Verifying attestations

Release files / scraper_rust-0.10.0-cp310-abi3-win_amd64.whl

Download URL	scraper_rust-0.10.0-cp310-abi3-win_amd64.whl
Size	2.1 MB
Tags	CPython 3.10 Windows x86-64 abi3
SHA-256 checksum How to use checksums	`2fa76a10e43eed7a11f4920c1cc2fad3b31836ee0cb539674eb41baab4296d70`
BLAKE2b-256 checksum How to use checksums	`8ba984064a4a31ae3ea56a79609f9c9da73b7cd443fce1c85cb287a26446c27d`
Upload date	Sep 25, 2026
Uploaded using Trusted Publishing? What is trusted publishing?	Yes
Uploaded via	`twine/7.0.0 CPython/3.13.14`

Provenance

Identity

Publishing platform: GitHub Actions
Publishing repository: github.com/RustedBytes/scraper-rs
Publishing commit: github.com/RustedBytes/scraper-rs/tree/d30989ea4a1f185f8c9d82e5ff102c7f1944f09e

Workflow

Publishing configuration: github.com/RustedBytes/scraper-rs/blob/d30989ea4a1f185f8c9d82e5ff102c7f1944f09e/.github/workflows/release.yml
Publishing logs: github.com/RustedBytes/scraper-rs/actions/runs/36147786582/attempts/1

Artifact

Subject: scraper_rust-0.10.0-cp310-abi3-win_amd64.whl
SHA-256 checksum: 2fa76a10e43eed7a11f4920c1cc2fad3b31836ee0cb539674eb41baab4296d70 Verifying attestations

Release files / scraper_rust-0.10.0-cp310-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl

Download URL	scraper_rust-0.10.0-cp310-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Size	2.3 MB
Tags	CPython 3.10 Linux glibc 2.17+ x86-64 abi3
SHA-256 checksum How to use checksums	`c167e546a9cefd0305325af4e6f2b80e533663a8fcd8428b5c774b3a78a9912d`
BLAKE2b-256 checksum How to use checksums	`6824833735a3d83007066dfd3d1a61749456e803a96e4ed2d2b511eb6785b47a`
Upload date	Sep 25, 2026
Uploaded using Trusted Publishing? What is trusted publishing?	Yes
Uploaded via	`twine/7.0.0 CPython/3.13.14`

Provenance

Identity

Publishing platform: GitHub Actions
Publishing repository: github.com/RustedBytes/scraper-rs
Publishing commit: github.com/RustedBytes/scraper-rs/tree/d30989ea4a1f185f8c9d82e5ff102c7f1944f09e

Workflow

Publishing configuration: github.com/RustedBytes/scraper-rs/blob/d30989ea4a1f185f8c9d82e5ff102c7f1944f09e/.github/workflows/release.yml
Publishing logs: github.com/RustedBytes/scraper-rs/actions/runs/36147786582/attempts/1

Artifact

Subject: scraper_rust-0.10.0-cp310-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
SHA-256 checksum: c167e546a9cefd0305325af4e6f2b80e533663a8fcd8428b5c774b3a78a9912d Verifying attestations

Release files / scraper_rust-0.10.0-cp310-abi3-manylinux_2_17_aarch64.manylinux2014_aarch64.whl

Download URL	scraper_rust-0.10.0-cp310-abi3-manylinux_2_17_aarch64.manylinux2014_aarch64.whl
Size	2.2 MB
Tags	CPython 3.10 Linux glibc 2.17+ ARM64 abi3
SHA-256 checksum How to use checksums	`64697a4f06df195a7957559a968ff1c158bdb9a2313ddd1ff04f6dad83e7edc6`
BLAKE2b-256 checksum How to use checksums	`25a889cdd6ac3579c6351f62fe44daf413ed4883bcfdbeb2f88d8b855142efd6`
Upload date	Sep 25, 2026
Uploaded using Trusted Publishing? What is trusted publishing?	Yes
Uploaded via	`twine/7.0.0 CPython/3.13.14`

Provenance

Identity

Publishing platform: GitHub Actions
Publishing repository: github.com/RustedBytes/scraper-rs
Publishing commit: github.com/RustedBytes/scraper-rs/tree/d30989ea4a1f185f8c9d82e5ff102c7f1944f09e

Workflow

Publishing configuration: github.com/RustedBytes/scraper-rs/blob/d30989ea4a1f185f8c9d82e5ff102c7f1944f09e/.github/workflows/release.yml
Publishing logs: github.com/RustedBytes/scraper-rs/actions/runs/36147786582/attempts/1

Artifact

Subject: scraper_rust-0.10.0-cp310-abi3-manylinux_2_17_aarch64.manylinux2014_aarch64.whl
SHA-256 checksum: 64697a4f06df195a7957559a968ff1c158bdb9a2313ddd1ff04f6dad83e7edc6 Verifying attestations

Release files / scraper_rust-0.10.0-cp310-abi3-macosx_11_0_arm64.whl

Download URL	scraper_rust-0.10.0-cp310-abi3-macosx_11_0_arm64.whl
Size	2.0 MB
Tags	CPython 3.10 abi3 macOS 11.0+ ARM64
SHA-256 checksum How to use checksums	`86f5389e21c865805c027f33a3f89043a2717a92934ee59140f724d1f41192ac`
BLAKE2b-256 checksum How to use checksums	`4e2d7dbf85d3301f4b973c0285c8addf6b50a04de2ecf76f7405afbe0855a4e9`
Upload date	Sep 25, 2026
Uploaded using Trusted Publishing? What is trusted publishing?	Yes
Uploaded via	`twine/7.0.0 CPython/3.13.14`

Provenance

Identity

Publishing platform: GitHub Actions
Publishing repository: github.com/RustedBytes/scraper-rs
Publishing commit: github.com/RustedBytes/scraper-rs/tree/d30989ea4a1f185f8c9d82e5ff102c7f1944f09e

Workflow

Publishing configuration: github.com/RustedBytes/scraper-rs/blob/d30989ea4a1f185f8c9d82e5ff102c7f1944f09e/.github/workflows/release.yml
Publishing logs: github.com/RustedBytes/scraper-rs/actions/runs/36147786582/attempts/1

Artifact

Subject: scraper_rust-0.10.0-cp310-abi3-macosx_11_0_arm64.whl
SHA-256 checksum: 86f5389e21c865805c027f33a3f89043a2717a92934ee59140f724d1f41192ac Verifying attestations