scraper-rs

PyPI - Version Tests PyPI Downloads

Python bindings for high-performance Rust HTML parsing via PyO3. It provides a lightweight Document/Element API with CSS selectors backed by rustedbytes-tl, XPath (via xee-xpath), handy helpers, and zero Python-side parsing work.

Install

pip install scraper-rust

# or using uv
uv add scraper-rust

Quick start

from scraper_rs import Document, first, prettify, select, select_first, xpath

html = """
<html><body>
  <div class="item" data-id="1"><a href="/a">First</a></div>
  <div class="item" data-id="2"><a href="/b">Second</a></div>
</body></html>
"""

doc = Document(html)
print(doc.text)  # "First Second"

items = doc.select(".item")
print(items[0].attr("data-id"))  # "1"
print(items[0].to_dict())  # {"tag": "div", "text": "First", "html": "<a...>", ...}

first_link = doc.select_first("a[href]")  # alias: doc.find(...)
print(first_link.text, first_link.attr("href"))  # First /a
links_within_first_item = items[0].select("a[href]")
print([link.attr("href") for link in links_within_first_item])  # ["/a"]

# XPath (element results only)
xpath_items = doc.xpath("//div[@class='item']/a")
print([link.text for link in xpath_items])  # ["First", "Second"]
print(doc.xpath_first("//div[@data-id='1']/a").attr("href"))  # "/a"

# Functional helpers
links = select(html, "a[href]")
print([link.attr("href") for link in links])  # ["/a", "/b"]
print(first(html, "a[href]").text)  # First
print(select_first(html, "a[href]").text)  # First
print(
    [link.text for link in xpath(html, "//div[@class='item']/a")]
)  # ["First", "Second"]
print(prettify(html))

For runnable samples, see examples/demo.py and examples/demo_prettify_url.py. Quick URL prettify demo:

python examples/demo_prettify_url.py https://example.com --max-lines 80

Async usage

The scraper_rs.asyncio module exposes an async-first surface for coroutine code. AsyncDocument stores shareable HTML/text state instead of a thread-affine sync Document, and all selectors are awaitable for consistent async calling style:

import asyncio
from scraper_rs import asyncio as scraping_async

html = "<div class='item'><a href='/a'>First</a></div>"


async def main():
    async with await scraping_async.parse(html) as doc:
        items = await doc.select(".item")
        first_link = await items[0].select_first("a[href]")
        print(first_link.text)  # First

    links = await scraping_async.select(html, "a[href]")
    print([link.attr("href") for link in links])  # ["/a"]


asyncio.run(main())

All async functions accept the same keyword arguments as their sync counterparts (max_size_bytes, truncate_on_limit, etc.). CPU-bound async parsing and selection work is offloaded through the Rust async core onto Rayon worker threads so concurrent coroutines do not run that work on the event-loop thread. AsyncDocument supports async with for automatic cleanup in coroutine code, and AsyncElement / AsyncDocument both expose async .prettify() helpers.

Large documents and memory safety

To avoid runaway allocations, parsing defaults to a 1 GiB cap. Pass max_size_bytes to override:

from scraper_rs import Document, select

doc = Document(html, max_size_bytes=5_000_000)  # 5 MB guard
links = select(html, "a[href]", max_size_bytes=5_000_000)

If you want to parse a limited portion of an oversized HTML document instead of rejecting it entirely, use truncate_on_limit=True:

# Parse only the first 100KB of a large HTML document
doc = Document(large_html, max_size_bytes=100_000, truncate_on_limit=True)
links = doc.select("a[href]")  # Will only find links in the first 100KB

# Also works with top-level functions
items = select(large_html, ".item", max_size_bytes=100_000, truncate_on_limit=True)

Note: Truncation happens at valid UTF-8 character boundaries to prevent encoding errors.

API highlights

  • Document(html: str) / Document.from_html(html) parse HTML for CSS and keep the DOM; XPath parsing is initialized lazily on first XPath query.
  • .select(css) → list[Element], .select_first(css) / .find(css) → first Element | None, .css(css) is an alias.
  • .xpath(expr) / .xpath_first(expr) evaluate XPath expressions that return element nodes.
  • .prettify() renders the current DOM as an indented string for readable output/debugging.
  • .text returns normalized text; Document.html is the original input HTML; Element.html is inner HTML.
  • scraper_rs.asyncio exposes async parse/select/xpath wrappers plus awaitable AsyncDocument / AsyncElement methods.
  • Element exposes .tag, .text, .html, .attrs plus helpers .attr(name), .get(name, default), .to_dict().
  • Elements support nested CSS and XPath selection via .select(css), .select_first(css), .find(css), .css(css), .xpath(expr), .xpath_first(expr).
  • Elements also expose .prettify() to format element HTML with indentation.
  • Top-level helpers mirror the class methods: parse(html), prettify(html), select(html, css), select_first(html, css) / first(html, css), xpath(html, expr), xpath_first(html, expr).
  • Parse-tree helpers parse_document(html) and parse_fragment(html) return rustedbytes-tl backed structural dictionaries for callers that need node data.
  • CSS selection uses the native rustedbytes-tl selector subset: tag, id, class, *, attribute selectors, comma selectors, descendants, and direct children.
  • max_size_bytes lets you fail fast on oversized HTML; defaults to a 1 GiB limit.
  • truncate_on_limit allows parsing a truncated version (limited to max_size_bytes) of oversized HTML instead of raising an error.
  • Call doc.close() (or with Document(html) as doc: ...) to free parsed DOM resources when you're done.
  • In async workflows, use async with await scraper_rs.asyncio.parse(html) as doc: ... for automatic AsyncDocument cleanup.

Installation

Built wheels target abi3 (CPython 3.10+). To build locally:

# Install maturin (uv is used in this repo, but pip works too)
pip install maturin

# Build a wheel
maturin build --release --compatibility linux

# Install the generated wheel
pip install target/wheels/scraper_rust-*.whl

If you have just installed, the repo includes helpers: just build (local wheel), just install-wheel (install the built wheel), and just build_manylinux (via the official maturin Docker image).

Projects Using scraper-rs

Development

Requirements: Rust toolchain, Python 3.10+, uv, maturin, pytest, and pytest-asyncio for tests.

  • Run tests: just test or uv run pytest tests/
  • Format code: just fmt (or cargo fmt --all and uv run ruff format)
  • Lint Rust: just lint (or cargo clippy --all-targets --all-features -- -D warnings)
  • The PyO3 module name is scraper_rs; the Rust crate is built as cdylib.

Contributions and issues are welcome. If you add public API, please extend the relevant tests, type stubs, docs, and examples accordingly.

Project links

Data verified by PyPI on Jul 27, 2026
Data provided by the project maintainers, verified at the time the release was uploaded to PyPI.

Key dates

PyPI data
Data sourced directly from PyPI's database.
  • Released:
Latest release

1 maintainer

PyPI data
Data sourced directly from PyPI's database.
Avatar for Egor.Smolyakov from gravatar.com Egor.Smolyakov

Credits

Author: Yehor Smoliakov

GitHub Statistics

Data verified by PyPI on Jul 27, 2026
The GitHub source repository was provided by the project maintainers and verified by PyPI at the time of upload. Stars, forks, and open issues/PRs are derived from that repository and have not been independently verified.

Requires

Python >=3.10

Provides Extra

dev

Tags

html scraping css selectors xpath pyo3 rust

Classifiers

Report project as malware

Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

scraper_rust-0.7.0.tar.gz (77.4 kB view details)

Uploaded Source

Built Distributions

If you're not sure about the file name format, learn more about wheel file names.

scraper_rust-0.7.0-cp310-abi3-win_amd64.whl (2.1 MB view details)

Uploaded CPython 3.10+Windows x86-64

scraper_rust-0.7.0-cp310-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (2.3 MB view details)

Uploaded CPython 3.10+manylinux: glibc 2.17+ x86-64

scraper_rust-0.7.0-cp310-abi3-manylinux_2_17_aarch64.manylinux2014_aarch64.whl (2.2 MB view details)

Uploaded CPython 3.10+manylinux: glibc 2.17+ ARM64

scraper_rust-0.7.0-cp310-abi3-macosx_11_0_arm64.whl (2.0 MB view details)

Uploaded CPython 3.10+macOS 11.0+ ARM64

File details

Details for the file scraper_rust-0.7.0.tar.gz.

File metadata

  • Download URL: scraper_rust-0.7.0.tar.gz
  • Upload date:
  • Size: 77.4 kB
  • Tags: Source
  • Uploaded using Trusted Publishing? Yes
  • Uploaded via: twine/6.1.0 CPython/3.13.14

File hashes

Hashes for scraper_rust-0.7.0.tar.gz
Algorithm Hash digest
SHA256 a17242f19900c471d82ff4ba69fe932e18fc1b11cd78c9d0e21d267b638517d9
MD5 08531fff10774f0fe7fc78b5628ad759
BLAKE2b-256 3c8d066c3ee6af1db71ddac2a1b7bc3f81727f1f135c2fb6aa314497b693ea64

See more details on using hashes here.

Provenance

The following attestation bundles were made for scraper_rust-0.7.0.tar.gz:

Publisher: release.yml on RustedBytes/scraper-rs

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

File details

Details for the file scraper_rust-0.7.0-cp310-abi3-win_amd64.whl.

File metadata

  • Download URL: scraper_rust-0.7.0-cp310-abi3-win_amd64.whl
  • Upload date:
  • Size: 2.1 MB
  • Tags: CPython 3.10+, Windows x86-64
  • Uploaded using Trusted Publishing? Yes
  • Uploaded via: twine/6.1.0 CPython/3.13.14

File hashes

Hashes for scraper_rust-0.7.0-cp310-abi3-win_amd64.whl
Algorithm Hash digest
SHA256 55aa24a60bfac851b3daa241636fd327e9074f555f40f9eb227d8fd594037497
MD5 c8e4af832e91373d1a8929fc5596e610
BLAKE2b-256 3a0cffb1022151b9eabd137d2c0a10acc2b83236eae820ec9c2fc127c3442e53

See more details on using hashes here.

Provenance

The following attestation bundles were made for scraper_rust-0.7.0-cp310-abi3-win_amd64.whl:

Publisher: release.yml on RustedBytes/scraper-rs

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

File details

Details for the file scraper_rust-0.7.0-cp310-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.

File metadata

File hashes

Hashes for scraper_rust-0.7.0-cp310-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm Hash digest
SHA256 42db5e0c0668f064eec1402184dff42f34ad04766b52b96ed547b10677205029
MD5 8f6c97c4dcf8addce17f400e38c4d74e
BLAKE2b-256 c2989b1550b7d483679804eca3f6ecb3b911f3deebc0bfada48caa26742b4e16

See more details on using hashes here.

Provenance

The following attestation bundles were made for scraper_rust-0.7.0-cp310-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl:

Publisher: release.yml on RustedBytes/scraper-rs

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

File details

Details for the file scraper_rust-0.7.0-cp310-abi3-manylinux_2_17_aarch64.manylinux2014_aarch64.whl.

File metadata

File hashes

Hashes for scraper_rust-0.7.0-cp310-abi3-manylinux_2_17_aarch64.manylinux2014_aarch64.whl
Algorithm Hash digest
SHA256 3d47c73fdd3268f9848e4b6bed1993ec65683430ecfd6371135867cb1d84c476
MD5 f7128d0f58432177263de40fd9e5e86c
BLAKE2b-256 389ec0cfcc91255df6937c8583ab1223a3d747d01c011cf958366ff7be29e9a4

See more details on using hashes here.

Provenance

The following attestation bundles were made for scraper_rust-0.7.0-cp310-abi3-manylinux_2_17_aarch64.manylinux2014_aarch64.whl:

Publisher: release.yml on RustedBytes/scraper-rs

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

File details

Details for the file scraper_rust-0.7.0-cp310-abi3-macosx_11_0_arm64.whl.

File metadata

File hashes

Hashes for scraper_rust-0.7.0-cp310-abi3-macosx_11_0_arm64.whl
Algorithm Hash digest
SHA256 0e9b3ba528f05a46718e818e283f18c32a0e504a6dffdd40ae7a774f8819d5db
MD5 3d48666811011248a396b844543b2208
BLAKE2b-256 d635302971c9d205d678db9e40ff9b8f98dd3b939206aa0f2c87380362806714

See more details on using hashes here.

Provenance

The following attestation bundles were made for scraper_rust-0.7.0-cp310-abi3-macosx_11_0_arm64.whl:

Publisher: release.yml on RustedBytes/scraper-rs

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

Release history Release notifications | RSS feed

This release

0.7.0 This release

5 files

0.6.0

5 files

0.5.1

5 files

0.5.0

5 files

0.4.3

7 files

0.4.2

7 files

0.4.1

10 files

0.4.0

10 files

0.3.3

10 files

0.3.2

10 files

0.3.1

10 files

0.3.0

10 files

0.2.32

10 files

0.2.31

10 files

0.2.29

10 files

0.2.28

10 files

0.2.27

10 files

0.2.22

10 files

0.2.21

10 files

0.2.20

10 files

0.2.19

8 files

0.2.17

6 files

0.2.16

4 files

0.2.14

4 files

0.2.13

2 files

0.2.12

2 files

0.2.11

2 files

0.2.10

2 files

0.2.9

2 files

0.2.8

2 files

0.2.7

2 files

0.2.6

2 files

0.2.5

2 files

0.2.4

2 files

0.2.3

2 files

0.2.2

2 files

0.2.1

2 files

0.1.3

2 files

Supported by

AWS Cloud computing and Security Sponsor Datadog Monitoring Depot Continuous Integration Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page