dom-tree-rs for Python

Parse messy real-world HTML, query it with CSS selectors, and get readable text or Markdown back — fast, typed, and it never chokes on bad markup. Python bindings for the dom-tree-rs Rust crate.

pip install dom-tree-rs

import domtree

doc = domtree.parse("<ul><li class='x'>a</li><li>b</li></ul>")
[li.text for li in doc.select("li")]   # ['a', 'b']
doc.select_first("li.x").html          # '<li class="x">a</li>'

The distribution is called dom-tree-rs (the name domtree was taken on PyPI); the import name is domtree. Wheels are abi3: one wheel per platform works on every CPython ≥ 3.9 (Linux x86_64/aarch64, macOS Intel/Apple Silicon, Windows).

Scraping

import domtree
import urllib.request

html = urllib.request.urlopen("https://example.com/shop").read().decode()
doc = domtree.parse(html)

products = [
    {
        "name": card.select_first(".title").text,
        "price": card.select_first(".price").text,
        "url": card.select_first("a.buy").attr("href"),
    }
    for card in doc.select("div.card")
]
# [{'name': 'Widget', 'price': '$9.99', 'url': '/buy/1'}, ...]

# Everything a selector engine should do, without a browser:
doc.select("ul > li:nth-child(odd)")
doc.select_first(".card:not(.featured) a[href^='/buy']")
doc.select_first("div:has(> h2:contains('Gadget'))")

# Navigate from any element.
price = doc.select_first("p.price")
price.parent.attrs          # {'class': 'card featured', 'data-id': '1'}
price.closest(".card")      # nearest matching ancestor (or itself)
price.parent.children       # child elements

HTML → Markdown for LLMs and RAG

Feeding raw HTML to a model wastes tokens on markup, scripts and navigation. Pick the content you want and convert it to clean Markdown (CommonMark + GFM: headings, lists, links, images, code blocks, tables, quotes):

doc = domtree.parse(html)

article = doc.select_first("article") or doc.root
markdown = article.to_markdown()
# # Release notes
#
# Version **2.0** is out. See the [docs](/docs).
#
# - Faster
# - Smaller
#
# | OS | Status |
# | --- | --- |
# | Linux | ok |

prompt = f"Summarize this page:\n\n{markdown}"

<script>, <style> and <head> never leak into the output. For plain text instead (a line per block, a blank line between paragraphs) use .text / to_text().

API

domtree.parse(html: str) -> Document

Never raises on malformed HTML — the parser recovers like a browser would (implicit </p>/</li>, stray and mis-nested end tags, …) and records what it recovered from in Document.errors.

An invalid or unsupported selector raises ValueError with the reason and position (invalid selector 'a:hover': unsupported pseudo-class ':hover' at byte 1). Supported: type, .class, #id, *, attribute selectors ([a], =, ~=, |=, ^=, $=, *=, i flag), all four combinators, :nth-child()/:nth-of-type() and friends, :first-child, :only-child, :empty, :root, :scope, :not(), :is(), :where(), :has(), :contains(), :checked, :disabled, … — see the main README for the full list.

Elements keep their document alive, compare equal when they are the same node, and are hashable. Documents and elements are immutable, so they are safe to share between threads; large inputs are parsed and serialized with the GIL released, so a thread pool parses pages in parallel.

The package ships type stubs (py.typed), so editors and type checkers know every signature.

Development

cd python
python -m venv .venv && . .venv/bin/activate
pip install maturin pytest
maturin develop --release
pytest

License

MIT