turbohtml 0.4.0

A fast, fully typed HTML toolkit for Python, powered by a C-accelerated core.

turbohtml

A fast, fully typed HTML toolkit for Python with a C-accelerated core. turbohtml escapes and unescapes HTML to match the standard library byte for byte, tokenizes markup with a WHATWG-conformant streaming tokenizer, and parses whole documents into a navigable element tree you query with CSS selectors, edit in place, build from scratch, and serialize back to conformant HTML. Each operation runs several times faster than its pure-Python counterpart and supports the free-threaded build.

Install

$ pip install turbohtml

Wheels ship per interpreter for CPython 3.10–3.15 (including free-threading), so there is nothing to compile.

Usage

Escape text before interpolating it into HTML so it cannot break out of its context:

import turbohtml

print(turbohtml.escape('<a href="?x=1&y=2">Tom & Jerry</a>'))
# &lt;a href=&quot;?x=1&amp;y=2&quot;&gt;Tom &amp; Jerry&lt;/a&gt;

Inside a text node the quotes are safe, so pass quote=False to keep the output smaller:

print(turbohtml.escape('He said "hi" & left', quote=False))
# He said "hi" & left

Turn HTML character references back into text, following the full HTML5 rules (named, numeric, and longest-match references that omit the trailing semicolon):

print(turbohtml.unescape("café & résumé 🎉"))
# café & résumé 🎉

escape and unescape reproduce html.escape and html.unescape exactly, so turbohtml is a drop-in replacement on hot paths.

Tokenize markup into a stream of tokens that follows the WHATWG tokenization algorithm:

for token in turbohtml.tokenize('<p class="x">Tom &amp; Jerry</p>'):
    print(token.type.name, token.tag or token.data, token.attrs)
# START_TAG p [('class', 'x')]
# TEXT Tom & Jerry None
# END_TAG p []

For incremental input, Tokenizer.feed() returns the tokens completed by each chunk and close() flushes the rest:

tokenizer = turbohtml.Tokenizer()
print([token.tag for token in tokenizer.feed("<div><sp")])  # ['div']
print([token.tag for token in tokenizer.feed("an>")])  # ['span']
print(list(tokenizer.close()))  # []

Parse a whole document into a tree and walk it with find, find_all, and the navigation accessors:

doc = turbohtml.parse("<ul><li>one<li>two</ul>")
print([li.text for li in doc.find_all("li")])  # ['one', 'two']
print(doc.find("ul").children[0].tag)  # li

Query with a CSS selector, and serialize a node back to HTML with the escaping you choose:

from turbohtml import Formatter

doc = turbohtml.parse("<article><h1>Tea</h1><p class=note>café &amp; cake</p></article>")
print(doc.select_one("p.note").text)
# café & cake
print(doc.select_one("p").serialize(formatter=Formatter.NAMED_ENTITIES))
# <p class="note">caf&eacute; &amp; cake</p>

Pass bytes to sniff the encoding the WHATWG way (byte-order mark, then a <meta> declaration):

doc = turbohtml.parse(b'<meta charset="iso-8859-2"><p>\xe1</p>')
print((doc.encoding, doc.find("p").text))  # ('iso-8859-2', 'á')

Parse a fragment as the contents of a context element, the way innerHTML does:

cell = turbohtml.parse_fragment("<td>data", context="tr")
print((cell.tag, cell.text))  # ('tr', 'data')

Build a tree from scratch with the node constructors, then assemble it (a list value for a token-list attribute like class joins on a space, and the text setter fills an element with a single text child):

from turbohtml import Element

card = Element("article", {"class": ["card", "lg"]})
heading = Element("h2")
heading.text = "Tea"
card.append(heading)
print(card.html)
# <article class="card lg"><h2>Tea</h2></article>

Edit a parsed tree in place. unwrap, decompose, wrap, insert_before, replace_with, and the rest move nodes within a tree or adopt them from another, and element.attrs is a live mapping you assign to:

doc = turbohtml.parse("<p>keep <b>bold</b> <span>drop</span></p>")
doc.find("b").unwrap()
doc.find("span").decompose()
doc.find("p").attrs["class"] = "lead"
print(doc.find("p").html)
# <p class="lead">keep bold </p>

The sealed node hierarchy (Element, Text, Comment, Doctype, ProcessingInstruction, CData, and Document) sets __match_args__ for structural pattern matching, and any node deep-copies with copy.copy, copy.deepcopy, or pickle.

Performance

turbohtml's C core makes every operation several times faster than its pure-Python counterpart, and it runs faster than the other C libraries on the read-path benchmarks. Measured with pyperf on an Apple M4:

• escape and unescape match the standard library byte for byte while running several times faster, up to 22× on no-op text and 13× on entity-dense input.
• tokenize is 9–16× faster than html.parser wherever markup appears.
• parse builds a full WHATWG tree 2–5× faster than the C parsers lxml and selectolax, and 30–80× faster than the pure-Python BeautifulSoup and html5lib.
• find_all and CSS select run 2–40× faster than lxml's C XPath and cssselect at every size and 100× faster than BeautifulSoup.
• serializing a tree back to HTML runs 2–4× faster than lxml and selectolax and about 40× faster than BeautifulSoup.
• building a tree from scratch and editing a parsed one both run about twice as fast as lxml and an order of magnitude faster than BeautifulSoup.

See the performance page for the full sectioned tables and the methodology.

Documentation

Full documentation, including tutorials, how-to guides, migration guides from BeautifulSoup, lxml, selectolax, html5lib, and the standard library, the API reference, and the design rationale, lives at turbohtml.readthedocs.io.

License

turbohtml is released under the MIT license.