soup5ever

A BeautifulSoup 4 tree builder backed by Rust's html5ever. Same HTML5 trees as BeautifulSoup(html, "html5lib"), 6–14x faster.

from bs4 import BeautifulSoup
import soup5ever

soup = BeautifulSoup(html, "html5ever")

Installation

pip install soup5ever

Wheels for Linux (x86_64, aarch64; glibc and musl) and macOS (arm64), one abi3 wheel per platform for CPython 3.10+. The only dependency is beautifulsoup4>=4.13; no Rust, libxml2 or html5lib needed.

Usage

Importing soup5ever registers the "html5ever" parser (also "soup5ever"). The generic features "html5" and "html" are registered at the lowest priority, so importing it doesn't change what BeautifulSoup(markup, "html5") or BeautifulSoup(markup) select.

The usual options work (from_encoding, element_classes, multi_valued_attributes, store_line_numbers, attribute_dict_class). Like html5lib, parse_only isn't supported.

Compatibility

The target is BeautifulSoup(html, "html5lib"); where html5lib and the HTML standard disagree, soup5ever follows the standard.

• BS4's own tests for its html5lib builder: 93/93 pass.
• html5lib-tests tree construction: 1,588/1,592 match the spec. Against html5lib the trees are identical in 1,417 cases; the other 175 are classified automatically (167 html5lib behind the spec, 4 BS4 html5lib-adapter bugs, 4 <selectedcontent> misses both share).
• 182 targeted differential cases and ~23k fuzz inputs (triaged with Chromium): every remaining difference is an html5lib bug or listed below.

Differences from html5lib:

• Current HTML standard where html5lib 1.1 (2020) is behind: the 2025 <select> rules, <template> in <head>, </p> in SVG/MathML, <main>/<summary>, <search>, <dialog>, ruby. See tests/corpus.py for each case.
• BS4's html5lib adapter never applies the Noah's Ark clause (AttrList has no __eq__); soup5ever does.
• Undeclared, valid UTF-8 bytes are decoded as UTF-8 (html5lib guesses windows-1252 without chardet). Decoding follows the WHATWG Encoding Standard, not Python's codecs.
• Lone surrogates in a str become U+FFFD.
• sourceline/sourcepos match html5lib for ~98.5% of elements.

Shared with html5lib: no parse_only, no Script/Stylesheet string subclasses, no <selectedcontent> cloning.

Experimental: html5ever-experimental

soup = BeautifulSoup(html, "html5ever-experimental")

Builds the same trees as "html5ever", 10–30% faster, by creating BeautifulSoup's objects without running their constructors (Tag.__init__, NavigableString.__new__): the extension sets the attributes those would set, once each and already linked into the tree.

That relies on BeautifulSoup internals, verified against beautifulsoup4 4.13–4.15:

• The first parse checks that the installed BeautifulSoup still sets exactly the attributes soup5ever expects. If not, it warns once and falls back to the regular construction.
• Custom element_classes with their own __init__/__new__/__setattr__/setup, __getattribute__ or properties for the attributes BS4 sets, attribute_dict_classes with their own __setitem__, and builder subclasses that override how tags are set up get the regular construction automatically.
• What it can't detect: a BeautifulSoup release that keeps those attributes but changes what they mean.

The test suite compares every object's full state (vars() in order, value types, links) with "html5ever"'s over the whole corpus, the html5lib-tests and the fuzz inputs, and runs BS4's own builder tests against it. It is never selected through the generic "html5" or "html" features.

Benchmarks

BeautifulSoup(markup, parser) on an in-memory str, median of 7 rounds, 4-vCPU Linux VM, CPython 3.11:

Parsing in Rust is 14–32% of soup5ever's time; the rest is BeautifulSoup's own object constructors. html5ever-experimental skips most of the constructor work (it has its own column in the benchmark output). To reproduce:

uv pip install -e .[dev]
python benchmarks/run.py

Development

uv venv venv && source venv/bin/activate
uv pip install -e .[dev]                # builds the Rust extension
python scripts/fetch_upstream_tests.py  # BS4's builder tests + html5lib-tests
pytest

Re-run uv pip install -e .[dev] after changing Rust code.

edwh fmt && edwh lint                   # Python (ruff)
cargo fmt && cargo clippy               # Rust (lint levels in Cargo.toml)

pytest includes a 3000-input differential fuzz run; pytest --fuzz 20000 --fuzz-seed 7 runs a bigger one.

License

MIT