soup5ever

A BeautifulSoup 4 tree builder backed by Rust's html5ever. Same HTML5 trees as BeautifulSoup(html, "html5lib"), 6–14x faster.

from bs4 import BeautifulSoup
import soup5ever

soup = BeautifulSoup(html, "html5ever")

Installation

pip install soup5ever

Wheels for Linux (x86_64, aarch64; glibc and musl) and macOS (arm64), one abi3 wheel per platform for CPython 3.10+. The only dependency is beautifulsoup4>=4.13; no Rust, libxml2 or html5lib needed.

Usage

Importing soup5ever registers the "html5ever" parser (also "soup5ever"). The generic features "html5" and "html" are registered at the lowest priority, so importing it doesn't change what BeautifulSoup(markup, "html5") or BeautifulSoup(markup) select.

The usual options work (from_encoding, element_classes, multi_valued_attributes, store_line_numbers, attribute_dict_class). Like html5lib, parse_only isn't supported.

Compatibility

The target is BeautifulSoup(html, "html5lib"); where html5lib and the HTML standard disagree, soup5ever follows the standard.

• BS4's own tests for its html5lib builder: 93/93 pass.
• html5lib-tests tree construction: 1,588/1,592 match the spec. Against html5lib the trees are identical in 1,417 cases; the other 175 are classified automatically (167 html5lib behind the spec, 4 BS4 html5lib-adapter bugs, 4 <selectedcontent> misses both share).
• 182 targeted differential cases and ~23k fuzz inputs (triaged with Chromium): every remaining difference is an html5lib bug or listed below.

Differences from html5lib:

• Current HTML standard where html5lib 1.1 (2020) is behind: the 2025 <select> rules, <template> in <head>, </p> in SVG/MathML, <main>/<summary>, <search>, <dialog>, ruby. See tests/corpus.py for each case.
• BS4's html5lib adapter never applies the Noah's Ark clause (AttrList has no __eq__); soup5ever does.
• Undeclared, valid UTF-8 bytes are decoded as UTF-8 (html5lib guesses windows-1252 without chardet). Decoding follows the WHATWG Encoding Standard, not Python's codecs.
• Lone surrogates in a str become U+FFFD.
• sourceline/sourcepos match html5lib for ~98.5% of elements.

Shared with html5lib: no parse_only, no Script/Stylesheet string subclasses, no <selectedcontent> cloning.

Benchmarks

BeautifulSoup(markup, parser) on an in-memory str, median of 7 rounds, 4-vCPU Linux VM, CPython 3.11:

Parsing in Rust is 14–32% of soup5ever's time; the rest is BeautifulSoup's own object constructors. To reproduce:

uv pip install -e .[dev]
python benchmarks/run.py

Development

uv venv venv && source venv/bin/activate
uv pip install -e .[dev]                # builds the Rust extension
python scripts/fetch_upstream_tests.py  # BS4's builder tests + html5lib-tests
pytest

Re-run uv pip install -e .[dev] after changing Rust code.

edwh fmt && edwh lint                   # Python (ruff)
cargo fmt && cargo clippy               # Rust (lint levels in Cargo.toml)

pytest includes a 3000-input differential fuzz run; pytest --fuzz 20000 --fuzz-seed 7 runs a bigger one.

License

MIT