WhiskeySour

A high-performance drop-in replacement for Python's BeautifulSoup, written in Rust and published as a native Python package via PyO3.

Status: stable — 561 unit, 58 BS4-parity integration and 16 property-based fuzz tests passing.

Why WhiskeySour?

BeautifulSoup is beloved but slow: every node is a Python object and all parsing and matching runs in Python. WhiskeySour keeps the same API and moves the tree, the parser and the query engine into Rust.

Measured with a release build (maturin develop --release) against bs4 4.15 + html.parser, Python 3.14, Apple Silicon. Query rows use the 100 KB document; each figure is the best of three medians.

¹ bs4's tree is a tree of Python objects, so walking it node by node or reading one attribute is a plain attribute/dict access. WhiskeySour has to cross into Rust and build a Python wrapper for each node it hands back (the Rust walk itself takes ~2.4 ns per node). When you need many nodes, let Rust do the filtering — find_all, select, get_text(), find_all(string=…) — rather than iterating .descendants / .strings in Python.

Regenerate this table with python tests/python/performance/readme_table.py and the memory row with pytest tests/python/performance/bench_memory.py -s. For an HTML report across five document shapes, run python tests/python/performance/bench_comparison.py.

Key implementation choices:

• Rust core via PyO3 + maturin
• html5ever — spec-compliant HTML5 parser (from the Servo project)
• Compact arena — the whole tree is one flat Vec of 56-byte nodes addressed by u32 ids (element data is boxed, links are 4 bytes); traversal follows index links without allocating
• Hand-written CSS selector engine — parsed selectors are cached per thread, and :nth-* indices are memoised per query so long lists stay linear
• GIL release — parsing, and queries or serialisation over large subtrees, run outside the Python GIL, so threads can work on documents concurrently

API — drop-in compatible with BeautifulSoup

from whiskeysour import WhiskeySour

# Drop-in replacement for BeautifulSoup — no parser argument needed
soup = WhiskeySour(html)

# All standard bs4 operations work identically:
soup.find("h1")
soup.find_all("a", class_="external")
soup.select("div.container > p:first-child")
soup.title.string
soup.find(id="main").get_text(strip=True)

# BS4-compatible NavigableString (name is None, exactly like bs4)
for child in tag.children:
    if child.name:          # None for text nodes, str for elements — same as bs4
        print(child.name)

# Drop-in alias
from whiskeysour import BeautifulSoup   # same class, different name

WhiskeySour extensions (not in bs4)

# Reusable CSS selector — the parsed selector is cached, so repeated use skips re-parsing
q = soup.compile("div.item > a[href]")
for doc in documents:
    results = q.select(doc)

# Streaming parser — feed chunks incrementally
from whiskeysour import StreamParser, parse_stream

with StreamParser() as parser:
    for chunk in response.iter_content(4096):
        parser.feed(chunk)
soup = parser.close()

# Generator-style streaming with automatic extraction
import io
with open("large.html", "rb") as f:
    for article in parse_stream(f, selector="article.post"):
        print(article.find("h1").get_text())

BS4 compatibility notes

WhiskeySour is a faithful drop-in for the vast majority of BeautifulSoup code. A handful of behaviours differ due to html5ever's spec compliance:

The first two rows are identical; the remaining differences only affect malformed HTML.

Project Structure

WhiskeySour/
├── Cargo.toml                  # Rust workspace root
├── pyproject.toml              # maturin build config
├── pytest.ini                  # test configuration
│
├── crates/
│   ├── whiskysour-core/         # Pure Rust library (no Python deps)
│   │   └── src/
│   │       ├── parser/         # html5ever integration
│   │       ├── node.rs         # Arena-allocated node pool
│   │       ├── selector/       # CSS selector parser, matcher + per-thread cache
│   │       ├── traversal/      # Tree iterators
│   │       ├── query/          # find() / find_all() / select()
│   │       └── serialize/      # HTML serialisation + prettify
│   │   └── benches/            # Criterion benchmarks (parse, find, select, serialize)
│   │
│   └── whiskysour-py/           # PyO3 bindings layer
│       └── src/
│           └── lib.rs          # _Tag, _Document Python classes
│
├── python/
│   └── whiskeysour/
│       ├── __init__.py         # Public API + BeautifulSoup alias
│       └── _core.pyi           # Type stubs for Rust extension
│
└── tests/
    └── python/
        ├── conftest.py
        ├── unit/               # 561 tests across 11 files
        ├── integration/        # bs4 API parity tests (58 tests)
        ├── performance/        # pytest-benchmark suites + comparison report
        └── fuzz/               # Hypothesis property tests (16 tests)

Quick start

# Prerequisites: Python 3.9+, Rust toolchain
curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh

# Create virtual environment
python3 -m venv .venv
source .venv/bin/activate        # macOS / Linux

# Install dependencies
pip install maturin pytest pytest-benchmark hypothesis beautifulsoup4

# Build the Rust extension
maturin develop                  # dev build (fast to compile)
# maturin develop --release      # optimised (use for benchmarks)

# Run tests
pytest tests/python/unit/

Running tests

# All unit tests (fastest, no extra deps needed)
pytest tests/python/unit/ -q

# Single test file
pytest tests/python/unit/test_parsing.py -v

# Integration tests (requires beautifulsoup4)
pytest tests/python/integration/ -v

# Skip slow tests (large documents, deep nesting)
pytest -m "not slow"

# Fuzz / property-based tests (requires hypothesis)
pytest tests/python/fuzz/ -v

# Benchmark suites (requires pytest-benchmark; build with --release first).
# Includes per-item navigation benchmarks and a 4-thread scaling check.
pytest tests/python/performance/ --override-ini="addopts=" -q

# Memory usage vs bs4 (peak RSS in a fresh process; POSIX only)
pytest tests/python/performance/bench_memory.py --override-ini="addopts=" -q -s

# Rust-level Criterion benchmarks (parse, find_all/find_one, tree walk,
# get_text, select incl. :nth-*, serialize/prettify)
cargo bench -p whiskysour-core

# Performance comparison report (WhiskeySour vs BeautifulSoup)
python tests/python/performance/bench_comparison.py
python tests/python/performance/bench_comparison.py --fixture small --rounds 50
python tests/python/performance/bench_comparison.py --output /tmp/report.html
open bench_report.html

Test file overview

Test markers

Development

# Rust checks (no build required)
~/.cargo/bin/cargo check -p whiskysour-py

# Dev build (fast recompile, debug symbols)
maturin develop

# Release build (use for perf work)
maturin develop --release

# Rust tests
cargo test

# Formatting / linting (clippy warnings are errors)
cargo fmt
cargo clippy --all-targets -- -D warnings

Building wheels

maturin build --release
maturin build --release --interpreter python3.9 python3.10 python3.11 python3.12 python3.13
maturin publish

Contributing

1. All changes must be accompanied by tests
2. Run pytest tests/python/unit/ -m "not slow" before submitting
3. Run cargo fmt and cargo clippy --all-targets -- -D warnings for Rust changes
4. Performance regressions > 5% against the baseline will block merge — benchmark with a release build

Licence

MIT