WhiskeySour

A high-performance drop-in replacement for Python's BeautifulSoup, written in Rust and published as a native Python package via PyO3.

Status: Beta — core implementation complete. 450 unit tests passing, 508 including integration tests.

Why WhiskeySour?

BeautifulSoup is beloved but slow. Every node is a Python object (~500 bytes), parsing is GIL-bound, and CSS selectors re-parse on every call. WhiskeySour fixes this at the foundation.

All numbers below are medians from a dev build (maturin develop). Release builds (maturin develop --release) are typically 2–3× faster still.

Key implementation choices:

• Rust core via PyO3 + maturin
• html5ever — spec-compliant HTML5 parser (same as Firefox/Chrome)
• Arena allocation — compact ~40 byte/node layout vs ~500 bytes in bs4
• cssparser — CSS selectors compiled to DFA, LRU-cached
• GIL release — all Rust tree operations run outside the Python GIL
• memchr — SIMD byte scanning (SSE2 / AVX2 / NEON)

API — drop-in compatible with BeautifulSoup

from whiskeysour import WhiskeySour

# Drop-in replacement for BeautifulSoup — no parser argument needed
soup = WhiskeySour(html)

# All standard bs4 operations work identically:
soup.find("h1")
soup.find_all("a", class_="external")
soup.select("div.container > p:first-child")
soup.title.string
soup.find(id="main").get_text(strip=True)

# BS4-compatible NavigableString (name is None, exactly like bs4)
for child in tag.children:
    if child.name:          # None for text nodes, str for elements — same as bs4
        print(child.name)

# Drop-in alias
from whiskeysour import BeautifulSoup   # same class, different name

WhiskeySour extensions (not in bs4)

# Pre-compiled CSS selector — zero parse overhead on repeated use
q = soup.compile("div.item > a[href]")
for doc in documents:
    results = q.select(doc)

# Streaming parser — feed chunks incrementally
from whiskeysour import StreamParser, parse_stream

with StreamParser() as parser:
    for chunk in response.iter_content(4096):
        parser.feed(chunk)
soup = parser.close()

# Generator-style streaming with automatic extraction
import io
with open("large.html", "rb") as f:
    for article in parse_stream(f, selector="article.post"):
        print(article.find("h1").get_text())

BS4 compatibility notes

WhiskeySour is a faithful drop-in for the vast majority of BeautifulSoup code. A handful of behaviours differ due to html5ever's spec compliance:

The first two rows are identical; the remaining differences only affect malformed HTML.

Project Structure

WhiskeySour/
├── Cargo.toml                  # Rust workspace root
├── pyproject.toml              # maturin build config
├── pytest.ini                  # test configuration
│
├── crates/
│   ├── whiskeysour-core/        # Pure Rust library (no Python deps)
│   │   └── src/
│   │       ├── parser/         # html5ever integration
│   │       ├── node.rs         # Arena-allocated node pool
│   │       ├── selector/       # CSS selector DFA + LRU cache
│   │       ├── traversal/      # Tree iterators
│   │       ├── query/          # find() / find_all() / select()
│   │       └── serialize/      # HTML serialisation + prettify
│   │
│   └── whiskeysour-py/          # PyO3 bindings layer
│       └── src/
│           └── lib.rs          # _Tag, _Document Python classes
│
├── python/
│   └── whiskeysour/
│       ├── __init__.py         # Public API + BeautifulSoup alias
│       └── _core.pyi           # Type stubs for Rust extension
│
└── tests/
    └── python/
        ├── conftest.py
        ├── unit/               # 450 tests across 10 files
        ├── integration/        # bs4 API parity tests (58 tests)
        ├── performance/        # pytest-benchmark suites + comparison report
        └── fuzz/               # Hypothesis property tests (16 tests)

Quick start

# Prerequisites: Python 3.9+, Rust toolchain
curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh

# Create virtual environment
python3 -m venv .venv
source .venv/bin/activate        # macOS / Linux

# Install dependencies
pip install maturin pytest pytest-benchmark hypothesis beautifulsoup4

# Build the Rust extension
maturin develop                  # dev build (fast to compile)
# maturin develop --release      # optimised (use for benchmarks)

# Run tests
pytest tests/python/unit/

Running tests

# All unit tests (fastest, no extra deps needed)
pytest tests/python/unit/ -q

# Single test file
pytest tests/python/unit/test_parsing.py -v

# Integration tests (requires beautifulsoup4)
pytest tests/python/integration/ -v

# Skip slow tests (large documents, deep nesting)
pytest -m "not slow"

# Fuzz / property-based tests (requires hypothesis)
pytest tests/python/fuzz/ -v

# Benchmark suites (requires pytest-benchmark)
pytest tests/python/performance/ --benchmark-only -v

# Performance comparison report (WhiskeySour vs BeautifulSoup)
python tests/python/performance/bench_comparison.py
python tests/python/performance/bench_comparison.py --fixture small --rounds 50
python tests/python/performance/bench_comparison.py --output /tmp/report.html
open bench_report.html

Test file overview

Test markers

Development

# Rust checks (no build required)
~/.cargo/bin/cargo check -p whiskeysour-py

# Dev build (fast recompile, debug symbols)
maturin develop

# Release build (use for perf work)
maturin develop --release

# Rust tests
cargo test

# Formatting / linting
cargo fmt
cargo clippy

Building wheels

maturin build --release
maturin build --release --interpreter python3.9 python3.10 python3.11 python3.12 python3.13
maturin publish

Contributing

1. All changes must be accompanied by tests
2. Run pytest tests/python/unit/ -m "not slow" before submitting
3. Run cargo fmt and cargo clippy for Rust changes
4. Performance regressions > 5% against the baseline will block merge

Licence

MIT