web2textpy

Python reimplementation of the Web2Text pipeline for labeling HTML DOM nodes as content or boilerplate using paired (raw_html, clean_text) data.

Installation

uv add web2textpy

Quick Start

from datasets import load_dataset
from web2text import run_pipeline

ds = load_dataset("williambrach/html-boilerplate-labeled", split="test")
row = ds[0]

tree, extracted_text, metrics = run_pipeline(row["html"], row["text"])

print(extracted_text[:200])
print(metrics)

Get text from labeled HTML

from web2text import extract_text_from_labeled_html
text = extract_text_from_labeled_html(labeled_html_string)

Step-by-Step API

Each stage of the pipeline is exposed as a standalone function:

from web2text import build_cdom, extract_leaves, align, label_nodes, extract_text, evaluate

# 1. Parse HTML into a collapsed DOM tree
tree = build_cdom(html_string)

# 2. Extract ordered text-bearing leaf nodes
leaves = extract_leaves(tree)  # [(element, "normalized text"), ...]

# 3. Align leaf texts against ground-truth clean text
scores = align(leaves, clean_text)  # {leaf_id: 0.0-1.0 match score}

# 4. Label each node as "content" or "boilerplate"
tree = label_nodes(tree, scores, threshold=0.667)

# 5. Extract text from content-labeled nodes
result = extract_text(tree)

# 6. Evaluate against ground truth
metrics = evaluate(result, clean_text)
# => {'token_f1': 0.99, 'precision': 0.99, 'recall': 0.99, 'rouge1_f': 0.99, 'bleu': 98.5, 'chrf': 98.8}

How the Matching Algorithm Works

Given raw HTML and its known clean text, the algorithm determines which DOM nodes are content versus boilerplate in six steps:

1. Simplify the DOM — strip non-content tags (<script>, <style>, etc.) and collapse single-child chains into a Collapsed DOM (CDOM) representation
2. Collect leaf text — walk the CDOM, concatenate text from every leaf node into one source string with tracked character offsets
3. Find anchors — identify 10-character substrings that appear exactly once in both the source and clean text, splitting the problem into independent segments
4. DP alignment — for each segment between anchors, run character-level dynamic programming with affine gap penalties to map source characters to clean-text characters
5. Score leaves — map alignment results back to leaf boundaries via stored offsets, giving each leaf a score: matched_chars / total_chars
6. Label nodes — leaves scoring above 0.667 are labeled "content", the rest "boilerplate", with labels propagating upward to parents

Dataset

Dataset: williambrach/html-boilerplate-labeled — ~4k pages from CleanEval, Dragnet, CETD, Readability, and others (3,985 pages total).

Sample counts in parentheses.

Original Work

• Paper: Vogels et al., "Web2Text: Deep Structured Boilerplate Removal" (ECIR 2018) — arxiv.org/abs/1801.02607
• Original implementation (Scala): github.com/dalab/web2text