peduncle

very very very simple DOM based HTML content extraction tool, get rid of boilerplate dressing of a web page[1].

easy but useable

work with python 3.7+

[1] the word comes from dragnet.

install

pip install peduncle

usage

import requests
from peduncle.peduncle import extract_text

# obtain the raw html
url="https://blog.rust-lang.org/2023/05/29/RustConf.html"
html = requests.get(url).text

# extract
print(extract_text(html))

benchmark

data

benchmark data comes from dragnet_data, which contains 1381 web pages.

result

• a: alpha, control how much the content extractor tens to extract larger content piece
• similarity: cosine similarity between sparse vectors of answer and extracted text
• 95hit rate: percentage of similarity larger than 95%
• length gap: extracted text length - answer text length
• std: std