Haruka Parser
A simple HTML Parser
Install
pip install haruka-parser
Usage
V2（推荐）
V2 是多引擎投票的抽取管线：trafilatura 与 readability 两路主内容启发式对段落投票， 只保留双引擎都命中的正文块；在此基础上叠加 math/code/table 的结构化转换与块级去重。
from haruka_parser.v2.extractors.auto_extractor import AutoExtractor
extractor = AutoExtractor()
result = extractor.extract(html, url="https://example.com/page.html")
print(result["content"]) # Markdown 正文
print(result["title"]) # 标题
print(result["time"]) # 发布时间 "2024-05-11 08:30:00"（无则 ""）
print(result["time_stamp"]) # 对应 unix timestamp（字符串，无则 ""）
返回字段：
| 字段 | 说明 |
|---|---|
content |
Markdown 格式正文 |
paragraphs |
段落结构（含 unique_id、tag、dom meta），可回放 |
dom_attrs / links |
DOM 属性与页面链接（链接按 base_url 补全） |
title / time / time_stamp |
标题、发布时间及其 timestamp |
base_url / encoding |
最终基准 URL 与检测到的编码 |
常用参数：
extractor.extract(
html,
url="https://example.com/page.html", # 用于链接补全、域名特判、标题回退
separator="\n\n", # 段落分隔符
table_format="github", # github / grid / simple / html / html_raw / html_auto
escape_dollars=False, # 非 LaTeX 上下文的 $ 转义
use_ftfy=False, # ftfy 文本修复
magic_html_heuristic=False, # 引入 magic-html 作为第三路投票
dedup=True, # 块级去重（页面内重复渲染的内容只留第一份）
)
clean 模式：已清洗的上游输入
当上游 HTML 已经清洗过（例如只有正文的 body 片段）时，用 mode="clean" 跳过所有会删正文
的内容启发式（traf/read/magic 投票、link_list 打标、dedup、clean_tree、wiki 特判、
交互/不可见/广告元素清理），只做 math/code/table 等结构化 DOM 处理和 paragraphs 转换。
标题/时间抽取照常执行；script/head/iframe 等非正文标签仍会剔除。
result = extractor.extract(cleaned_body_html, mode="clean", url="https://example.com/docs/")
V1（旧接口）
from haruka_parser.extract import extract_text
html = """<!DOCTYPE html>
<html>
<body>
<!-- Using MathML -->
<p>Using MathJax:</p>
<script type="math/tex; mode=display" id="MathJax-Element-1">{e}^{i\pi }=-1</script>
<!-- Using MathML -->
<p>Using MathML:</p>
<math xmlns="http://www.w3.org/1998/Math/MathML">
<msup>
<mi>e</mi>
<mrow>
<mi>i</mi>
<mi>π</mi>
</mrow>
</msup>
<mo>=</mo>
<mn>-1</mn>
</math>
<!-- Using AsciiMath -->
<p>Using AsciiMath:</p>
<script type="math/asciimath">
e^(i*pi) = -1
</script>
</body>
</html>"""
text, info = extract_text(html)
print(text)
print(info)
Configurations
from haruka_parser.extract import DEFAULT_CONFIG
DEFAULT_CONFIG = {
"readability": False,
"skip_large_links": False,
"extract_latex": True,
"extract_cnki_latex": False,
"escape_dollars": True,
"remove_buttons": True,
"remove_edit_buttons": True,
"remove_image_figures": True,
"markdown_code": True,
"markdown_headings": True,
"remove_chinese": False,
"boilerplate_config": {
"enable": False,
"ratio_threshold": 0.18,
"absolute_threshold": 10,
"end_threshold": 15,
},
}
Parsing speed
10k Page:
| method | haruka-parser 0.5.2 | haruku-parser 0.4.9 | html2text | inscriptis | trafilatura |
|---|---|---|---|---|---|
| Speed | 379.4s | 391.6s | 272.8s | 114.7s | 343.9s |
1 maintainer
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
haruka_parser-1.3.3.tar.gz
(34.4 MB
view details)
Built Distribution
Filter files by name, interpreter, ABI, and platform.
If you're not sure about the file name format, learn more about wheel file names.
Copy a direct link to the current filters
File details
Details for the file haruka_parser-1.3.3.tar.gz.
File metadata
- Download URL: haruka_parser-1.3.3.tar.gz
- Upload date:
- Size: 34.4 MB
- Tags: Source
- Uploaded using Trusted Publishing? No
- Uploaded via:
twine/7.0.0 CPython/3.13.14
File hashes
| Algorithm | Hash digest | |
|---|---|---|
| SHA256 |
7ef60b20b23b26302bd0341f8611b50503ae173f3ba8283b909a972407a1926a
|
|
| MD5 |
945c581265663fbd789758e31bfc553d
|
|
| BLAKE2b-256 |
5d42644aedd2d077766cbcf6073d682302660a839eb3fc401b5fcb7c0639d23d
|
File details
Details for the file haruka_parser-1.3.3-py3-none-any.whl.
File metadata
- Download URL: haruka_parser-1.3.3-py3-none-any.whl
- Upload date:
- Size: 34.7 MB
- Tags: Python 3
- Uploaded using Trusted Publishing? No
- Uploaded via:
twine/7.0.0 CPython/3.13.14
File hashes
| Algorithm | Hash digest | |
|---|---|---|
| SHA256 |
4e16079efca5120d6b4f6adf3b5b454e3c0f5ab7ff1f20b2fccf1b5f9d765f80
|
|
| MD5 |
badabe8574086a37166506e915a2f305
|
|
| BLAKE2b-256 |
3a52477f9d5d8b7fe7f3bd92a098b84c13a7fea559212e553294f25161a7ed64
|
