Haruka Parser

A simple HTML Parser

Install

pip install haruka-parser

Usage

V2（推荐）

V2 是多引擎投票的抽取管线：trafilatura 与 readability 两路主内容启发式对段落投票， 只保留双引擎都命中的正文块；在此基础上叠加 math/code/table 的结构化转换与块级去重。

from haruka_parser.v2.extractors.auto_extractor import AutoExtractor

extractor = AutoExtractor()
result = extractor.extract(html, url="https://example.com/page.html")

print(result["content"])     # Markdown 正文
print(result["title"])       # 标题
print(result["time"])        # 发布时间 "2024-05-11 08:30:00"（无则 ""）
print(result["time_stamp"])  # 对应 unix timestamp（字符串，无则 ""）

返回字段：

字段 说明
content Markdown 格式正文
paragraphs 段落结构（含 unique_id、tag、dom meta），可回放
dom_attrs / links DOM 属性与页面链接（链接按 base_url 补全）
title / time / time_stamp 标题、发布时间及其 timestamp
base_url / encoding 最终基准 URL 与检测到的编码

常用参数：

extractor.extract(
    html,
    url="https://example.com/page.html",  # 用于链接补全、域名特判、标题回退
    separator="\n\n",                     # 段落分隔符
    table_format="github",                # github / grid / simple / html / html_raw / html_auto
    escape_dollars=False,                 # 非 LaTeX 上下文的 $ 转义
    use_ftfy=False,                       # ftfy 文本修复
    magic_html_heuristic=False,           # 引入 magic-html 作为第三路投票
    dedup=True,                           # 块级去重（页面内重复渲染的内容只留第一份）
)

clean 模式：已清洗的上游输入

当上游 HTML 已经清洗过（例如只有正文的 body 片段）时，用 mode="clean" 跳过所有会删正文 的内容启发式（traf/read/magic 投票、link_list 打标、dedup、clean_tree、wiki 特判、 交互/不可见/广告元素清理），只做 math/code/table 等结构化 DOM 处理和 paragraphs 转换。 标题/时间抽取照常执行；script/head/iframe 等非正文标签仍会剔除。

result = extractor.extract(cleaned_body_html, mode="clean", url="https://example.com/docs/")

V1（旧接口）

from haruka_parser.extract import extract_text

html = """<!DOCTYPE html>
<html>
<body>
<!-- Using MathML -->
<p>Using MathJax:</p>
<script type="math/tex; mode=display" id="MathJax-Element-1">{e}^{i\pi }=-1</script>
<!-- Using MathML -->
<p>Using MathML:</p>
<math xmlns="http://www.w3.org/1998/Math/MathML">
  <msup>
    <mi>e</mi>
    <mrow>
      <mi>i</mi>
      <mi>&#x03C0;</mi>
    </mrow>
  </msup>
  <mo>=</mo>
  <mn>-1</mn>
</math>

<!-- Using AsciiMath -->
<p>Using AsciiMath:</p>
<script type="math/asciimath">
e^(i*pi) = -1
</script>

</body>
</html>"""

text, info = extract_text(html)
print(text)
print(info)

Configurations

from haruka_parser.extract import DEFAULT_CONFIG
DEFAULT_CONFIG = {
    "readability": False,
    "skip_large_links": False,
    "extract_latex": True,
    "extract_cnki_latex": False,
    "escape_dollars": True,
    "remove_buttons": True,
    "remove_edit_buttons": True,
    "remove_image_figures": True,
    "markdown_code": True,
    "markdown_headings": True,
    "remove_chinese": False,
    "boilerplate_config": {
        "enable": False,
        "ratio_threshold": 0.18,
        "absolute_threshold": 10,
        "end_threshold": 15,
    },
}

Parsing speed

10k Page:

method haruka-parser 0.5.2 haruku-parser 0.4.9 html2text inscriptis trafilatura
Speed 379.4s 391.6s 272.8s 114.7s 343.9s

Project links

Key dates

PyPI data
Data sourced directly from PyPI's database.
  • Released:
Newer release available (1.3.4)

1 maintainer

PyPI data
Data sourced directly from PyPI's database.
Avatar for papersnake from gravatar.com papersnake

Credits

Author: papersnake

License

MIT License

Provides Extra

dev justext

Classifiers

Report project as malware

Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

haruka_parser-1.2.1.tar.gz (34.4 MB view details)

Uploaded Source

Built Distribution

If you're not sure about the file name format, learn more about wheel file names.

haruka_parser-1.2.1-py3-none-any.whl (34.7 MB view details)

Uploaded Python 3

File details

Details for the file haruka_parser-1.2.1.tar.gz.

File metadata

  • Download URL: haruka_parser-1.2.1.tar.gz
  • Upload date:
  • Size: 34.4 MB
  • Tags: Source
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/7.0.0 CPython/3.13.14

File hashes

Hashes for haruka_parser-1.2.1.tar.gz
Algorithm Hash digest
SHA256 74d1c1c4e46e809d55097d9a82ed174ee380e10eeab53398782df3f6542b09f8
MD5 633de1dabbf4ba6d399b1083c40ad29a
BLAKE2b-256 5a6fd451706a2cb96dddc8420b77f035ff67008fed8ccccd95ef2525dde22bc8

See more details on using hashes here.

File details

Details for the file haruka_parser-1.2.1-py3-none-any.whl.

File metadata

  • Download URL: haruka_parser-1.2.1-py3-none-any.whl
  • Upload date:
  • Size: 34.7 MB
  • Tags: Python 3
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/7.0.0 CPython/3.13.14

File hashes

Hashes for haruka_parser-1.2.1-py3-none-any.whl
Algorithm Hash digest
SHA256 cf3a663f5e0189e42bd677dba56a6257c451a88eee799bb9434077bf3ba883c5
MD5 35b9c08b21a57a53cf0d1393bfdaa7d1
BLAKE2b-256 12bafd6a9a39664c1ac1dc77029a632860473f920e018e975d4582669c74a82e

See more details on using hashes here.

Release history Release notifications | RSS feed

1.3.4

2 files

1.3.3

2 files

1.3.1

2 files

1.3.0

2 files

1.2.3

2 files

1.2.2

2 files

This release

1.2.1 This release

2 files

1.2.0

2 files

1.1.3

2 files

1.1.2

2 files

1.1.1

2 files

1.1.0

2 files

1.0.8

2 files

1.0.6

2 files

1.0.5

2 files

1.0.4

2 files

1.0.3

2 files

1.0.2

2 files

1.0.1

2 files

1.0.0

2 files

0.10.3

2 files

0.10.2

2 files

0.10.1

2 files

0.10.0

2 files

0.9.6

2 files

0.9.5

2 files

0.9.3

2 files

0.9.2

2 files

0.9.1

2 files

0.9.0

2 files

0.8.3

2 files

0.8.1

2 files

0.8.0

2 files

0.7.3

2 files

0.7.2

2 files

0.7.1

2 files

0.7.0

2 files

0.6.5

2 files

0.6.4

2 files

0.6.3

2 files

0.6.2

2 files

0.6.1

2 files

0.6.0

2 files

0.5.2

2 files

0.5.1

2 files

0.5.0

2 files

0.4.9

2 files

0.4.8

2 files

0.4.7

2 files

0.4.6

2 files

0.4.5

2 files

0.4.3

2 files

0.4.2

2 files

0.4.1

2 files

0.4.0

2 files

0.3.6

2 files

0.3.5

2 files

0.3.4

2 files

0.3.2

2 files

0.3.1

2 files

0.3.0

2 files

0.2.9

2 files

0.2.8

2 files

0.2.7

2 files

0.2.6

2 files

0.2.5

2 files

0.2.4

2 files

0.2.0

2 files

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page