Haruka Parser

A simple HTML Parser

Install

pip install haruka-parser

Usage

V2（推荐）

V2 是多引擎投票的抽取管线：trafilatura 与 readability 两路主内容启发式对段落投票， 只保留双引擎都命中的正文块；在此基础上叠加 math/code/table 的结构化转换与块级去重。

from haruka_parser.v2.extractors.auto_extractor import AutoExtractor

extractor = AutoExtractor()
result = extractor.extract(html, url="https://example.com/page.html")

print(result["content"])     # Markdown 正文
print(result["title"])       # 标题
print(result["time"])        # 发布时间 "2024-05-11 08:30:00"（无则 ""）
print(result["time_stamp"])  # 对应 unix timestamp（字符串，无则 ""）

返回字段：

字段 说明
content Markdown 格式正文
paragraphs 段落结构（含 unique_id、tag、dom meta），可回放
dom_attrs / links DOM 属性与页面链接（链接按 base_url 补全）
title / time / time_stamp 标题、发布时间及其 timestamp
base_url / encoding 最终基准 URL 与检测到的编码

常用参数：

extractor.extract(
    html,
    url="https://example.com/page.html",  # 用于链接补全、域名特判、标题回退
    separator="\n\n",                     # 段落分隔符
    table_format="github",                # github / grid / simple / html / html_raw / html_auto
    escape_dollars=False,                 # 非 LaTeX 上下文的 $ 转义
    use_ftfy=False,                       # ftfy 文本修复
    magic_html_heuristic=False,           # 引入 magic-html 作为第三路投票
    dedup=True,                           # 块级去重（页面内重复渲染的内容只留第一份）
)

clean 模式：已清洗的上游输入

当上游 HTML 已经清洗过（例如只有正文的 body 片段）时，用 mode="clean" 跳过所有会删正文 的内容启发式（traf/read/magic 投票、link_list 打标、dedup、clean_tree、wiki 特判、 交互/不可见/广告元素清理），只做 math/code/table 等结构化 DOM 处理和 paragraphs 转换。 标题/时间抽取照常执行；script/head/iframe 等非正文标签仍会剔除。

result = extractor.extract(cleaned_body_html, mode="clean", url="https://example.com/docs/")

V1（旧接口）

from haruka_parser.extract import extract_text

html = """<!DOCTYPE html>
<html>
<body>
<!-- Using MathML -->
<p>Using MathJax:</p>
<script type="math/tex; mode=display" id="MathJax-Element-1">{e}^{i\pi }=-1</script>
<!-- Using MathML -->
<p>Using MathML:</p>
<math xmlns="http://www.w3.org/1998/Math/MathML">
  <msup>
    <mi>e</mi>
    <mrow>
      <mi>i</mi>
      <mi>&#x03C0;</mi>
    </mrow>
  </msup>
  <mo>=</mo>
  <mn>-1</mn>
</math>

<!-- Using AsciiMath -->
<p>Using AsciiMath:</p>
<script type="math/asciimath">
e^(i*pi) = -1
</script>

</body>
</html>"""

text, info = extract_text(html)
print(text)
print(info)

Configurations

from haruka_parser.extract import DEFAULT_CONFIG
DEFAULT_CONFIG = {
    "readability": False,
    "skip_large_links": False,
    "extract_latex": True,
    "extract_cnki_latex": False,
    "escape_dollars": True,
    "remove_buttons": True,
    "remove_edit_buttons": True,
    "remove_image_figures": True,
    "markdown_code": True,
    "markdown_headings": True,
    "remove_chinese": False,
    "boilerplate_config": {
        "enable": False,
        "ratio_threshold": 0.18,
        "absolute_threshold": 10,
        "end_threshold": 15,
    },
}

Parsing speed

10k Page:

method haruka-parser 0.5.2 haruku-parser 0.4.9 html2text inscriptis trafilatura
Speed 379.4s 391.6s 272.8s 114.7s 343.9s

Project links

Key dates

PyPI data
Data sourced directly from PyPI's database.
  • Released:
Newer release available (1.3.4)

1 maintainer

PyPI data
Data sourced directly from PyPI's database.
Avatar for papersnake from gravatar.com papersnake

Credits

Author: papersnake

License

MIT License

Provides Extra

dev justext

Classifiers

Report project as malware

Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

haruka_parser-1.2.3.tar.gz (34.4 MB view details)

Uploaded Source

Built Distribution

If you're not sure about the file name format, learn more about wheel file names.

haruka_parser-1.2.3-py3-none-any.whl (34.7 MB view details)

Uploaded Python 3

File details

Details for the file haruka_parser-1.2.3.tar.gz.

File metadata

  • Download URL: haruka_parser-1.2.3.tar.gz
  • Upload date:
  • Size: 34.4 MB
  • Tags: Source
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/7.0.0 CPython/3.13.14

File hashes

Hashes for haruka_parser-1.2.3.tar.gz
Algorithm Hash digest
SHA256 b8eec67451e8c3c2ac8276d418dfdb8561cae4d0f7e0929da8ecc655b39b0fb4
MD5 32ce8be2ea0bb3fc08d49a42447889cd
BLAKE2b-256 f19a33aaaa9c6ef6d9756a34942ed7cf6fb6158e31ea9f9ed6bef7bf637c9f64

See more details on using hashes here.

File details

Details for the file haruka_parser-1.2.3-py3-none-any.whl.

File metadata

  • Download URL: haruka_parser-1.2.3-py3-none-any.whl
  • Upload date:
  • Size: 34.7 MB
  • Tags: Python 3
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/7.0.0 CPython/3.13.14

File hashes

Hashes for haruka_parser-1.2.3-py3-none-any.whl
Algorithm Hash digest
SHA256 62f8155dbf7ea8b1b610d2aadb13828a6a5fbc479e5be82e76586c5df8722006
MD5 e69738321597ade83947a934f8ea61cd
BLAKE2b-256 31883ad7a9f31a04ea0e31ba94308a1efcc61d46cb9ecb99f27c1e83b9950a14

See more details on using hashes here.

Release history Release notifications | RSS feed

1.3.4

2 files

1.3.3

2 files

1.3.1

2 files

1.3.0

2 files

This release

1.2.3 This release

2 files

1.2.2

2 files

1.2.1

2 files

1.2.0

2 files

1.1.3

2 files

1.1.2

2 files

1.1.1

2 files

1.1.0

2 files

1.0.8

2 files

1.0.6

2 files

1.0.5

2 files

1.0.4

2 files

1.0.3

2 files

1.0.2

2 files

1.0.1

2 files

1.0.0

2 files

0.10.3

2 files

0.10.2

2 files

0.10.1

2 files

0.10.0

2 files

0.9.6

2 files

0.9.5

2 files

0.9.3

2 files

0.9.2

2 files

0.9.1

2 files

0.9.0

2 files

0.8.3

2 files

0.8.1

2 files

0.8.0

2 files

0.7.3

2 files

0.7.2

2 files

0.7.1

2 files

0.7.0

2 files

0.6.5

2 files

0.6.4

2 files

0.6.3

2 files

0.6.2

2 files

0.6.1

2 files

0.6.0

2 files

0.5.2

2 files

0.5.1

2 files

0.5.0

2 files

0.4.9

2 files

0.4.8

2 files

0.4.7

2 files

0.4.6

2 files

0.4.5

2 files

0.4.3

2 files

0.4.2

2 files

0.4.1

2 files

0.4.0

2 files

0.3.6

2 files

0.3.5

2 files

0.3.4

2 files

0.3.2

2 files

0.3.1

2 files

0.3.0

2 files

0.2.9

2 files

0.2.8

2 files

0.2.7

2 files

0.2.6

2 files

0.2.5

2 files

0.2.4

2 files

0.2.0

2 files

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page