Haruka Parser

A simple HTML Parser

Install

pip install haruka-parser

Usage

V2（推荐）

V2 是多引擎投票的抽取管线：trafilatura 与 readability 两路主内容启发式对段落投票， 只保留双引擎都命中的正文块；在此基础上叠加 math/code/table 的结构化转换与块级去重。

from haruka_parser.v2.extractors.auto_extractor import AutoExtractor

extractor = AutoExtractor()
result = extractor.extract(html, url="https://example.com/page.html")

print(result["content"])     # Markdown 正文
print(result["title"])       # 标题
print(result["time"])        # 发布时间 "2024-05-11 08:30:00"（无则 ""）
print(result["time_stamp"])  # 对应 unix timestamp（字符串，无则 ""）

返回字段：

字段 说明
content Markdown 格式正文
paragraphs 段落结构（含 unique_id、tag、dom meta），可回放
dom_attrs / links DOM 属性与页面链接（链接按 base_url 补全）
title / time / time_stamp 标题、发布时间及其 timestamp
base_url / encoding 最终基准 URL 与检测到的编码

常用参数：

extractor.extract(
    html,
    url="https://example.com/page.html",  # 用于链接补全、域名特判、标题回退
    separator="\n\n",                     # 段落分隔符
    table_format="github",                # github / grid / simple / html / html_raw / html_auto
    escape_dollars=False,                 # 非 LaTeX 上下文的 $ 转义
    use_ftfy=False,                       # ftfy 文本修复
    magic_html_heuristic=False,           # 引入 magic-html 作为第三路投票
    dedup=True,                           # 块级去重（页面内重复渲染的内容只留第一份）
)

clean 模式：已清洗的上游输入

当上游 HTML 已经清洗过（例如只有正文的 body 片段）时，用 mode="clean" 跳过所有会删正文 的内容启发式（traf/read/magic 投票、link_list 打标、dedup、clean_tree、wiki 特判、 交互/不可见/广告元素清理），只做 math/code/table 等结构化 DOM 处理和 paragraphs 转换。 标题/时间抽取照常执行；script/head/iframe 等非正文标签仍会剔除。

result = extractor.extract(cleaned_body_html, mode="clean", url="https://example.com/docs/")

V1（旧接口）

from haruka_parser.extract import extract_text

html = """<!DOCTYPE html>
<html>
<body>
<!-- Using MathML -->
<p>Using MathJax:</p>
<script type="math/tex; mode=display" id="MathJax-Element-1">{e}^{i\pi }=-1</script>
<!-- Using MathML -->
<p>Using MathML:</p>
<math xmlns="http://www.w3.org/1998/Math/MathML">
  <msup>
    <mi>e</mi>
    <mrow>
      <mi>i</mi>
      <mi>&#x03C0;</mi>
    </mrow>
  </msup>
  <mo>=</mo>
  <mn>-1</mn>
</math>

<!-- Using AsciiMath -->
<p>Using AsciiMath:</p>
<script type="math/asciimath">
e^(i*pi) = -1
</script>

</body>
</html>"""

text, info = extract_text(html)
print(text)
print(info)

Configurations

from haruka_parser.extract import DEFAULT_CONFIG
DEFAULT_CONFIG = {
    "readability": False,
    "skip_large_links": False,
    "extract_latex": True,
    "extract_cnki_latex": False,
    "escape_dollars": True,
    "remove_buttons": True,
    "remove_edit_buttons": True,
    "remove_image_figures": True,
    "markdown_code": True,
    "markdown_headings": True,
    "remove_chinese": False,
    "boilerplate_config": {
        "enable": False,
        "ratio_threshold": 0.18,
        "absolute_threshold": 10,
        "end_threshold": 15,
    },
}

Parsing speed

10k Page:

method haruka-parser 0.5.2 haruku-parser 0.4.9 html2text inscriptis trafilatura
Speed 379.4s 391.6s 272.8s 114.7s 343.9s

Project links

Key dates

PyPI data
Data sourced directly from PyPI's database.
  • Released:
Newer release available (1.3.4)

1 maintainer

PyPI data
Data sourced directly from PyPI's database.
Avatar for papersnake from gravatar.com papersnake

Credits

Author: papersnake

License

MIT License

Provides Extra

dev justext

Classifiers

Report project as malware

Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

haruka_parser-1.2.0.tar.gz (34.4 MB view details)

Uploaded Source

Built Distribution

If you're not sure about the file name format, learn more about wheel file names.

haruka_parser-1.2.0-py3-none-any.whl (34.7 MB view details)

Uploaded Python 3

File details

Details for the file haruka_parser-1.2.0.tar.gz.

File metadata

  • Download URL: haruka_parser-1.2.0.tar.gz
  • Upload date:
  • Size: 34.4 MB
  • Tags: Source
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/7.0.0 CPython/3.13.14

File hashes

Hashes for haruka_parser-1.2.0.tar.gz
Algorithm Hash digest
SHA256 61aa635e258cb58fa9f48d3bbaa95890d6bc31b1d3785b23210c54a1c5ae5d85
MD5 a31b0faa6337be66611501fdc928eb7c
BLAKE2b-256 144dbf0288aeeec452a348400dde43ebe3d24c795f59a9d87b8da9c0093db8ef

See more details on using hashes here.

File details

Details for the file haruka_parser-1.2.0-py3-none-any.whl.

File metadata

  • Download URL: haruka_parser-1.2.0-py3-none-any.whl
  • Upload date:
  • Size: 34.7 MB
  • Tags: Python 3
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/7.0.0 CPython/3.13.14

File hashes

Hashes for haruka_parser-1.2.0-py3-none-any.whl
Algorithm Hash digest
SHA256 a85c8610b17397d25faf609806c486361f949aa2b10ff28bc1de90c8f25c129c
MD5 ac74f06ad3d8004aa9712bf0e7c9ea44
BLAKE2b-256 b3935fd0a46fb8c5fb481a2fc7e7e218f0a8b0485ad3d398693d4fd0d3459f44

See more details on using hashes here.

Release history Release notifications | RSS feed

1.3.4

2 files

1.3.3

2 files

1.3.1

2 files

1.3.0

2 files

1.2.3

2 files

1.2.2

2 files

1.2.1

2 files

This release

1.2.0 This release

2 files

1.1.3

2 files

1.1.2

2 files

1.1.1

2 files

1.1.0

2 files

1.0.8

2 files

1.0.6

2 files

1.0.5

2 files

1.0.4

2 files

1.0.3

2 files

1.0.2

2 files

1.0.1

2 files

1.0.0

2 files

0.10.3

2 files

0.10.2

2 files

0.10.1

2 files

0.10.0

2 files

0.9.6

2 files

0.9.5

2 files

0.9.3

2 files

0.9.2

2 files

0.9.1

2 files

0.9.0

2 files

0.8.3

2 files

0.8.1

2 files

0.8.0

2 files

0.7.3

2 files

0.7.2

2 files

0.7.1

2 files

0.7.0

2 files

0.6.5

2 files

0.6.4

2 files

0.6.3

2 files

0.6.2

2 files

0.6.1

2 files

0.6.0

2 files

0.5.2

2 files

0.5.1

2 files

0.5.0

2 files

0.4.9

2 files

0.4.8

2 files

0.4.7

2 files

0.4.6

2 files

0.4.5

2 files

0.4.3

2 files

0.4.2

2 files

0.4.1

2 files

0.4.0

2 files

0.3.6

2 files

0.3.5

2 files

0.3.4

2 files

0.3.2

2 files

0.3.1

2 files

0.3.0

2 files

0.2.9

2 files

0.2.8

2 files

0.2.7

2 files

0.2.6

2 files

0.2.5

2 files

0.2.4

2 files

0.2.0

2 files

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page