Haruka Parser

A simple HTML Parser

Install

pip install haruka-parser

Usage

V2（推荐）

V2 是多引擎投票的抽取管线：trafilatura 与 readability 两路主内容启发式对段落投票， 只保留双引擎都命中的正文块；在此基础上叠加 math/code/table 的结构化转换与块级去重。

from haruka_parser.v2.extractors.auto_extractor import AutoExtractor

extractor = AutoExtractor()
result = extractor.extract(html, url="https://example.com/page.html")

print(result["content"])     # Markdown 正文
print(result["title"])       # 标题
print(result["time"])        # 发布时间 "2024-05-11 08:30:00"（无则 ""）
print(result["time_stamp"])  # 对应 unix timestamp（字符串，无则 ""）

返回字段：

字段 说明
content Markdown 格式正文
paragraphs 段落结构（含 unique_id、tag、dom meta），可回放
dom_attrs / links DOM 属性与页面链接（链接按 base_url 补全）
title / time / time_stamp 标题、发布时间及其 timestamp
base_url / encoding 最终基准 URL 与检测到的编码

常用参数：

extractor.extract(
    html,
    url="https://example.com/page.html",  # 用于链接补全、域名特判、标题回退
    separator="\n\n",                     # 段落分隔符
    table_format="github",                # github / grid / simple / html / html_raw / html_auto
    escape_dollars=False,                 # 非 LaTeX 上下文的 $ 转义
    use_ftfy=False,                       # ftfy 文本修复
    magic_html_heuristic=False,           # 引入 magic-html 作为第三路投票
    dedup=True,                           # 块级去重（页面内重复渲染的内容只留第一份）
)

clean 模式：已清洗的上游输入

当上游 HTML 已经清洗过（例如只有正文的 body 片段）时，用 mode="clean" 跳过所有会删正文 的内容启发式（traf/read/magic 投票、link_list 打标、dedup、clean_tree、wiki 特判、 交互/不可见/广告元素清理），只做 math/code/table 等结构化 DOM 处理和 paragraphs 转换。 标题/时间抽取照常执行；script/head/iframe 等非正文标签仍会剔除。

result = extractor.extract(cleaned_body_html, mode="clean", url="https://example.com/docs/")

V1（旧接口）

from haruka_parser.extract import extract_text

html = """<!DOCTYPE html>
<html>
<body>
<!-- Using MathML -->
<p>Using MathJax:</p>
<script type="math/tex; mode=display" id="MathJax-Element-1">{e}^{i\pi }=-1</script>
<!-- Using MathML -->
<p>Using MathML:</p>
<math xmlns="http://www.w3.org/1998/Math/MathML">
  <msup>
    <mi>e</mi>
    <mrow>
      <mi>i</mi>
      <mi>&#x03C0;</mi>
    </mrow>
  </msup>
  <mo>=</mo>
  <mn>-1</mn>
</math>

<!-- Using AsciiMath -->
<p>Using AsciiMath:</p>
<script type="math/asciimath">
e^(i*pi) = -1
</script>

</body>
</html>"""

text, info = extract_text(html)
print(text)
print(info)

Configurations

from haruka_parser.extract import DEFAULT_CONFIG
DEFAULT_CONFIG = {
    "readability": False,
    "skip_large_links": False,
    "extract_latex": True,
    "extract_cnki_latex": False,
    "escape_dollars": True,
    "remove_buttons": True,
    "remove_edit_buttons": True,
    "remove_image_figures": True,
    "markdown_code": True,
    "markdown_headings": True,
    "remove_chinese": False,
    "boilerplate_config": {
        "enable": False,
        "ratio_threshold": 0.18,
        "absolute_threshold": 10,
        "end_threshold": 15,
    },
}

Parsing speed

10k Page:

method haruka-parser 0.5.2 haruku-parser 0.4.9 html2text inscriptis trafilatura
Speed 379.4s 391.6s 272.8s 114.7s 343.9s

Project links

Key dates

PyPI data
Data sourced directly from PyPI's database.
  • Released:
Newer release available (1.3.4)

1 maintainer

PyPI data
Data sourced directly from PyPI's database.
Avatar for papersnake from gravatar.com papersnake

Credits

Author: papersnake

License

MIT License

Provides Extra

dev justext

Classifiers

Report project as malware

Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

haruka_parser-1.3.1.tar.gz (34.4 MB view details)

Uploaded Source

Built Distribution

If you're not sure about the file name format, learn more about wheel file names.

haruka_parser-1.3.1-py3-none-any.whl (34.7 MB view details)

Uploaded Python 3

File details

Details for the file haruka_parser-1.3.1.tar.gz.

File metadata

  • Download URL: haruka_parser-1.3.1.tar.gz
  • Upload date:
  • Size: 34.4 MB
  • Tags: Source
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/7.0.0 CPython/3.13.14

File hashes

Hashes for haruka_parser-1.3.1.tar.gz
Algorithm Hash digest
SHA256 a2f765da960cfe094ce86cba07bc5401b4ab0de6fe26e8ce719d57666eabc32f
MD5 5ae5ae37da7409369ec779a898c81d75
BLAKE2b-256 1517e9578b1d46375d6697da2af00793cf34dde8e3c64db0bc54a362106a999c

See more details on using hashes here.

File details

Details for the file haruka_parser-1.3.1-py3-none-any.whl.

File metadata

  • Download URL: haruka_parser-1.3.1-py3-none-any.whl
  • Upload date:
  • Size: 34.7 MB
  • Tags: Python 3
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/7.0.0 CPython/3.13.14

File hashes

Hashes for haruka_parser-1.3.1-py3-none-any.whl
Algorithm Hash digest
SHA256 f6b602d79c90ee15d7371e7985f74e2b83c172b94246c822a02f82fe7a80b0e8
MD5 35ce74c25eef78557d963f6b2f71980d
BLAKE2b-256 2a58838f2aa9588271cb4b38b7b6b9a069da1830008bfd3a338894f339e1d6c7

See more details on using hashes here.

Release history Release notifications | RSS feed

1.3.4

2 files

1.3.3

2 files

This release

1.3.1 This release

2 files

1.3.0

2 files

1.2.3

2 files

1.2.2

2 files

1.2.1

2 files

1.2.0

2 files

1.1.3

2 files

1.1.2

2 files

1.1.1

2 files

1.1.0

2 files

1.0.8

2 files

1.0.6

2 files

1.0.5

2 files

1.0.4

2 files

1.0.3

2 files

1.0.2

2 files

1.0.1

2 files

1.0.0

2 files

0.10.3

2 files

0.10.2

2 files

0.10.1

2 files

0.10.0

2 files

0.9.6

2 files

0.9.5

2 files

0.9.3

2 files

0.9.2

2 files

0.9.1

2 files

0.9.0

2 files

0.8.3

2 files

0.8.1

2 files

0.8.0

2 files

0.7.3

2 files

0.7.2

2 files

0.7.1

2 files

0.7.0

2 files

0.6.5

2 files

0.6.4

2 files

0.6.3

2 files

0.6.2

2 files

0.6.1

2 files

0.6.0

2 files

0.5.2

2 files

0.5.1

2 files

0.5.0

2 files

0.4.9

2 files

0.4.8

2 files

0.4.7

2 files

0.4.6

2 files

0.4.5

2 files

0.4.3

2 files

0.4.2

2 files

0.4.1

2 files

0.4.0

2 files

0.3.6

2 files

0.3.5

2 files

0.3.4

2 files

0.3.2

2 files

0.3.1

2 files

0.3.0

2 files

0.2.9

2 files

0.2.8

2 files

0.2.7

2 files

0.2.6

2 files

0.2.5

2 files

0.2.4

2 files

0.2.0

2 files

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page