Convert HTML to Markdown for LLM input extraction

Navigation

Verified details

These details have been verified by PyPI
Project links
GitHub Statistics
Maintainers
Avatar for ancs21 from gravatar.com ancs21

Unverified details

These details have not been verified by PyPI
Meta
Classifiers
Report project as malware

Project description

fast-html2md PyPI version Run Tests codecov

Convert HTML to Markdown for LLM input extraction.

Installation

# use pip
pip install fast-html2md

# or use poetry
poetry add fast-html2md

# or use uv
uv add fast-html2md

Usage

from fast_html2md import HTMLToMarkdown

converter = HTMLToMarkdown()

html = """
<!DOCTYPE html>
<html>
<body>
  <h1 id="title" data-updated="20201101">Hi there</h1>
  <div class="post">
    Lorem Ipsum is simply dummy text of the printing and typesetting industry.
  </div>
  <div class="post">
    Lorem ipsum dolor sit amet, consectetur adipiscing elit.
  </div>
</body>
</html>
"""

markdown = converter.convert(html)

print(markdown)

# Count tokens
token_count = converter.count_tokens(markdown)
print(f"Token count: {token_count}")

# Compute cost
cost = converter.compute_cost(token_count)
print(f"Estimated cost: ${cost:.6f}")

Features

  • Blazing-fast HTML to Markdown conversion powered by a Rust engine (html-to-markdown) — ~60x faster than the previous pure-Python (BeautifulSoup + markdownify) pipeline
  • CommonMark-compliant output with GFM tables; links and structure are preserved
  • Optimized for LLM input processing
  • Built-in token counting using tiktoken
  • Clean and minimal output

Examples

# Prerequisites
# ----
# This script is used to scrape the GitHub Trending page and extract the data into a JSON object.
# !uv pip install fast-html2md hrequests google-genai python-dotenv
# create .env file with GOOGLE_API_KEY=your_api_key
# uv run python examples/github-trending.py
# ----

# Run the script
uv run python examples/github-trending.py

# Result (JSON / 10 Jan 2025)
{
  "repositories": [
    {
      "description": "Your AI second brain. Self-hostable. Get answers from the web or your docs. Build custom agents, schedule automations, do deep research. Turn any online or local LLM into your personal, autonomous AI (gpt, claude, gemini, llama, qwen, mistral). Get started - free.",
      "forks": 1214,
      "href": "khoj-ai /  khoj",
      "name": "khoj",
      "owner": "khoj-ai",
      "rank": 1,
      "repository_name": "khoj-ai /  khoj",
      "stars": 23173,
     "built_by": [],
      "language": "Python",
      "todays_stars": 1508
    },
    {
      "description": "Resume Matcher is an open source, free tool to improve your resume. It works by using AI, Reader LLMs, to compare and rank resumes with job descriptions.",
      "forks": 2528,
      "href": "srbhr /  Resume-Matcher",
      "name": "Resume-Matcher",
      "owner": "srbhr",
      "rank": 2,
      "repository_name": "srbhr /  Resume-Matcher",
      "stars": 6814,
    "built_by": [],
      "language": "Python",
      "todays_stars": 436
    },
    {
       "description": "How to run an Ink Node",
      "forks": 232,
      "href": "inkonchain /  node",
      "name": "node",
      "owner": "inkonchain",
      "rank": 3,
      "repository_name": "inkonchain /  node",
      "stars": 17265,
     "built_by": [],
     "language": "Shell",
      "todays_stars": 3303
    },
    {
     "description": "VILA is a family of state-of-the-art vision language models (VLMs) for diverse multimodal AI tasks across the edge, data center, and cloud.",
      "forks": 198,
      "href": "NVlabs /  VILA",
       "name": "VILA",
      "owner": "NVlabs",
      "rank": 4,
      "repository_name": "NVlabs /  VILA",
      "stars": 2498,
    "built_by": [],
      "language": "Python",
      "todays_stars": 60
    },
    {
      "description": "🚀🤖 Crawl4AI: Open-source LLM Friendly Web Crawler & Scraper",
      "forks": 1691,
      "href": "unclecode /  crawl4ai",
      "name": "crawl4ai",
      "owner": "unclecode",
      "rank": 5,
      "repository_name": "unclecode /  crawl4ai",
      "stars": 23533,
    "built_by": [],
      "language": "Python",
      "todays_stars": 1106
    },
    {
      "description": "Ink Documentation",
      "forks": 172,
      "href": "inkonchain /  docs",
       "name": "docs",
      "owner": "inkonchain",
      "rank": 6,
      "repository_name": "inkonchain /  docs",
      "stars": 17200,
     "built_by": [],
     "language": "MDX",
      "todays_stars": 3301
    },
    {
     "description": "GoogleTest - Google Testing and Mocking Framework",
      "forks": 10223,
      "href": "google /  googletest",
       "name": "googletest",
      "owner": "google",
      "rank": 7,
      "repository_name": "google /  googletest",
      "stars": 35180,
    "built_by": [],
      "language": "C++",
      "todays_stars": 10
    },
        {
      "description": "Build your own AI friend",
       "forks": 338,
      "href": "78 /  xiaozhi-esp32",
      "name": "xiaozhi-esp32",
      "owner": "78",
      "rank": 8,
      "repository_name": "78 /  xiaozhi-esp32",
      "stars": 1973,
     "built_by": [],
     "language": "C",
      "todays_stars": 437
    },
    {
     "description": "Firebase SDK for Apple App Development",
      "forks": 1504,
      "href": "firebase /  firebase-ios-sdk",
      "name": "firebase-ios-sdk",
      "owner": "firebase",
      "rank": 9,
      "repository_name": "firebase /  firebase-ios-sdk",
      "stars": 5820,
    "built_by": [],
      "language": "C++",
      "todays_stars": 38
    },
    {
     "description": "PyTorch native post-training library",
      "forks": 476,
      "href": "pytorch /  torchtune",
      "name": "torchtune",
       "owner": "pytorch",
       "rank": 10,
      "repository_name": "pytorch /  torchtune",
      "stars": 4601,
     "built_by": [],
      "language": "Python",
      "todays_stars": 8
    },
    {
       "description": "Fast C++ logging library.",
      "forks": 4636,
      "href": "gabime /  spdlog",
      "name": "spdlog",
      "owner": "gabime",
      "rank": 11,
      "repository_name": "gabime /  spdlog",
      "stars": 24964,
   "built_by": [],
      "language": "C++",
      "todays_stars": 15
    },
    {
      "description": "Apache Thrift",
      "forks": 4037,
      "href": "apache /  thrift",
       "name": "thrift",
      "owner": "apache",
      "rank": 12,
      "repository_name": "apache /  thrift",
      "stars": 10508,
    "built_by": [],
      "language": "C++",
      "todays_stars": 12
    },
    {
      "description": "Autonomous coding agent right in your IDE, capable of creating/editing files, executing commands, using the browser, and more with your permission every step of the way.",
      "forks": 1762,
      "href": "cline /  cline",
      "name": "cline",
       "owner": "cline",
      "rank": 13,
      "repository_name": "cline /  cline",
      "stars": 20784,
    "built_by": [],
      "language": "TypeScript",
      "todays_stars": 345
    },
    {
      "description": "A BNB Smart Chain client based on the go-ethereum fork",
      "forks": 1587,
      "href": "bnb-chain /  bsc",
      "name": "bsc",
       "owner": "bnb-chain",
      "rank": 14,
      "repository_name": "bnb-chain /  bsc",
      "stars": 2833,
   "built_by": [],
      "language": "Go",
      "todays_stars": 13
    }
  ]
}

License

This project is licensed under the MIT License - see the LICENSE file for details.

Contributing

Contributions are welcome! Please feel free to submit a Pull Request.

Project details

Verified details

These details have been verified by PyPI
Project links
GitHub Statistics
Maintainers
Avatar for ancs21 from gravatar.com ancs21

Unverified details

These details have not been verified by PyPI
Meta
Classifiers

Release history Release notifications | RSS feed

This version

0.2.0

0.1.3

0.1.2

0.1.1

0.1.0

Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

fast_html2md-0.2.0.tar.gz (39.0 kB view details)

Uploaded Source

Built Distribution

If you're not sure about the file name format, learn more about wheel file names.

fast_html2md-0.2.0-py3-none-any.whl (8.5 kB view details)

Uploaded Python 3

File details

Details for the file fast_html2md-0.2.0.tar.gz.

File metadata

  • Download URL: fast_html2md-0.2.0.tar.gz
  • Upload date:
  • Size: 39.0 kB
  • Tags: Source
  • Uploaded using Trusted Publishing? Yes
  • Uploaded via: twine/6.1.0 CPython/3.13.12

File hashes

Hashes for fast_html2md-0.2.0.tar.gz
Algorithm Hash digest
SHA256 6fa5a3e470ccd4c0bae27b0466daaf2f554b27f89c1bb18438eb5a21a3c297ab
MD5 08f6ffef17a298f5248957fa01e83203
BLAKE2b-256 f8e2d7b456fa47a04f5c8b72cdce013bfe99e82ffae5300437a690d16b67627e

See more details on using hashes here.

Provenance

The following attestation bundles were made for fast_html2md-0.2.0.tar.gz:

Publisher: publish.yml on ancs21/fast-html2md

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

File details

Details for the file fast_html2md-0.2.0-py3-none-any.whl.

File metadata

  • Download URL: fast_html2md-0.2.0-py3-none-any.whl
  • Upload date:
  • Size: 8.5 kB
  • Tags: Python 3
  • Uploaded using Trusted Publishing? Yes
  • Uploaded via: twine/6.1.0 CPython/3.13.12

File hashes

Hashes for fast_html2md-0.2.0-py3-none-any.whl
Algorithm Hash digest
SHA256 77ee31ac4cb6ee4fb445973e4e0b18fbf77b06c5fb224c62cc00054fbcaaff40
MD5 d3d3da8aecaa5875616f7abf0771cc08
BLAKE2b-256 fb68a7f1a278fc912de0b0f7a3048c96025d746d6a141bea50ce212e55a659f9

See more details on using hashes here.

Provenance

The following attestation bundles were made for fast_html2md-0.2.0-py3-none-any.whl:

Publisher: publish.yml on ancs21/fast-html2md

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

Supported by

AWS Cloud computing and Security Sponsor Datadog Monitoring Depot Continuous Integration Fastly CDN Google Download Analytics Pingdom Monitoring Sentry Error logging StatusPage Status page