html2md-clean

A clean, controllable HTML -> Markdown converter with noise removal (ads, popups, navigation), built for web scraping and preparing text for LLMs.

---

Table of contents

• Overview
• Features
• Installation
• Quick start
• CLI
• Usage examples
• Public API
• Conversion options
• Optional dependencies
• Performance
• Development
• Documentation
• License

---

Overview

html2md-clean turns the "messy" HTML of real-world web pages into clean Markdown. Unlike naive converters, the library first strips boilerplate — navigation, footers, banners, cookie popups, ad blocks — and only then renders the main content. That makes it a good fit for:

• web scraping and dataset building;
• preparing text for RAG / LLM pipelines (fewer noise tokens);
• archiving articles as readable Markdown.

The library runs on two parsers: Selectolax (fast, the default when installed) and BeautifulSoup (fallback), hidden behind a single Adapter interface.

Features

• Noise removal by tag, CSS class, id, aria-hidden, and inline display:none.
• Main-content extraction via link-density and content-density heuristics.
• Links: inline [text](url) or reference-style [text][N], with optional UTM/tracker stripping (courlan).
• Citations: a Markdown variant with footnotes [1], [2] plus a reference list.
• Metadata: title, first h1, description, and counters for words / characters / headings / links.
• Publication / update dates (optional, via htmldate).
• Mojibake repair (ftfy), NFKC Unicode normalization, HTML sanitization (nh3).
• Synchronous and asynchronous APIs, with a lazily generated raw_markdown.
• Immutable options (MarkdownOptions.with_overrides(...)) — thread-safe.

Installation

Requires Python 3.11+.

pip install html2md-clean

With optional extras (dates, sanitization, encoding fixes, URL cleaning):

pip install "html2md-clean[full]"

Install straight from the github repository:

pip install git+https://github.com/0-EternalJunior-0/html2md-clean.git

Quick start

from markdown import MarkdownGenerator

html = """
<html><body>
  <nav>site menu</nav>
  <article>
    <h1>Article title</h1>
    <p>This is the main paragraph with useful text.</p>
  </article>
  <footer>&copy; 2026</footer>
</body></html>
"""

generator = MarkdownGenerator()
result = generator.generate_from_html(html)

print(result.fit_markdown)
# # Article title
#
# This is the main paragraph with useful text.

print(result.word_count, result.h1)

The shortest variant:

from markdown import MarkdownGenerator

result = MarkdownGenerator.quick_generate(html)
print(result.markdown)  # alias for fit_markdown

CLI

Once installed, the html2md command is available:

# from a file to stdout
html2md page.html

# from stdin to a file
cat page.html | html2md -o page.md

# keep links (disabled by default in the CLI profile)
html2md page.html --include-links

# force BeautifulSoup instead of Selectolax
html2md page.html --force-bs4

You can also run it as a module without installing:

python main_service.py page.html -o page.md

Usage examples

Custom options

from markdown import MarkdownGenerator, MarkdownOptions

options = MarkdownOptions(
    include_links=True,
    include_images=True,
    generate_citations=True,
    max_length=50_000,
    clean_urls=True,          # strip utm_*, fbclid, gclid
    base_url="https://example.com",
)

generator = MarkdownGenerator(options)
result = generator.generate_from_html(html)

print(result.markdown_with_citations)
for ref in result.references:
    print(ref)

Reference-style links

from markdown import MarkdownGenerator, MarkdownOptions
from markdown.options import LinkStyle

options = MarkdownOptions(link_style=LinkStyle.REFERENCE)
result = MarkdownGenerator(options).generate_from_html(html)

Asynchronous call

import asyncio
from markdown import generate_markdown_async

async def run():
    result = await generate_markdown_async(html)
    print(result.fit_markdown)

asyncio.run(run())

Extracting publication dates (optional)

from markdown import MarkdownGenerator, MarkdownOptions

options = MarkdownOptions(extract_dates=True)
result = MarkdownGenerator(options).generate_from_html(html)
print(result.published_date, result.updated_date)

Web-scraping facade

from main_service import HTMLToMarkdownConverter

converter = HTMLToMarkdownConverter()   # more aggressive scraping defaults
markdown = converter.convert(html)

Real-world example: configuring MarkdownOptions for scraping a blog

Below is how you would do it "for real": take the HTML of a news/blog page, deliberately enable/disable the options you need, and understand why each one matters.

from markdown import MarkdownGenerator, MarkdownOptions
from markdown.options import LinkStyle, AGGRESSIVE_NOISE_CLASSES, COMMON_NOISE_PATTERNS

options = MarkdownOptions(
    # --- main-content extraction ---
    use_content_density=True,       # look for <article>/<main>, not the whole <body>
    link_density_threshold=0.4,     # discard menus/footers more strictly (more links = noise)
    min_content_words=40,           # short blocks are not treated as main content

    # --- noise removal ---
    remove_ads=True,
    noise_classes=AGGRESSIVE_NOISE_CLASSES,   # + sidebar/related/comments/carousel
    remove_display_none=True,                 # drop hidden inline blocks
    dynamic_noise_patterns=COMMON_NOISE_PATTERNS,  # "5 min ago", "Apply now", counters

    # --- what to keep in the Markdown ---
    include_links=True,
    include_images=True,
    include_tables=True,
    generate_citations=True,        # produce a variant with footnotes [1], [2]

    # --- links and URLs ---
    link_style=LinkStyle.REFERENCE, # cleaner than inline in long articles
    normalize_urls=True,
    base_url="https://example-blog.com",  # relative /article -> absolute
    clean_urls=True,                # strip utm_*, fbclid, gclid

    # --- text normalization ---
    fix_text_encoding=True,         # repair mojibake (requires ftfy)
    normalize_unicode=True,         # fullwidth/ligatures -> ASCII (NFKC)
    max_length=200_000,             # don't truncate long-reads prematurely
)

generator = MarkdownGenerator(options)
result = generator.generate_from_html(html)

print(result.markdown_with_citations)   # text with [1], [2]
print("words:", result.word_count, "| links:", result.rendered_link_count)
for ref in result.references:
    print(ref)   # {'id': 1, 'text': '...', 'url': '...'}

Tip. Options are immutable, so keep one "base" profile and make targeted variations with with_overrides — no need to duplicate the whole set:

BASE = MarkdownOptions(use_content_density=True, include_links=True)

# for e-commerce cards: a shorter candidate-length threshold
ecommerce = BASE.with_overrides(main_content_min_length=100, include_images=True)

# for a "raw" dump without link filtering
raw_dump = BASE.with_overrides(include_links=False, use_content_density=False)

Public API

Import: from markdown import ...

Object	Purpose
MarkdownGenerator	The main generator. generate_from_html(html) -> MarkdownResult, quick_generate(html), generate(source).
MarkdownOptions	Immutable conversion config. Use with_overrides(**kwargs).
MarkdownResult	Result: fit_markdown, raw_markdown (lazy), markdown_with_citations, metadata and counters.
generate_markdown_async(html, options=None)	Async wrapper (thread executor).
configure_logging(level=WARNING)	Configure the package logger.
COMMON_NOISE_PATTERNS	A ready-made set of regexes for dynamic_noise_patterns.

Key MarkdownResult fields:

Field	Type	Description
fit_markdown / markdown	str	The main cleaned Markdown.
raw_markdown	str	The full Markdown without aggressive filtering (lazy generation).
markdown_with_citations	str	Variant with footnotes [1], [2].
references	list[dict]	List of sources for the citations.
title, h1, description	str	Page metadata.
word_count, char_count, heading_count, link_count, image_count	int	Statistics.
published_date, updated_date	str | None	Dates (only with extract_dates=True).
is_truncated	bool	Whether the text was truncated to max_length.

Conversion options

MarkdownOptions is the single source of truth for defaults. The most common:

Option	Default	Description
remove_nav / remove_header / remove_footer / remove_aside	True	Remove the corresponding semantic blocks.
remove_ads	True	Remove by noise_classes / noise_ids.
remove_hidden	True	Strip aria-hidden, HTML5 hidden, hidden classes.
include_links	True	Render links.
include_images	False	Render images.
include_tables / include_lists / include_code_blocks / include_blockquotes	True	Support the corresponding elements.
max_length	100000	Maximum text length.
min_paragraph_length	3	Minimum paragraph length.
generate_citations	False	Generate markdown_with_citations.
link_style	INLINE	INLINE or REFERENCE.
clean_urls	False	Strip UTM/tracker parameters (courlan).
normalize_urls / base_url	True / ""	Absolutize relative URLs.
use_content_density	False	Main-content extraction heuristic.
link_density_threshold	0.5	Link-density threshold for discarding menus/footers.
force_beautifulsoup	False	Force BS4 instead of Selectolax.

The full list is in markdown/options.py and in the documentation.

Options are immutable: create new variants with with_overrides:

base = MarkdownOptions()
scraping = base.with_overrides(include_links=False, max_length=200_000)

Optional dependencies

Some features are enabled by installing extra packages (the library degrades gracefully if they are missing):

Feature	Option	Package
Date extraction	extract_dates	htmldate
HTML sanitization	sanitize_html	nh3
Mojibake repair	fix_text_encoding	ftfy
URL cleaning	clean_urls	courlan
Boilerplate extraction	use_trafilatura_extraction	trafilatura
Fast parser	— (auto)	selectolax

Performance

• Selectolax is used automatically when installed and is significantly faster than BeautifulSoup.
• raw_markdown is generated lazily — you pay for it only when you access it.
• MarkdownOptions is immutable, so generator instances can safely be shared across threads.