extracts the main content from web pages and returns cleaned HTML, optional markdown, and structured metadata.

Navigation

Verified details

These details have been verified by PyPI
Project links
GitHub Statistics
Maintainers
Avatar for jmcristobal from gravatar.com jmcristobal

Unverified details

These details have not been verified by PyPI
Meta
Classifiers
Report project as malware

Project description

domdown

domdown banner

domdown turns article-like web pages into clean, structured Markdown.

It is built for pages where the shape matters: long-form posts, research writeups, technical blogs, security reports, and other content-heavy pages that need to become readable Markdown without losing useful structure.

What it does

domdown takes care of the full HTML-to-Markdown pipeline:

  • Parses messy web HTML
  • Selects the main article content
  • Removes navigation, promo blocks, and other chrome
  • Extracts metadata
  • Preserves images, tables, code blocks, links, and lists
  • Optionally emits YAML frontmatter
  • Renders the final Markdown document

The result is Markdown that is ready to read, reuse, archive, or feed into another model.

Why it exists

Most pages are not written like clean documents. They mix article content with menus, banners, share widgets, related links, and other page furniture.

domdown is designed for cases where you want the content to stay faithful to the original page while still producing a clean Markdown output that is easy to consume downstream.

Example

from domdown import DomdownOptions, html_to_markdown

html = """
<html>
  <head>
    <title>Credential theft campaign expands</title>
    <meta name="description" content="A concise security article." />
    <link rel="canonical" href="https://example.com/research/campaign" />
  </head>
  <body>
    <nav>Home Pricing Docs</nav>
    <article>
      <h1>Credential theft campaign expands</h1>
      <p>Researchers observed a new wave of phishing infrastructure.</p>
      <figure>
        <img src="/images/chart.png" alt="Campaign infrastructure chart" />
        <figcaption>Campaign infrastructure by week.</figcaption>
      </figure>
      <ul>
        <li>Windows targets increased.</li>
        <li>Linux staging remained stable.</li>
      </ul>
    </article>
  </body>
</html>
"""

markdown = html_to_markdown(
    html,
    DomdownOptions(
        base_url="https://example.com/research/campaign",
    ),
)

print(markdown)

Output:

---
title: Credential theft campaign expands
source: "https://example.com/research/campaign"
domdown_version: 0.3.1
description: A concise security article.
---
# Credential theft campaign expands

Researchers observed a new wave of phishing infrastructure.

![Campaign infrastructure chart](https://example.com/images/chart.png)

Campaign infrastructure by week.

- Windows targets increased.
- Linux staging remained stable.

What it preserves

domdown is optimized for article-style pages where useful structure should survive the conversion:

  • Titles and headings
  • Visible author and publication metadata
  • Canonical URLs and source references
  • Images and captions
  • Tables and code blocks
  • Inline links and emphasized text
  • Lists, quotes, and other document structure

Using domdown

Client usage

Use html_to_markdown() when you only need the final Markdown document as a string.

from domdown import DomdownOptions, html_to_markdown

markdown = html_to_markdown(
    html,
    DomdownOptions(
        base_url="https://example.com/post",
        emit_frontmatter=False,
    ),
)

When emit_frontmatter=True or left at the default, the returned string includes YAML frontmatter followed by the Markdown body.

API usage

Use HtmlToMarkdownPipeline when you want structured output.

from domdown import DomdownOptions, HtmlToMarkdownPipeline

pipeline = HtmlToMarkdownPipeline(
    DomdownOptions(base_url="https://example.com/post")
)
result = pipeline.run(html)

print(result.document)
print(result.markdown)
print(result.cleaned_html)
print(result.frontmatter)
print(result.warnings)

HtmlToMarkdownResult exposes:

Field Type Description
markdown str Markdown rendered from the selected content.
cleaned_html str | None HTML after parsing, selection, cleaning, and preservation.
metadata HtmlMetadata | None Normalized metadata extracted from the source HTML.
frontmatter str | None YAML frontmatter when enabled.
document str | None Final document string, including frontmatter when enabled.
warnings tuple[str, ...] Non-fatal pipeline warnings.

HtmlMetadata exposes:

Field Type
title str | None
site_name str | None
source str | None
author tuple[str, ...]
published str | None
created str | None
description str | None
tags tuple[str, ...]
language str | None
canonical_url str | None
image str | None

Options

DomdownOptions controls parsing, cleanup, metadata extraction, and output shaping.

Option Default Behavior
base_url None Source URL used for metadata and relative URL resolution.
frontmatter_opts {} Per-key fallback values used when rendering frontmatter.
created None Creation date to include in metadata/frontmatter.
extract_metadata True Enables metadata extraction.
emit_frontmatter True Prepends YAML frontmatter to document.
prefer_article_body True Prefers article-like containers during selection.
author_priority "visible" Chooses visible author text before metadata unless set otherwise.
frontmatter_tags () Extra tags to include in generated frontmatter.
preserve_images True Keeps images for Markdown rendering.
preserve_tables True Keeps tables for Markdown rendering.
preserve_code_blocks True Keeps code/preformatted blocks.
strip_hidden True Removes hidden or non-visible elements.
remove_selectors () CSS selectors to remove.
keep_selectors () CSS selectors to protect during cleaning.
unwrap_selectors () CSS selectors whose wrapper is removed while children remain.

Example:

from domdown import DomdownOptions

options = DomdownOptions(
    base_url="https://example.com/article",
    frontmatter_opts={
        "title": "Example Article",
        "source": "https://example.com/article",
    },
    emit_frontmatter=True,
    preserve_images=True,
    remove_selectors=(".share-widget", ".newsletter-signup"),
)

Real-world coverage

domdown includes curated real-world HTML/Markdown pairs under tests/real/ to protect the pipeline against regressions on live site shapes.

  • html/ stores the captured HTML for each case.
  • raw/ stores the expected Markdown output for the same case.
  • manifest.json declares the cases and their relative fixture paths.

To run the real-example suite:

pytest tests/real/test_real_examples.py -q

Public API

domdown exports these names from domdown.__init__:

from domdown import (
    DomdownOptions,
    HtmlMetadata,
    HtmlToMarkdownPipeline,
    HtmlToMarkdownResult,
    html_to_markdown,
)

Installation

Install from this repository:

pip install domdown

Install locally for development:

git clone https://github.com/juanmcristobal/domdown.git
cd domdown
pip install -e ".[dev]"

Runtime dependencies:

  • beautifulsoup4
  • lxml
  • soupsieve
  • httpx

Support & Connect

History

0.1.0 (2026-05-21)

  • First release.

0.2.0 (2026-06-03)

  • Fix release workflow checkout for PyPI publish.
  • Change installation instructions to use pip install domdown (breaking change).

Project details

Verified details

These details have been verified by PyPI
Project links
GitHub Statistics
Maintainers
Avatar for jmcristobal from gravatar.com jmcristobal

Unverified details

These details have not been verified by PyPI
Meta
Classifiers

Release history Release notifications | RSS feed

0.3.6

0.3.5

0.3.4

0.3.3

0.3.2

This version

0.3.1

0.3.0

0.1.1

Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

domdown-0.3.1.tar.gz (8.6 MB view details)

Uploaded Source

Built Distribution

If you're not sure about the file name format, learn more about wheel file names.

domdown-0.3.1-py3-none-any.whl (55.4 kB view details)

Uploaded Python 3

File details

Details for the file domdown-0.3.1.tar.gz.

File metadata

  • Download URL: domdown-0.3.1.tar.gz
  • Upload date:
  • Size: 8.6 MB
  • Tags: Source
  • Uploaded using Trusted Publishing? Yes
  • Uploaded via: twine/6.1.0 CPython/3.13.12

File hashes

Hashes for domdown-0.3.1.tar.gz
Algorithm Hash digest
SHA256 7ba84fe29532ff8a454dcbdf04e55ef911ffb677a0ae1d5f6b433fd3637a2045
MD5 d1b068b816625a4788be8872ffe204fd
BLAKE2b-256 24815d3d1cf4f3d8272bae8e7350b2a6483ca160d238850176c4dec8d8e38659

See more details on using hashes here.

Provenance

The following attestation bundles were made for domdown-0.3.1.tar.gz:

Publisher: release.yml on juanmcristobal/domdown

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

File details

Details for the file domdown-0.3.1-py3-none-any.whl.

File metadata

  • Download URL: domdown-0.3.1-py3-none-any.whl
  • Upload date:
  • Size: 55.4 kB
  • Tags: Python 3
  • Uploaded using Trusted Publishing? Yes
  • Uploaded via: twine/6.1.0 CPython/3.13.12

File hashes

Hashes for domdown-0.3.1-py3-none-any.whl
Algorithm Hash digest
SHA256 2d7e1b8e6e19eb41fc02b25efe1586b999a867c97ddb3121a051ef8d69167bbd
MD5 86981c6d4bbd7cd5b312c90fd06268fc
BLAKE2b-256 85ae997300dc2c91e60d87a5f349f3caf20d4afc9ed2574699ab0bd46906af64

See more details on using hashes here.

Provenance

The following attestation bundles were made for domdown-0.3.1-py3-none-any.whl:

Publisher: release.yml on juanmcristobal/domdown

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

Supported by

AWS Cloud computing and Security Sponsor Datadog Monitoring Depot Continuous Integration Fastly CDN Google Download Analytics Pingdom Monitoring Sentry Error logging StatusPage Status page