A Python library for intelligent HTML segmentation and ROI extraction. It builds a DOM tree from raw HTML and extracts content-rich regions for efficient web scraping and analysis.

Navigation

Verified details

These details have been verified by PyPI
Project links
GitHub Statistics
Maintainers
Avatar for carlosplanchon from gravatar.com carlosplanchon

Unverified details

These details have not been verified by PyPI
Meta
Classifiers
Report project as malware

Project description

betterhtmlchunking

Banner

CI PyPI version Python versions License: MIT DeepWiki

A Python library for intelligently chunking HTML documents into structured, size-limited segments based on DOM tree analysis.

Our Discord Server:

https://discord.gg/2uVF5FZg

Overview

This library processes HTML content to split it into semantically coherent chunks while respecting specified size constraints. It analyzes the DOM structure to identify optimal split points, preserving contextual information and document hierarchy.

Key Features

  • Custom DOM tree representation.
  • Configurable chunk size limits (counting by text or HTML length).
  • Intelligent region-of-interest detection.
  • Dual output formats: HTML and plain text chunks.
  • Preservation of structure relationships.
  • Customizable tag filtering.

Installation

pip install betterhtmlchunking

Dependencies

  • Python 3.10+
  • attrs
  • treelib
  • beautifulsoup4
  • parsel-text
  • lxml
  • attrs-strict

Usage

Basic Example

from betterhtmlchunking import DomRepresentation
from betterhtmlchunking.main import ReprLengthComparisionBy
from betterhtmlchunking.main import tag_list_to_filter_out

html_content = """
<html>
  <body>
    <div id="content">
      <h1>Document Title</h1>
      <p>First paragraph...</p>
      <p>Second paragraph...</p>
    </div>
  </body>
</html>
"""

# Create document representation with 20 character chunks.
dom_repr = DomRepresentation(
    MAX_NODE_REPR_LENGTH=20,
    website_code=html_content,
    repr_length_compared_by=ReprLengthComparisionBy.HTML_LENGTH
    # tag_list_to_filter_out=["/head", "/header", "..."]  # By default tag_list_to_filter_out is used.
)
dom_repr.start()

# Render HTML:
for idx in dom_repr.tree_regions_system.sorted_roi_by_pos_xpath:
    print("*" * 50)
    print(f"IDX: {idx}")
    roi_html_render: str =\
        dom_repr.render_system.get_roi_html_render_with_pos_xpath(
            roi_idx=idx
        )
    print(roi_html_render)


# Render text:
for idx in dom_repr.tree_regions_system.sorted_roi_by_pos_xpath:
    print("*" * 50)
    print(f"IDX: {idx}")
    roi_text_render: str =\
        dom_repr.render_system.get_roi_text_render_with_pos_xpath(
            roi_idx=idx
        )
    print(roi_text_render)

Render output (HTML):

**************************************************
IDX: 0
<h1>
 Document Title
</h1>

**************************************************
IDX: 1
<p>
 First paragraph...
</p>

**************************************************
IDX: 2
<p>
 Second paragraph...
</p>

Render output (text):

**************************************************
IDX: 0
Document Title

**************************************************
IDX: 1
First paragraph...

**************************************************
IDX: 2
Second paragraph...

Configuration

Key Parameters

  • MAX_NODE_REPR_LENGTH: Maximum allowed length for each chunk (in characters)
  • repr_length_compared_by: Length calculation method:
    • ReprLengthComparisionBy.HTML_LENGTH: HTML source length
    • ReprLengthComparisionBy.TEXT_LENGTH: Rendered text length
  • website_code: Input HTML content

Advanced Features

# Access the DOM tree structure
tree = dom_repr.tree_representation.tree

# Get node metadata:
for node in tree.all_nodes():
    if node.data is not None:
        print(f"XPath: {node.identifier}")
        print(f"Text length: {node.data.text_length}")
        print(f"HTML length: {node.data.html_length}")

How It Works

  1. DOM Parsing

    • Builds a tree representation of the HTML document.
    • Calculates metadata (text length, HTML length) for each node.

  2. Region Detection

    • Uses Breadth First Search (BFS) to traverse the DOM tree in a level-order fashion, ensuring that each node is processed systematically.
    • Combines nodes until the specified size limit is reached.
    • Preserves parent-child relationships to maintain contextual integrity.

  3. Chunk Generation

    • Creates HTML chunks with original markup.
    • Generates parallel text-only chunks.
    • Maintains chunk order based on document structure.

Comparison to popular Chunking Techniques

The actual practice (Feb. 2025) is to use plain-text or token-based chunking strategies, primarily aimed at keeping prompts within certain token limits for large language models. This approach is ideal for quick semantic retrieval or QA tasks on unstructured text.

By contrast, betterhtmlchunking preserves the HTML DOM structure, calculating chunk boundaries based on each node’s text or HTML length. This approach is especially useful when you want to:

  • Retain or leverage the hierarchical relationships in the HTML (e.g., headings, nested divs)
  • Filter out undesired tags or sections (like <script> or <style>)
  • Pinpoint exactly where each chunk originated in the document (via positional XPaths)

You can even combine the two techniques if you need both structured extraction (via betterhtmlchunking) and LLM-friendly text chunking (via LangChain) for advanced tasks such as summarization, semantic search, or large-scale QA pipelines.

CLI

The package ships with a command line interface built with Typer. You can pipe HTML to the tool and work with chunks in multiple ways.

Basic Usage - Single Chunk

Get a specific chunk as HTML:

cat input.html | betterhtmlchunking --max-length 32768 --chunk-index 0 > chunk.html

By default the command reads from stdin, processes chunks up to a maximum length of 32,768 characters, and prints the HTML corresponding to chunk index 0 to stdout.

List All Chunks

See how many chunks are created and their sizes without processing the full content:

cat input.html | betterhtmlchunking --max-length 32768 --list-chunks

Output:

Total chunks: 5
Chunk 0: 28543 chars HTML, 12834 chars text
Chunk 1: 31245 chars HTML, 15234 chars text
Chunk 2: 19823 chars HTML, 9234 chars text
...

Save All Chunks

Extract all chunks to separate files:

cat input.html | betterhtmlchunking --max-length 32768 --all-chunks --output-dir ./chunks

This creates chunk_0.html, chunk_1.html, etc. in the specified directory.

Text-Only Output

Get plain text content without HTML markup:

# Single chunk as text
cat input.html | betterhtmlchunking --max-length 32768 --chunk-index 0 --text-only > chunk.txt

# All chunks as text files
cat input.html | betterhtmlchunking --max-length 32768 --all-chunks --text-only --output-dir ./chunks

This creates chunk_0.txt, chunk_1.txt, etc. with plain text content.

JSON Output

Get structured JSON output with all chunks and metadata:

cat input.html | betterhtmlchunking --max-length 32768 --format json > output.json

Output format:

{
  "total_chunks": 5,
  "max_length": 32768,
  "compared_by": "html",
  "chunks": [
    {
      "index": 0,
      "html": "<h1>...</h1>",
      "text": "...",
      "html_length": 28543,
      "text_length": 12834
    },
    ...
  ]
}

This is useful for programmatic processing. You can pipe the output to tools like jq or process it with scripts:

# Extract only text from all chunks
cat input.html | betterhtmlchunking --format json | jq -r '.chunks[].text'

# Get chunk count
cat input.html | betterhtmlchunking --format json | jq '.total_chunks'

# Filter chunks by size
cat input.html | betterhtmlchunking --format json | jq '.chunks[] | select(.text_length > 1000)'

Verbose Mode

Enable progress logging with --verbose. Logs are written to stderr so they don't interfere with chunk output:

cat input.html | betterhtmlchunking --max-length 32768 --all-chunks --output-dir ./chunks --verbose

Maximal Verbose Mode

For detailed inspection of the DOM, nodes, ROIs, and chunk lengths, use --maximal-verbose. This logs:

  • Total DOM nodes and their HTML/text lengths
  • Each ROI (region of interest) with constituent node XPaths and lengths
  • Final chunk HTML and text sizes
cat input.html | betterhtmlchunking --max-length 32768 --chunk-index 0 --maximal-verbose > chunk.html 2> logs.txt
  • chunk.html contains the selected chunk HTML.
  • logs.txt captures all detailed logging information.

This mode is useful for debugging, testing, or analyzing how the document is split into chunks.

CLI Options Summary

Option Short Description
--max-length -l Maximum length for each chunk (default: 32768)
--chunk-index -c Index of specific chunk to output (default: 0)
--list-chunks List all chunks with their sizes
--all-chunks Output all chunks (requires --output-dir)
--output-dir -o Directory to save chunks when using --all-chunks
--text-only Output plain text instead of HTML
--format -f Output format: json for structured JSON output
--text Compare length using text instead of HTML
--verbose -v Enable progress logging
--maximal-verbose Enable detailed debug logging

License

MIT License

Contributing

Feel free to open issues or submit pull requests if you have suggestions or improvements.

Project details

Verified details

These details have been verified by PyPI
Project links
GitHub Statistics
Maintainers
Avatar for carlosplanchon from gravatar.com carlosplanchon

Unverified details

These details have not been verified by PyPI
Meta
Classifiers

Release history Release notifications | RSS feed

0.13.3

0.13.2

0.13.1

0.13.0

0.12.0

0.11.0

This version

0.10.0

0.9.8

0.9.7

0.9.6

0.9.5

0.9.4

0.9.3

0.9.2

0.9.1

Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

betterhtmlchunking-0.10.0.tar.gz (24.6 kB view details)

Uploaded Source

Built Distribution

If you're not sure about the file name format, learn more about wheel file names.

betterhtmlchunking-0.10.0-py3-none-any.whl (17.6 kB view details)

Uploaded Python 3

File details

Details for the file betterhtmlchunking-0.10.0.tar.gz.

File metadata

  • Download URL: betterhtmlchunking-0.10.0.tar.gz
  • Upload date:
  • Size: 24.6 kB
  • Tags: Source
  • Uploaded using Trusted Publishing? Yes
  • Uploaded via: twine/6.1.0 CPython/3.13.12

File hashes

Hashes for betterhtmlchunking-0.10.0.tar.gz
Algorithm Hash digest
SHA256 47775b4f7b0fc8b79aac031107da3a269816b18e42cd386aadfc39b9b8af5cd5
MD5 2a943ceb0fa8aefc7d66df7f16f87968
BLAKE2b-256 2112d01414b6416a7d04dbe3157820ac45f0bfdf0e06492ad20cf1e5e4066669

See more details on using hashes here.

Provenance

The following attestation bundles were made for betterhtmlchunking-0.10.0.tar.gz:

Publisher: release.yml on carlosplanchon/betterhtmlchunking

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

File details

Details for the file betterhtmlchunking-0.10.0-py3-none-any.whl.

File metadata

File hashes

Hashes for betterhtmlchunking-0.10.0-py3-none-any.whl
Algorithm Hash digest
SHA256 a5f0f34515ae0a8f25d936a3de12508f8c4b47a04bb8de597d55fef735657a7e
MD5 4fa16d686b572118bca39e7d2cf1a2d6
BLAKE2b-256 751b514866e13daebb902685f36d4962322961d87228be14ff736745fb4d91d7

See more details on using hashes here.

Provenance

The following attestation bundles were made for betterhtmlchunking-0.10.0-py3-none-any.whl:

Publisher: release.yml on carlosplanchon/betterhtmlchunking

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

Supported by

AWS Cloud computing and Security Sponsor Datadog Monitoring Depot Continuous Integration Fastly CDN Google Download Analytics Pingdom Monitoring Sentry Error logging StatusPage Status page