markdown-chunkify

A Python library that splits Markdown into hierarchical sections. It keeps code blocks intact and links each section to its parent section.

Features

• Lines inside fenced code blocks (``` or ~~~, including fences in list items), HTML comment blocks, <pre>, <script>, <style> and <textarea> blocks and YAML front matter are never treated as headers. Section text keeps them exactly as written
• Splits on headers from H1 to H6. MarkdownSplitter(max_level=3) keeps deeper headers as text of the enclosing section
• Gives each section an id and its parent section's id, so the header hierarchy can be rebuilt as a tree or loaded into a graph. Headers with the same text are kept as separate sections
• Text before the first header, or a document without headers, is returned as a level-0 section with an empty header
• section.breadcrumb() returns the section's header path, such as Docker > Installation, to send with the section text to an embedding model or LLM
• Records where each section is in the source text. text[start:end] with start, end = section.metadata.span returns the section's header line and text as written. metadata.char_count is the length of that span, not of content or to_markdown()

Installation

pip install markdown-chunkify

The splitter depends on Pydantic.

Usage

from markdown_chunkify import MarkdownSplitter

splitter = MarkdownSplitter()

# Split from text
sections = splitter.split_text(markdown_text)

# Split from file
sections = splitter.split_file("document.md")

MarkdownSplitter is immutable, so one instance can be reused.

Section ids and parents

metadata.id is the section's position in document order, starting at 0, so sections[i].metadata.id == i. metadata.parent_id is the id of the enclosing section, or None for top-level sections and text before the first header.

text = "# Docker\n## Installation\n## Usage\n# Kubernetes"
sections = MarkdownSplitter().split_text(text)

[(s.header, s.metadata.id, s.metadata.parent_id) for s in sections]
# [('Docker', 0, None), ('Installation', 1, 0), ('Usage', 2, 0), ('Kubernetes', 3, None)]

sections[sections[1].metadata.parent_id].header  # 'Docker'

The other relationships follow from these two fields:

• A section's children are the sections whose parent_id is its id.
• Sections with the same parent_id are siblings, whatever their header levels. In # A, ### C, ## B, both C and B are children of A. A skipped level adds no placeholder section.
• The previous and next sections in reading order have ids id - 1 and id + 1.

Ids are unique within one document. To use them across documents, combine them with your own document key, such as f"{doc_key}:{section.metadata.id}". Ids are positions, not content versions: inserting or removing a section renumbers the sections after it, and editing a section's text keeps its id. To detect changed sections between runs, for example to skip re-extracting entities from unchanged text, hash content. Changing max_level changes the tree and the ids.

Breadcrumbs

section.breadcrumb() joins the parent headers and the section's own header. Sent with the section text, it tells an embedding model or LLM what the section is about when the text itself says "it" or "this step".

text = "# Docker\n## Installation\nRun the installer."
section = MarkdownSplitter().split_text(text)[1]

section.breadcrumb()  # 'Docker > Installation'
section.breadcrumb(separator=" / ")  # 'Docker / Installation'

prompt_text = f"{section.breadcrumb()}\n\n{section.content}"

Text before the first header returns an empty string. The breadcrumb is computed from metadata.parents, so it is not part of the serialized section.

Cleaning text from PDF parsers

PDF parsers often produce ligatures (fi, fl), non-breaking spaces and fullwidth characters. A chunk with find doesn't match a search for find, and entity extraction can miss the word. The splitter doesn't clean text. Clean it before calling split_text, so the spans index into the cleaned string you pass in.

import unicodedata

from markdown_chunkify import MarkdownSplitter

clean = unicodedata.normalize("NFKC", pdf_markdown)
clean = clean.replace("\u00ad", "").replace("\u200b", "")  # Soft hyphen, zero-width space
sections = MarkdownSplitter().split_text(clean)

NFKC normalization from the standard library replaces ligatures, non-breaking spaces and fullwidth characters, and keeps accents:

NFKC also folds superscripts and subscripts, so x² becomes x2. It leaves soft hyphens and zero-width spaces in the text, so remove them with str.replace.

Limitations

• Only ATX headers (# to ######) are recognized. Setext headers, underlined with === or ---, stay in the section text.
• Headers and fences inside blockquotes (> # Title) are not recognized.
• A line indented by 4 or more spaces that starts with ``` or ~~~ is treated as a fence, even inside an indented code block. If that fence is never closed, the headers after it are treated as text and a warning is logged.
• An HTML comment or a <pre>, <script>, <style> or <textarea> block is only recognized when it starts the line. One that opens mid-line (text <!-- or text <pre>) does not hide the # lines inside it.
• Headers inside <details> are split, as GitHub renders Markdown after a blank line inside <details>. The <details> and </details> tags stay in the text of the sections around them.
• A header with no text (#, ## ##, or only whitespace after the #) is kept as text of the enclosing section, not an empty section. CommonMark renders it as an empty heading.
• A document that starts with --- is treated as having YAML front matter only if the line after it is a key: entry and every line up to the next --- or ... looks like YAML (key: value, - item, indented lines, comments or blank lines). Otherwise the --- is a horizontal rule and the headers after it are split as usual.
• Sections have no size limit. A document without headers is returned as one section and a long section stays whole, so split long sections before embedding them.

Data Models

Section and SectionMetadata are exported from the package root:

from markdown_chunkify import Section
from markdown_chunkify import SectionMetadata

The models are frozen, so to change a section, call section.model_copy(update={"content": "..."}), which returns a changed copy. The freezing is shallow: metadata.parents is a plain dict, so treat it as read-only.

Section

A section returned by the split_text and split_file methods. It holds the section's header, text, header level and metadata.

class Section(BaseModel):
    model_config = ConfigDict(frozen=True)

    # The header of the section (without #), empty for text before the first header
    header: str
    # The content of the section
    content: str
    # Number of # symbols (1-6), 0 for text before the first header
    level: int
    # Hierarchy and position information
    metadata: SectionMetadata

    # Convert the section back to Markdown
    def to_markdown(self) -> str: ...

    # Join the parent headers and the section's header, such as "Docker > Installation"
    def breadcrumb(self, separator: str = " > ") -> str: ...

SectionMetadata

A section's id, its parent's id, its parent headers and where it is in the source text.

class SectionMetadata(BaseModel):
    model_config = ConfigDict(frozen=True)

    # Position in document order, starting at 0
    id: int
    # Id of the enclosing section, None for top-level sections
    parent_id: int | None
    # Parent headers by level, such as {"h1": "Guide", "h2": "Install"}
    parents: dict[str, str]
    # Start (inclusive) and end (exclusive) offsets in the source text
    span: tuple[int, int]
    # Number of characters in the span, computed from it
    char_count: int

The span starts at the header line and ends at the last non-whitespace character of the section text. It does not include subsections, leading blank lines or trailing whitespace. Offsets count Python characters (code points), not UTF-8 bytes or UTF-16 units, in the string passed to split_text, including a byte order mark. split_file offsets index into the file's contents as decoded with its encoding, with line endings kept as written.

Requirements

Python 3.12 or later.

Contributing

Report bugs and request features in GitHub issues. For a large change, open an issue to discuss it before you start.

Set up a development environment with uv:

git clone https://github.com/Gal-Gilor/markdown-chunkify.git
cd markdown-chunkify
uv sync

Run the tests and Ruff before opening a pull request:

uv run pytest -vv
uv run ruff check .
uv run ruff format .

Open pull requests against main. The Lint, test and build workflow runs Ruff, the tests on Python 3.12, 3.13 and 3.14, and a wheel build on each pull request. Add tests for changed behavior, and add user-facing changes to CHANGELOG.md.

To release a version, update version in pyproject.toml and the changelog, then publish a GitHub release tagged v<version>, such as v0.3.0. The Release workflow builds the package and publishes it to PyPI with trusted publishing.

License

Apache License 2.0. See LICENSE.