Skip to main content

split-markdown4gpt: split large Markdown for LLM context windows

LLMs have finite context windows. A 50,000-word documentation site won't fit in one API call. split-markdown4gpt breaks Markdown files into token-bounded chunks that respect document structure — so each chunk stays coherent rather than cutting mid-sentence or mid-section.

Why splitting matters

Every LLM API call has a hard token limit (e.g. 4,096 tokens for GPT-3.5, 8,192 for GPT-4). Exceeding it raises an error. Naive character-based chunking cuts through sentences, code blocks, and headings — destroying context. This tool:

  1. Parses the Markdown AST (via mistletoe)
  2. Counts tokens using the same tokenizer the model uses (via tiktoken)
  3. Packs as many complete sections as fit within the limit
  4. When a single section exceeds the limit, falls back to sentence-level splitting (via syntok)

The result: chunks that respect heading hierarchy, never split a sentence across chunks, and are as large as possible without exceeding the token budget.

Install

pip install split_markdown4gpt

CLI

mdsplit4gpt large_doc.md --model gpt-3.5-turbo --limit 4096 --separator "=== SPLIT ==="

Outputs all sections joined by the separator. Redirect to a file or pipe to another tool.

Flag Default Description
--model gpt-3.5-turbo Tokenizer model (determines token counting)
--limit model max Max tokens per chunk
--separator === SPLIT === String inserted between chunks in output

Python API

from split_markdown4gpt import split

# Returns a list of Markdown strings, each within the token limit
chunks = split("large_doc.md", model="gpt-3.5-turbo", limit=4096)

for i, chunk in enumerate(chunks):
    response = openai_client.chat.completions.create(
        model="gpt-3.5-turbo", messages=[{"role": "user", "content": chunk}]
    )

Also accepts a string or file-like object:

chunks = split(markdown_string, model="gpt-4", limit=8000)
chunks = split(open("doc.md"), limit=2000)

How it works

split(md)
  ├── meta_data()        strip YAML front matter
  ├── MarkdownLLMSplitter.build_md_dict()
  │     ├── mistletoe Document  parse Markdown AST
  │     └── build dict tree keyed by heading level
  ├── calculate_sizes()  count tokens recursively (cached with lru_cache)
  └── get_sections_from_md_dict_by_limit()
        ├── walk tree, accumulate blocks until limit reached
        └── oversized blocks → sentence-split via syntok

MarkdownLLMSplitter class

from split_markdown4gpt import MarkdownLLMSplitter

splitter = MarkdownLLMSplitter(gptok_model="gpt-4", gptok_limit=6000)
splitter.load_md("doc.md")
splitter.build()

for section in splitter.gen_section_dicts():
    print(section["gptok_size"], "tokens")
    print(section["md"])

Methods: load_md(), load_md_path(), load_md_str(), build(), list_section_texts(), list_section_dicts(), gen_section_texts(), gen_section_dicts(), split().

Supported models

Any model supported by tiktoken, including all gpt-3.5-turbo and gpt-4 variants. The limit defaults to the model's published context window.

Dependencies

  • tiktoken — OpenAI-compatible token counting
  • mistletoe — Markdown AST parser
  • syntok — sentence segmentation for oversized sections
  • frontmatter — YAML front matter stripping
  • fire — CLI
  • regex, PyYAML — utilities

License

Apache License 2.0. Copyright © 2023 Adam Twardoch.

Metadata

Release files for split-markdown4gpt 2.7.7

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for split-markdown4gpt 2.7.7
File Size Uploaded Provenance Actions
split_markdown4gpt-2.7.7.tar.gz 712.7 kB With provenance Details for split_markdown4gpt-2.7.7.tar.gz

Built distribution (wheel)

Table of built distributions (wheels) for split-markdown4gpt 2.7.7
File Interpreter ABI Platform Actions
split_markdown4gpt-2.7.7-py3-none-any.whl Python 3 none any Details for split_markdown4gpt-2.7.7-py3-none-any.whl

Total release size: 727.3 kB

Release files / split_markdown4gpt-2.7.7.tar.gz

Download URL split_markdown4gpt-2.7.7.tar.gz
Size 712.7 kB
Tags Source
SHA-256 checksum
How to use checksums
56a69385fafef60b589afae253cbbce5f65c26b1501b34b3a4a22d33ab3b3182
BLAKE2b-256 checksum
How to use checksums
6c98ffba706672a358a7a18e711d941b0224f3e794b3b395105b5440de9b3df3
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
Yes
Uploaded via twine/7.0.0 CPython/3.13.16

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Nov 17, 2026.

Transparency log

Release files / split_markdown4gpt-2.7.7-py3-none-any.whl

Download URL split_markdown4gpt-2.7.7-py3-none-any.whl
Size 14.6 kB
Tags Python 3
SHA-256 checksum
How to use checksums
754ea7f80172c57d81fbef62f74a45f2c7b8f7f29fb4ac906368d2bde3bf17b6
BLAKE2b-256 checksum
How to use checksums
9c48f748b7789d7eb23a2c129ce9f927cec0f43f13b56f100c96811d1b037cf7
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
Yes
Uploaded via twine/7.0.0 CPython/3.13.16

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Nov 17, 2026.

Transparency log

Security for split-markdown4gpt 2.7.7

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

Full provenance

Both files in this release have been uploaded with provenance metadata.

Select from 2 files

Attestation for split_markdown4gpt-2.7.7.tar.gz

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Nov 17, 2026.

Transparency log

A note on package security

PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.

To learn more about protecting yourself from malicious actors, visit our package security documentation.

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page