split-markdown4gpt: split large Markdown for LLM context windows
LLMs have finite context windows. A 50,000-word documentation site won't fit in one API call. split-markdown4gpt breaks Markdown files into token-bounded chunks that respect document structure — so each chunk stays coherent rather than cutting mid-sentence or mid-section.
Why splitting matters
Every LLM API call has a hard token limit (e.g. 4,096 tokens for GPT-3.5, 8,192 for GPT-4). Exceeding it raises an error. Naive character-based chunking cuts through sentences, code blocks, and headings — destroying context. This tool:
- Parses the Markdown AST (via
mistletoe) - Counts tokens using the same tokenizer the model uses (via
tiktoken) - Packs as many complete sections as fit within the limit
- When a single section exceeds the limit, falls back to sentence-level splitting (via
syntok)
The result: chunks that respect heading hierarchy, never split a sentence across chunks, and are as large as possible without exceeding the token budget.
Install
pip install split_markdown4gpt
CLI
mdsplit4gpt large_doc.md --model gpt-3.5-turbo --limit 4096 --separator "=== SPLIT ==="
Outputs all sections joined by the separator. Redirect to a file or pipe to another tool.
| Flag | Default | Description |
|---|---|---|
--model |
gpt-3.5-turbo |
Tokenizer model (determines token counting) |
--limit |
model max | Max tokens per chunk |
--separator |
=== SPLIT === |
String inserted between chunks in output |
Python API
from split_markdown4gpt import split
# Returns a list of Markdown strings, each within the token limit
chunks = split("large_doc.md", model="gpt-3.5-turbo", limit=4096)
for i, chunk in enumerate(chunks):
response = openai_client.chat.completions.create(
model="gpt-3.5-turbo", messages=[{"role": "user", "content": chunk}]
)
Also accepts a string or file-like object:
chunks = split(markdown_string, model="gpt-4", limit=8000)
chunks = split(open("doc.md"), limit=2000)
How it works
split(md)
├── meta_data() strip YAML front matter
├── MarkdownLLMSplitter.build_md_dict()
│ ├── mistletoe Document parse Markdown AST
│ └── build dict tree keyed by heading level
├── calculate_sizes() count tokens recursively (cached with lru_cache)
└── get_sections_from_md_dict_by_limit()
├── walk tree, accumulate blocks until limit reached
└── oversized blocks → sentence-split via syntok
MarkdownLLMSplitter class
from split_markdown4gpt import MarkdownLLMSplitter
splitter = MarkdownLLMSplitter(gptok_model="gpt-4", gptok_limit=6000)
splitter.load_md("doc.md")
splitter.build()
for section in splitter.gen_section_dicts():
print(section["gptok_size"], "tokens")
print(section["md"])
Methods: load_md(), load_md_path(), load_md_str(), build(), list_section_texts(), list_section_dicts(), gen_section_texts(), gen_section_dicts(), split().
Supported models
Any model supported by tiktoken, including all gpt-3.5-turbo and gpt-4 variants. The limit defaults to the model's published context window.
Dependencies
tiktoken— OpenAI-compatible token countingmistletoe— Markdown AST parsersyntok— sentence segmentation for oversized sectionsfrontmatter— YAML front matter strippingfire— CLIregex,PyYAML— utilities
License
Apache License 2.0. Copyright © 2023 Adam Twardoch.
Metadata
Release files for split-markdown4gpt 2.7.7
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | Provenance | Actions |
|---|---|---|---|---|
| split_markdown4gpt-2.7.7.tar.gz | 712.7 kB | With provenance | Details for split_markdown4gpt-2.7.7.tar.gz |
Built distribution (wheel)
| File | Interpreter | ABI | Platform | Actions Reset |
|---|---|---|---|---|
| split_markdown4gpt-2.7.7-py3-none-any.whl | Python 3 | none | any | Details for split_markdown4gpt-2.7.7-py3-none-any.whl |
Total release size: 727.3 kB
Release files / split_markdown4gpt-2.7.7.tar.gz
| Download URL | split_markdown4gpt-2.7.7.tar.gz |
|---|---|
| Size | 712.7 kB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
56a69385fafef60b589afae253cbbce5f65c26b1501b34b3a4a22d33ab3b3182
|
|
BLAKE2b-256 checksum How to use checksums |
6c98ffba706672a358a7a18e711d941b0224f3e794b3b395105b5440de9b3df3
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
Yes |
| Uploaded via |
twine/7.0.0 CPython/3.13.16
|
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on Nov 17, 2026.
Transparency logRelease files / split_markdown4gpt-2.7.7-py3-none-any.whl
| Download URL | split_markdown4gpt-2.7.7-py3-none-any.whl |
|---|---|
| Size | 14.6 kB |
| Tags | Python 3 |
|
SHA-256 checksum How to use checksums |
754ea7f80172c57d81fbef62f74a45f2c7b8f7f29fb4ac906368d2bde3bf17b6
|
|
BLAKE2b-256 checksum How to use checksums |
9c48f748b7789d7eb23a2c129ce9f927cec0f43f13b56f100c96811d1b037cf7
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
Yes |
| Uploaded via |
twine/7.0.0 CPython/3.13.16
|
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on Nov 17, 2026.
Transparency logSecurity for split-markdown4gpt 2.7.7
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
Full provenance
Both files in this release have been uploaded with provenance metadata.
Attestation for split_markdown4gpt-2.7.7.tar.gz
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on Nov 17, 2026.
Transparency logA note on package security
PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.
To learn more about protecting yourself from malicious actors, visit our package security documentation.