boundary-smart-splitter

A boundary-first, size-second text splitter for Python. LangChain-compatible. Framework-agnostic core. No ML dependencies.

Boundary-smart-splitter respects your text structure.
Word → Sentence → Paragraph → Section/Topic

Most text splitters hit a character count and slice — regardless of whether they cut through a sentence, a paragraph, or a heading. This library flips that: chunk_size is measured in semantic units (words, sentences, paragraphs, sections), and max_chars acts only as a hard safety ceiling to protect your embedding model's context window.

---

The Best of Both Worlds: Quality vs. Speed

When chunking text for LLMs and RAG pipelines, developers are usually forced to choose between two extremes:

1. Naive Splitters (e.g., RecursiveCharacterTextSplitter): Orders of magnitude faster, but completely blind to structure. They routinely slice through sentences, headings, and lists, splitting a single topic across multiple chunks. This leads to abysmal Topic Coherence (as low as 18.5% on complex documents).
2. Model-Based Splitters (e.g., SemanticChunker): Smart enough to find semantic shifts, but extremely slow (often 100ms–600ms+ per document), resource-heavy, and dependent on active embedding model calls or heavy Python ML libraries.

boundary-smart-splitter gives you the best of both worlds:

• ML-Grade Quality: By parsing the actual Markdown or HTML hierarchy, it preserves headings, lists, and tables, ensuring Topic Coherence of up to 88% with flawless sentence boundaries (10.0/10).
• Regex-Grade Speed: Runs entirely on lightweight, deterministic Python structural rules. No heavy dependencies, no GPUs, and no LLM API calls—delivering semantic chunks in 10ms to 50ms.

---

Benchmarks & Evaluation

While naive character-count splitters like RecursiveCharacterTextSplitter are orders of magnitude faster (as they run simple regex on character counts without structural analysis), they fail to preserve context.

On real-world articles, StructureSplitter shows a massive win in Topic Coherence (up to 88.1% vs 18.5% on the Lyzr Agent Orchestration article). This translates directly to better RAG retrieval accuracy and fewer hallucinations by keeping logical sections intact.

💡 Why do we win / tie?

• Irregular & Mixed Documents (Our Win Condition): When documents have mixed heading levels, custom lists, and uneven section lengths, naive splitters slice right through headings and split topics across chunks. StructureSplitter wins by massive margins.
• Uniform Documents (The Tie/Loss): On highly uniform, paragraph-structured content (like the DataCamp RAG Guide), naive splitting at double newlines (\n\n) naturally aligns with paragraph boundaries by luck, narrowing the quality gap while avoiding structural parsing overhead.

📐 Evaluation Methodology & Reproducing Results

Anyone can verify these benchmarks or evaluate their own splitters using the built-in Evaluator class:

from boundary_smart_splitter import Evaluator, StructureSplitter
from langchain_text_splitters import RecursiveCharacterTextSplitter

evaluator = Evaluator()

# Option A: Compare using built-in shorthand keys (automatically imported and configured)
# "recursive" -> RecursiveCharacterTextSplitter, "character" -> CharacterTextSplitter
report_shorthand = evaluator.evaluate(
    text=text, 
    splitter=StructureSplitter(),
    compare_with=["recursive", "character"]
)

# Option B: Pass custom pre-configured splitter instances directly
report_custom = evaluator.evaluate(
    text=text, 
    splitter=StructureSplitter(max_chars=1000),
    compare_with=[
        RecursiveCharacterTextSplitter(chunk_size=1000, chunk_overlap=0)
    ]
)

# Print a formatted Markdown comparison table of the results
print(report_shorthand.summary())

Scoring Weights

• Topic Coherence (40%): Heading Preservation Rate (20 pts) + (1.0 - Cross-topic Mixing Rate) (20 pts).
• Sizing Safety (30%): (1.0 - Under 100 Tokens Rate) (15 pts) + (1.0 - Over 512 Tokens Rate) (15 pts).
• Processing Speed (20%): Stepped score based on execution time:
  • $\le 1.0\text{ms}$: 20 pts
  • $\le 5.0\text{ms}$: 18 pts
  • $\le 20.0\text{ms}$: 15 pts
  • $\le 100.0\text{ms}$: 10 pts
  • $\le 500.0\text{ms}$: 5 pts
  • $> 500.0\text{ms}$: 2 pts
• Boundary Accuracy (10%): Linguistic boundary score out of 10 (no mid-sentence cuts = 10 pts, mid-sentence = 6 pts, mid-word = 0 pts).

1. AI Agents Glossary

Evaluated on the NVIDIA AI Agents Glossary Article:

Splitter	Overall Score (/100)	Topic Coherence	Sizing Safety	Avg Speed	Boundary Accuracy
StructureSplitter (Ours)	77.3	62.8%	90.5%	16.50ms	10.0/10 (Clean Sentence splits)
RecursiveCharacterTextSplitter	73.8	50.2%	92.5%	0.19ms	6.0/10 (Cuts mid-sentence)
TokenTextSplitter	55.3	6.1%	96.2%	2.42ms	6.0/10 (Cuts mid-sentence)
SemanticChunker	54.0	33.1%	85.7%	127.08ms	10.0/10 (Clean Sentence splits)

2. RAG Complete Guide

Evaluated on the DataCamp RAG Complete Guide Article:

Splitter	Overall Score (/100)	Topic Coherence	Sizing Safety	Avg Speed	Boundary Accuracy
RecursiveCharacterTextSplitter	82.3	65.6%	86.8%	0.10ms	10.0/10 (Paragraph aligned)
StructureSplitter (Ours)	79.9	70.2%	89.5%	11.41ms	10.0/10 (Clean Sentence splits)
TokenTextSplitter	55.2	6.2%	95.8%	2.08ms	6.0/10 (Cuts mid-sentence)
SemanticChunker	51.6	32.6%	78.6%	132.00ms	10.0/10 (Clean Sentence splits)

3. Agent Orchestration

Evaluated on the Lyzr Agent Orchestration Article:

Splitter	Overall Score (/100)	Topic Coherence	Sizing Safety	Avg Speed	Boundary Accuracy
StructureSplitter (Ours)	80.8	88.1%	85.2%	50.46ms	10.0/10 (Clean Sentence splits)
RecursiveCharacterTextSplitter	62.9	18.5%	98.4%	0.44ms	6.0/10 (Cuts mid-sentence)
SemanticChunker	50.9	43.7%	71.4%	684.26ms	10.0/10 (Clean Sentence splits)
TokenTextSplitter	49.7	11.6%	100.0%	7.02ms	0.0/10 (Cuts mid-word)

---

Installation

pip install boundary-smart-splitter

For LangChain support, install the optional dependency:

pip install boundary-smart-splitter[langchain]

Token-Aware Splitting (Highly Recommended)

To use token limits (max_tokens / min_tokens) accurately and consistently, you should install tiktoken:

pip install tiktoken

[!IMPORTANT] Strictly Recommended for Production: While boundary-smart-splitter runs out-of-the-box using character-count fallback approximation, installing tiktoken is strongly urged to prevent inconsistent split sizes. True token-aware boundaries prevent context-window overflow and guarantee optimal coherence for embedding models.

---

Design Philosophy

Principle	How it works
Boundary-first, size-second	Clean boundaries are more important than exact character counts
Semantic units	chunk_size means words for V1, sentences for V2, paragraphs for V3, sections for V4
max_chars as safety net	Never exceeds your embedding model's context window, but only as a last resort
Graceful fallback chain	V4 → V3 → V2 → V1 → hard char-cut
LangChain-compatible, not dependent	The core never imports LangChain; wrappers are optional

---

Quick Start

V1 — Word-Boundary Splitter

Never cuts a word in half. chunk_size = number of words.

from boundary_smart_splitter import WordSplitter

text = "The quick brown fox jumps over the lazy dog. " * 100
splitter = WordSplitter(chunk_size=60, max_chars=500, tolerance=10)
chunks = splitter.split(text)

for chunk in chunks:
    print(chunk)

---

V2 — Sentence-Boundary Splitter

Never cuts a sentence in half. Finds the nearest ., ?, or ! within tolerance.

from boundary_smart_splitter import SentenceSplitter

text = "Hello world. How are you? I am fine! " * 100
splitter = SentenceSplitter(chunk_size=5, max_chars=500, tolerance=2)
chunks = splitter.split(text)

for chunk in chunks:
    print(chunk)

Custom Abbreviations

Prevent mid-abbreviation splits (e.g. Dr., U.S.A., e.g.):

splitter = SentenceSplitter(
    chunk_size=5,
    max_chars=500,
    tolerance=2,
    abbreviations={"Dr.", "U.S.A.", "e.g.", "i.e."},
)

Boundary Preference

# Forward-first (default): scan forward for sentence end, then backward
splitter = SentenceSplitter(chunk_size=5, max_chars=500, tolerance=2, boundary_preference="forward")

# Backward-first: scan backward first, then forward
splitter = SentenceSplitter(chunk_size=5, max_chars=500, tolerance=2, boundary_preference="backward")

---

V3 — Paragraph-Boundary Splitter

Never breaks a paragraph (\n\n). chunk_size = number of paragraphs.

from boundary_smart_splitter import ParagraphSplitter

text = "Para one.\n\nPara two.\n\nPara three."
splitter = ParagraphSplitter(chunk_size=2, max_chars=500, tolerance=1)
chunks = splitter.split(text)

for chunk in chunks:
    print(chunk)

Markdown Mode

Respects horizontal rules and headings as extra paragraph boundaries:

splitter = ParagraphSplitter(
    chunk_size=2,
    max_chars=500,
    tolerance=1,
    use_markdown_mode=True,
)

---

V4 — Structure & Topic-Aware Splitter

Respects headings, numbered sections, and transition phrases. chunk_size = number of sections.

import tiktoken
from boundary_smart_splitter import StructureSplitter

text = """
# Intro
Welcome to our proposal.

1. Home Page
The homepage will act as a strong first impression...

2. About Us Page
To build credibility and trust.

However, we must also consider costs.

3. Services Page
A detailed breakdown of all services.
"""

splitter = StructureSplitter(
    chunk_size=1,
    max_tokens=300, # only remove if you want to go with normal flow
    max_chars=1500,
    respect_headings=True,
    respect_numbered_sections=True,
    split_on_transitions=True,
)
chunks = splitter.split(text)

for chunk in chunks:
    print("---")
    print(chunk)

V4 Configuration

Parameter	Default	Description
respect_headings	True	Treat detected headings as hard section boundaries
respect_numbered_sections	True	Treat 1. Title as a boundary
split_on_transitions	True	Treat "However," / "In summary," etc. as boundaries
transition_phrases	built-in list	Custom list of transition phrases
double_newline_as_boundary	True	Treat \n\n\n as a boundary
heading_detector	"auto"	Heading detector strategy ("auto", "markdown", "html", "plain", or a custom HeadingDetector class)
min_tokens	200	Target minimum tokens per chunk. Smaller sibling sections are merged up to this budget
orphan_strategy	"merge_backward"	How to handle content with no heading (boilerplate/byline/copyright). Choices: "merge_backward", "drop", "tag_only"
orphan_patterns	built-in list	List of regex patterns to match orphan/boilerplate text
overlap_mode	"heading"	Style of overlap: "heading" (context-aware: injects parent heading + first sentence context) or "tail" (repeats trailing tokens)
overlap_tokens	0	Size of overlap in tokens (opt-in)
overlap_prefix_template	"[context: {heading} — {first_sentence}]"	Template used in "heading" overlap mode
fallback_separators	["\n\n", "\n", ".", " "]	Ordered separators to fall back on when a section exceeds limits

Hierarchy-Aware Sibling Merging & Trailing Tails

Smaller sub-sections under the same parent heading path (siblings) are automatically merged up to the min_tokens budget (default 200 tokens). This prevents generating tiny chunks for brief subsections while avoiding cross-topic mixing (boundaries never cross top-level headings).

Rich Hierarchical Metadata

Use split_with_metadata() to retrieve chunk text along with structured metadata.

from boundary_smart_splitter import StructureSplitter

text = """
# Top Level
Welcome to the project.
## Sub 1
First sibling content.
## Sub 2
Second sibling content.
"""

splitter = StructureSplitter(min_tokens=50)
chunks = splitter.split_with_metadata(text)

for chunk in chunks:
    print(f"Text: {chunk.text!r}")
    print(f"Metadata: {chunk.metadata}")

Each Chunk contains text (str) and a metadata (dict) containing:

• section_title: The heading title of the section (str)
• heading_path: List of headings from root to this section (list of str)
• heading_level: Depth of the current section heading (int)
• token_count: Number of tokens in the chunk (int)
• char_count: Character length of the chunk (int)
• boundary_type: How the boundary was determined ("merged_sibling", "soft", "hard")
• overlap_injected: Whether context overlap was injected into the chunk (bool)
• source_format: Format detected ("markdown", "html", "plain")
• section_index: Order of section in the output (int)
• position_in_doc: Position in document ("start", "middle", "end")

Streaming & Generator Outputs

For memory-efficient processing of extremely large documents, all splitters support generator-based streaming via split_stream(). For V4 StructureSplitter, you can also use split_with_metadata_stream() to dynamically stream chunks along with metadata:

from boundary_smart_splitter import StructureSplitter

splitter = StructureSplitter(chunk_size=1)

# Yields chunks dynamically as they are formed without full buffering
for chunk in splitter.split_stream(large_text):
    print(chunk)

# Yields rich Chunk objects containing text and metadata
for chunk_obj in splitter.split_with_metadata_stream(large_text):
    print(chunk_obj.text, chunk_obj.metadata)

Context-Aware Overlaps

Instead of blindly repeating trailing characters, V4 can inject parent heading and first-sentence context from the previous chunk.

splitter = StructureSplitter(overlap_tokens=30, overlap_mode="heading")

Configurable Fallback Chains

Customize the order of separators when a section is oversized and needs to be split:

# Fall back sequentially using custom separators
splitter = StructureSplitter(
    max_chars=1000,
    fallback_separators=["\n\n", "|", "\n", " "]
)

---

Token-Aware Splitting

All splitters support token-aware splitting. While max_chars limits chunks by character length, you can also enforce max_tokens (measured using tiktoken or a custom callable).

import tiktoken
from boundary_smart_splitter import StructureSplitter

splitter = StructureSplitter(
    chunk_size=1,
    max_tokens=300,            # Hard token ceiling per chunk
    max_chars=1500,            # Optional secondary character ceiling
    length_function="gpt-4",   # Can be encoding name (e.g. "cl100k_base"), model name, or callable
)

If tiktoken is not installed, the library gracefully falls back to character-length counting (using len()) and issues an ImportWarning.

---

Heading Detection Interface

V4 StructureSplitter supports pluggable heading detection. You can specify a string key ("markdown", "html", "plain") or pass a custom class inheriting from HeadingDetector:

from boundary_smart_splitter import StructureSplitter, HeadingDetector, HeadingResult

class CustomHeadingDetector(HeadingDetector):
    def detect(self, line: str, context=None):
        if line.startswith("!!!"):
            return HeadingResult(level=1, text=line.strip("! "))
        return None

splitter = StructureSplitter(heading_detector=CustomHeadingDetector())

---

Fallback Chain

When a single unit exceeds max_chars or max_tokens, the library gracefully falls back to the next-smaller boundary:

StructureSplitter (V4)
  → section too big? → ParagraphSplitter (V3)
    → paragraph too big? → SentenceSplitter (V2)
      → sentence too big? → WordSplitter (V1)
        → word too big? → hard cut (character or token bounds)

This ensures you never exceed your model constraints, while always maintaining the cleanest possible boundaries.

---

LangChain Integration

All splitters have LangChain-compatible wrappers. The core never imports LangChain — wrappers are optional.

import tiktoken
from boundary_smart_splitter.langchain import (
    LangChainWordSplitter,
    LangChainSentenceSplitter,
    LangChainParagraphSplitter,
    LangChainStructureSplitter,
)

splitter = LangChainStructureSplitter(chunk_size=1, max_chars=1500, max_tokens=300)
chunks = splitter.split_text("Your long document here...")

---

API Comparison

Splitter	chunk_size unit	Primary limits	Fallback	Key Params
WordSplitter	words	max_chars, max_tokens	hard cut	tolerance, length_function
SentenceSplitter	sentences	if limits exceeded, fall back to V1	WordSplitter	tolerance, boundary_preference, abbreviations, length_function
ParagraphSplitter	paragraphs	if limits exceeded, fall back to V2	SentenceSplitter	tolerance, overlap, paragraph_separator, use_markdown_mode, length_function
StructureSplitter	sections / topics	if limits exceeded, fall back to V3	ParagraphSplitter	respect_headings, heading_detector, respect_numbered_sections, split_on_transitions, length_function

---

Why boundary-first matters

Size-first splitting (the common approach):

"...the quick brown fox jum" | "ps over the lazy dog..."
# Bad: cuts through a word

Boundary-first splitting (this library):

"...the quick brown fox" | "jumps over the lazy dog..."
# Good: clean word boundary

This becomes critical at sentence level:

"Please visit the U.S.A. for travel. Yes!"
# sentence-aware: kept together
# naive char-cut: "U.S.A." might get split

And at paragraph/section level for RAG:

"1. Home Page\n\nThe homepage..." | "2. About Us\n\nTo build..."
# section-aware: one chunk per page/section
# naive: cuts through the middle of a page description

---

Features

• Boundary-first, size-second: clean boundaries always take priority over exact counts
• Semantic units for chunk_size: words count words, sentences count sentences, paragraphs count paragraphs, sections count sections
• max_chars as universal safety net: protects embedding model limits without breaking the semantic contract
• Graceful fallback chain: V4 → V3 → V2 → V1, each level catches what the level above can't handle
• LangChain-compatible, not dependent: the core never imports LangChain
• Each version is a superset: V2 includes V1, V3 includes V2, V4 includes V3
• No ML dependencies: fast, deterministic, offline-capable
• Abbreviation-aware: configurable abbreviation list prevents false sentence breaks
• Markdown-aware: optional Markdown paragraph/heading detection
• PDF-friendly: handles leading whitespace from PDF text extraction

---

Requirements

• Python >= 3.9

---

Contributing

Pull requests are welcome! For major changes, please open an issue first to discuss what you would like to change.

1. Fork the repository
2. Create your feature branch (git checkout -b feature/AmazingFeature)
3. Commit your changes (git commit -m 'Add some AmazingFeature')
4. Push to the branch (git push origin feature/AmazingFeature)
5. Open a Pull Request

---

License

MIT © 2026