smart-slice

Fidelity-first document slicing for RAG pipelines. Feed it 197 file extensions across 30 handlers, get back retrieval-ready paragraphs that keep every original character reachable.

from smart_slice import slice_text

paragraphs = slice_text("# Chapter\n\nbody text", limit=1000)
# [{'title': 'Chapter', 'content': 'body text'}]

Why another splitter?

Most text splitters treat a document as a bag of characters and chop it on a fixed grid. That loses the two things retrieval actually depends on: where a passage sits in the document's structure, and the passage's exact original wording. smart-slice is built around three rules.

1. Fidelity before cleverness

No cleaning step deletes source text.

• Heading markers (#) are stripped only at a line start and only outside ``` code fences — a #RRGGBB colour or a Python comment in a code block survives.
• A markdown table cut by the length budget gets its header row appended to the continuation chunk, not rewritten into it. The original rows are untouched.
• An oversized spreadsheet row is re-split, never truncated.

2. Structure before length

limit is a budget, not a grid. The slicer walks the document's own shape first:

1. the markdown heading tree (levels 1–6),
2. then blank-line paragraph breaks,
3. then sentence boundaries (。 . ! ！ ? ？),
4. and only as a last resort, a hard character cut.

Each paragraph carries a title field holding its heading chain ("Part I Chapter 3"), so downstream chunks keep their section context.

3. Measurably fast

Slicing a 550 KB structured document takes ~140 ms (≈3.9 MB/s) on a laptop CPU. The hot path is a single linear scan that decides which heading levels a block can possibly contain, so provably-empty regex passes are skipped; an optional C extension (pip install smart-slice[accel]) runs that scan natively. Numbers and the reasoning are in docs/PERFORMANCE.md.

4. No framework

A pure library: no Django, no ORM, no network calls, no logging configuration (it emits records on the smart_slice logger and leaves routing to you). Errors are SliceError with an HTTP-style .code (400 = input problem, 500 = parse failure). Images pulled out of a document are handed to a save_image callback — the library never persists anything.

Installation

The core install is deliberately tiny (just a text decoder). Format parsers are optional extras, and a handler whose parser is missing simply reports "unsupported" instead of crashing.

pip install smart-slice                 # core: text/markdown/config/source files
pip install smart-slice[all]            # every format below

Targeted extras:

Call smart_slice.missing_dependencies() at runtime to see exactly which extras would unlock more formats.

Quickstart

Slice a string

from smart_slice import slice_text

rows = slice_text("# Guide\n\nIntro text.\n\n## Setup\n\nStep one.", limit=1000)
for r in rows:
    print(r["title"], "->", r["content"][:40])
# Guide -> Intro text.
# Guide  Setup -> Step one.

Slice a file from disk

from smart_slice import slice_path

rows = slice_path("report.pdf", limit=1000)        # name decides the handler

Slice from bytes (with image capture)

from smart_slice import slice_bytes, ImageAsset

images = []
def save_image(assets):           # assets: list[ImageAsset]
    images.extend(assets)

rows = slice_bytes(raw_bytes, "deck.pptx", limit=1000, save_image=save_image)
# each ImageAsset has .id, .file_name, .content (bytes); the paragraph text
# already references it as ![name](./oss/file/{id})

Output shape

Every entry point returns a list of {"title": str, "content": str}:

[
    {"title": "Part I  Chapter 3", "content": "the paragraph body..."},
    ...
]

title is the heading chain (empty when the document has no headings); content is the paragraph's verbatim text.

Supported formats

30 handlers covering 197 declared extensions. Every parser except charset-normalizer (text decoding) is an optional extra: a handler whose parser is missing reports 400 Unsupported file format instead of breaking import smart_slice.

Archives are unpacked recursively; each inner file goes through the same dispatch. The text handler is the fallback for any decodable content whose extension is not otherwise claimed.

Not supported (rejected with 400)

These are deliberately rejected rather than force-decoded into garbage:

Content-level rejections (right extension, unparsable content): encrypted PDFs and image-only/scanned PDFs with no extractable text raise 500 naming the cause; encrypted or oversized archives raise ResourceLimitError. Local OCR is off by default (SMART_SLICE_OCR_ENABLED=0); enable it with the ocr extra to recover text from embedded and scanned images.

Command line

smart-slice slice report.pdf --limit 1000 --format json -o out.json
smart-slice slice notes.md --overlap 150 --stats          # 15% overlap on a 1000-char budget
smart-slice slice notes.md --overlap-ratio 0.15 --format text
smart-slice chunk notes.md --chunk-size 256 --chunk-overlap 40 --carry-title
smart-slice detect mystery.bin
smart-slice formats            # list handlers + missing optional deps
python -m smart_slice slice notes.md --format md   # module form

• slice — --format {json,jsonl,text,md}, --limit, --overlap, --overlap-ratio, --overlap-section-only, --no-filter, --title-prefix, -o/--output, --stats (summary to stderr).
• chunk — slices first, then cuts paragraphs to an embedding window: --chunk-size, --chunk-overlap, --carry-title, -o/--output, --stats. Emits one JSON object per chunk.
• detect — prints the handler class that would claim the file (exit 2 if none).
• formats — supported extensions and which extras are not installed (--json for machines).

Advanced usage

Chunk size and overlap

Both stages default to sensible values and both are fully configurable — pass nothing for defaults, or pass numbers/objects for control.

from smart_slice import slice_bytes, chunk_paragraphs, ChunkingOptions

# defaults: 1000-char paragraphs, no overlap
rows = slice_bytes(data, "doc.md")

# custom: 800-char paragraphs carrying 15% of context forward
rows = slice_bytes(data, "doc.md", limit=800, overlap_ratio=0.15)

# reuse one configuration across many documents
opts = ChunkingOptions(limit=600, overlap=90, carry_title=True)
for name, blob in documents.items():
    rows = slice_bytes(blob, name, options=opts)

# second stage: embedding chunks with their own overlap
chunks = chunk_paragraphs(rows, chunk_size=256, chunk_overlap=40, carry_title=True)

What overlap does to the output. Paragraph count and titles are unchanged; each paragraph simply gains the previous paragraph's tail:

plain    = slice_bytes(data, "doc.md", limit=500)                # 42 paragraphs
overlapped = slice_bytes(data, "doc.md", limit=500, overlap=100) # 42 paragraphs
assert len(plain) == len(overlapped)
assert plain[0]["content"] in overlapped[0]["content"]   # additive, never lossy

With overlap=0 (the default) chunks tile the source exactly once, so concatenating them reproduces the original text. A positive overlap trades that strict tiling property for cross-boundary retrieval recall — the right choice when an answer spans a cut point, the wrong one when you need to reconstruct the document from its chunks.

ChunkingOptions fields:

Token-based budgeting needs no extra plumbing:

import tiktoken
enc = tiktoken.encoding_for_model("gpt-4o-mini")
opts = ChunkingOptions(limit=400, overlap=60, length_fn=lambda s: len(enc.encode(s)))
rows = slice_bytes(data, "doc.md", options=opts)   # 400-token paragraphs

Custom heading patterns

import re
from smart_slice import slice_bytes, MARKDOWN_HEADINGS, BLANK_LINE

# slice on a bespoke "SECTION n:" scheme, then blank lines
patterns = [re.compile(r"(?m)^SECTION \d+:.*"), *BLANK_LINE]
rows = slice_bytes(data, "spec.txt", limit=800, patterns=patterns)

Raw (non-normalised) structure

normalize=False returns the handler's native shape — nested groups for multi-sheet workbooks and archives, which preview UIs want:

raw = slice_bytes(data, "book.xlsx", limit=1000, normalize=False)
# [{"name": "Sheet1", "content": [...]}, {"name": "Sheet2", "content": [...]}]

Image OCR hook

from smart_slice import slice_bytes, build_image_text_extractor

ocr = build_image_text_extractor()   # None unless SMART_SLICE_OCR_ENABLED=1 and `ocr` extra installed
rows = slice_bytes(data, "scan.docx", limit=1000, image_text_extractor=ocr)

Progress heartbeat

progress_hook is a zero-arg callback fired before each handler, each archive member, and each OCR call — wire it to a job heartbeat while parsing large files:

slice_bytes(data, "huge.pdf", limit=1000, progress_hook=lambda: job.touch())

Second-stage chunking for embeddings

Slicing yields semantic paragraphs; embedding models still need fixed-size input. chunk_paragraphs cuts paragraphs to a window while the paragraph stays the retrieval unit. See Chunk size and overlap for the overlap and carry_title options.

Error handling

from smart_slice import slice_bytes, SliceError

try:
    slice_bytes(data, "movie.mp4")
except SliceError as e:
    print(e.code, e.message)   # 400 Unsupported file format: .mp4

Configuration

Defensive parser caps (anti decompression-bomb guards) come from the environment, canonical name SMART_SLICE_PARSER_<FIELD>:

Or override per call by passing a ParserLimits instance to the validators.

API reference

Building blocks: SplitModel, smart_split_paragraph, filter_special_char, MarkChunkHandle, ParserLimits, ImageAsset, SPLIT_HANDLERS, MARKDOWN_HEADINGS, DEFAULT_PATTERNS, BLANK_LINE, LITERAL_PATTERNS.

See docs/ARCHITECTURE.md for how slicing works internally and docs/PORTING.md for how this was extracted from the source platform.

Development

git clone https://github.com/guangxiangdebizi/smart-slice.git
cd smart-slice
pip install -e ".[dev]"
pytest
ruff check smart_slice tests

License

GPL-3.0. See LICENSE.