tinychunk 0.1.0

A tiny, zero-dependency text chunking library for Python.

tinychunk

A tiny, zero-dependency text chunking library for Python.

Split text into manageable pieces by characters, words, sentences, paragraphs, or lines — with optional overlap, boundary-aware modes, and metadata. Pure Python standard library, nothing else.

Why tinychunk?

Most chunking libraries pull in additional dependencies such as tokenizers, embedding models, or larger framework toolkits. tinychunk focuses on a single task: splitting text into chunks. It is implemented using only the Python standard library and keeps the API intentionally small and predictable.

Use it when you need simple, predictable text segmentation without adding runtime dependencies.

Installation

pip install tinychunk

Requires Python 3.10 or newer. No runtime dependencies.

Quick Start

from tinychunk import chunk

with open("article.txt", encoding="utf-8") as f:
    text = f.read()

chunks = chunk(text, size=500, mode="chars")

for c in chunks:
    print(c)

Features

• Five chunking modes: chars, words, sentences, paragraphs, lines
• Configurable chunk size and overlap
• Boundary-aware chunking for words, sentences, paragraphs, and lines
• Optional metadata: chunk index, start position, and end position
• Iterator mode to process chunks one by one without building the final result list
• Custom literal separators for domain-specific splitting
• Zero runtime dependencies
• Fully type-hinted public API
• MIT licensed

Quality

• Extensive pytest test suite covering normal usage, edge cases, Unicode, and error handling
• Type-hinted public API, passes mypy in strict mode and ruff with extended rules
• Small, dependency-free implementation
• Public API kept intentionally minimal

Usage

Basic chunking by characters

from tinychunk import chunk

text = "The quick brown fox jumps over the lazy dog. " * 50
chunks = chunk(text, size=200, mode="chars")

In chars mode, the text is split by character count. This mode may split inside words because it works at character level.

Chunking by words

from tinychunk import chunk

text = "The quick brown fox jumps over the lazy dog"
chunks = chunk(text, size=3, mode="words")

print(chunks)
# ['The quick brown', 'fox jumps over', 'the lazy dog']

Word mode keeps words intact and preserves the original whitespace between words inside each chunk.

Chunking with overlap

Overlap is useful when consecutive chunks should share some context.

chunks = chunk(text, size=3, overlap=1, mode="words")

print(chunks)
# ['The quick brown', 'brown fox jumps', 'jumps over the', 'the lazy dog']

The overlap value is measured in the unit of the selected mode:

• characters for mode="chars"
• words for mode="words"
• sentences for mode="sentences"
• paragraphs for mode="paragraphs"
• lines for mode="lines"

Chunking by sentences

from tinychunk import chunk

text = "First sentence. Second sentence. Third sentence. Fourth sentence."
chunks = chunk(text, size=2, mode="sentences")

print(chunks)
# ['First sentence. Second sentence.', 'Third sentence. Fourth sentence.']

Sentence splitting is simple and rule-based. It detects sentence boundaries after ., !, or ? followed by whitespace. It does not try to handle all linguistic edge cases such as abbreviations.

Chunking by paragraphs

from tinychunk import chunk

text = "First paragraph.\n\nSecond paragraph.\n\nThird paragraph."
chunks = chunk(text, size=1, mode="paragraphs")

print(chunks)
# ['First paragraph.', 'Second paragraph.', 'Third paragraph.']

Paragraphs are separated by blank lines.

Chunking by lines

from tinychunk import chunk

text = "line 1\nline 2\nline 3"
chunks = chunk(text, size=2, mode="lines")

print(chunks)
# ['line 1\nline 2', 'line 3']

Lines are split on newline characters.

Getting metadata

When you need to know where each chunk came from in the original text, use with_metadata=True.

from tinychunk import chunk

text = "The quick brown fox"
chunks = chunk(text, size=2, mode="words", with_metadata=True)

for c in chunks:
    print(f"Chunk {c.index}: chars {c.start}-{c.end}, text={c.text!r}")

This returns Chunk objects instead of plain strings.

Each Chunk contains:

• text: the chunk content
• index: zero-based chunk index
• start: start position in the original text
• end: end position in the original text

Iterator mode

Use chunk_iter() when you want to iterate over chunks one by one instead of building the final list of chunks.

from tinychunk import chunk_iter

text = "The quick brown fox jumps over the lazy dog"

for c in chunk_iter(text, size=3, mode="words"):
    print(c)

Note: tinychunk operates on strings. The input text is still held in memory. chunk_iter() avoids building the final list of chunks, but it is not a full streaming parser for files.

Custom separators

For domain-specific splitting, such as code, logs, or structured documents, you can provide custom literal separators.

from tinychunk import chunk

source_code = """import x

def foo():
    pass

def bar():
    pass
"""

chunks = chunk(source_code, size=1, separators=["\n\ndef "])

When separators is provided, it overrides mode.

Separators are matched as literal strings, not as regular expressions. This keeps the behavior predictable and safe for use with untrusted input.

If multiple separators are provided, the first separator that appears in the text is used.

separators = ["\n\nclass ", "\n\ndef ", "\n\n"]
chunks = chunk(source_code, size=1000, separators=separators)

API Reference

chunk(text, size, *, mode="chars", overlap=0, with_metadata=False, separators=None)

Splits a string into chunks and returns a list.

Parameters:

• text (str): The input text to split.
• size (int): Target size per chunk. The unit depends on mode.
• mode (str): One of "chars", "words", "sentences", "paragraphs", "lines". Default: "chars".
• overlap (int): Number of units shared between consecutive chunks. Must be greater than or equal to 0 and smaller than size. Default: 0.
• with_metadata (bool): If True, returns Chunk objects instead of strings. Default: False.
• separators (list[str] | tuple[str, ...] | None): Custom literal separators. Overrides mode when provided.

Returns:

• list[str] if with_metadata=False
• list[Chunk] if with_metadata=True

Raises:

• TypeError: If a parameter has the wrong type.
• ValueError: If a parameter has an invalid value.

chunk_iter(text, size, *, mode="chars", overlap=0, with_metadata=False, separators=None)

Same parameters as chunk(), but returns an iterator instead of a list.

Use this when you want to consume chunks one at a time.

Chunk

An immutable dataclass with positional metadata.

Fields:

• text (str): The chunk content.
• index (int): Zero-based position in the chunk sequence.
• start (int): Start position in the original text.
• end (int): End position in the original text.

Example:

from tinychunk import chunk

chunks = chunk("hello world", size=5, mode="chars", with_metadata=True)

first = chunks[0]
print(first.text)
print(first.index)
print(first.start)
print(first.end)

Behavior Notes

Character mode

mode="chars" splits by character count. It can split inside words.

chunk("hello world", size=5, mode="chars")
# ['hello', ' worl', 'd']

Word mode

mode="words" splits on whitespace and keeps words intact.

chunk("a b c d e", size=2, mode="words")
# ['a b', 'c d', 'e']

Sentence mode

mode="sentences" uses a simple punctuation-based rule. It is intentionally small and dependency-free, not a full natural-language sentence tokenizer.

Empty input

Empty input returns an empty list.

chunk("", size=5)
# []

Requirements

• Python 3.10 or newer
• No runtime dependencies

License

MIT — see LICENSE.