Chunkipy

chunkipy is a modular and extensible text chunking library for Python, built for NLP and LLM pipelines.

Why Chunkipy?

• ✅ Lightweight core with optional extras
• ✅ Configurable overlap support via overlap_ratio
• ✅ Composable architecture (chunkers + splitters + size estimators + language detectors)
• ✅ Practical defaults with customizable behavior

Quick Example

from chunkipy import FixedSizeTextChunker

text = "Chunkipy makes text processing modular, flexible, and powerful!"
chunker = FixedSizeTextChunker(chunk_size=20, overlap_ratio=0.2)
chunks = chunker.chunk(text)

for i, c in enumerate(chunks):
    print(f"Chunk {i + 1}: {c}")

Implemented vs Roadmap

Semantic sentence splitters and language detectors are already available and can be used today.

Installation

Install core package:

pip install chunkipy

Install optional feature groups:

pip install "chunkipy[language-detection]"  # Language detection (langdetect + fasttext)
pip install "chunkipy[nlp]"                  # NLP backends (spacy + stanza)
pip install "chunkipy[ai]"                   # LLM integration (openai + tiktoken)
pip install "chunkipy[all]"                  # All optional dependencies

Or install individual packages:

pip install "chunkipy[spacy]"
pip install "chunkipy[stanza]"
pip install "chunkipy[langdetect]"
pip install "chunkipy[fasttext]"
pip install "chunkipy[openai]"
pip install "chunkipy[tiktoken]"

Documentation

Full guides and API reference: 👉 https://gioelecrispo.github.io/chunkipy

Examples: 👉 https://github.com/gioelecrispo/chunkipy/tree/main/examples

Contributing

Issues and pull requests are welcome: 👉 https://github.com/gioelecrispo/chunkipy/issues

For local setup, see CONTRIBUTING.md.

License

chunkipy is released under the MIT License.