✨Chunk-Factory ✨
Chunk-Factory is a fast, efficient text chunking library with real-time evaluation.
Instroduction • Installation • Usage • Supported Methods • TODO • References • Citation •
Instroduction
Chunk-Factory is a Python library that offers various text chunking methods, including both traditional approaches and state-of-the-art techniques. It not only provides efficient text chunking but also offers real-time evaluation metrics, allowing immediate assessment of chunking results. These features are crucial for retrieval-augmented tasks, helping to optimize context extraction and utilization in the retrieval process.
With Chunk-Factory, users can easily chunk text and evaluate its effectiveness, making it suitable for a wide range of natural language processing applications, particularly in scenarios that require fine-grained retrieval and document segmentation.
Note: Every time I do RAG, I have to chop up semantically coherent text into chunks and then have no clue whether it’s good or not. I can only guess based on the retrieval results, but can’t tell if it’s the retriever’s fault or the chunking’s fault. This library is here to solve that problem by evaluating the quality of the chunking first. Hopefully, it can help some people out of their misery—so annoying!
Installation
To install Chunk-Factory, simply run:
pip install chunk-factory
Usage
Here's a basic example to get you started:
from chunk_factory import Chunker
text = 'Chunk-Factory is a Python library that offers various text chunking methods, including both traditional approaches and state-of-the-art techniques. It not only provides efficient text chunking but also offers real-time evaluation metrics, allowing immediate assessment of chunking results. These features are crucial for retrieval-augmented tasks, helping to optimize context extraction and utilization in the retrieval process.'
language = 'en'
ck = Chunker(text,language)
text_chunks = ck.basechunk(chunk_size=20,chunk_overlap=5)
for i,chunk in enumerate(text_chunks):
print(f'Number {i+1}: ', chunk)
Supported Methods
Chunk-Factory provides several chunkers to help you efficiently split your text for RAG tasks or other natural language processing tasks (such as information extraction). Here's a quick overview of the available chunkers:
-
BaseChunker: Splits text based on words or tokens.
-
SegmentChunker: Splits text into chunks based on sentence or paragraph boundaries.
-
DensexChunker: Splits text into propositions.
-
LumberChunker: Splits text based on semantics using LLM.
-
MspChunker: Splits text based on label probabilities from a small model to determine chunking.
-
PPLChunker: Splits text based on perplexity.
TODO
-
Add traditional text chunking methods
-
[✔️] Add segment chunking method
-
Add semantic chunking method
-
-
Add retrieval evaluation methods
References
- MoC: Mixtures of Text Chunking Learners for Retrieval-Augmented Generation System [Paper]
- Dense X Retrieval: What Retrieval Granularity Should We Use? [Paper]
- LumberChunker: Long-Form Narrative Document Segmentation [Paper]
- Meta-chunking: Learning efficient text segmentation via logical perception [Paper]
Citation
If you use Chunk-Factory in your research, please cite it as follows:
@misc{chunkfactory2025,
author = {Jie H},
title = {Chunk-Factory: A toolkit with a variety of text chunking methods},
year = {2025},
publisher = {GitHub},
journal = {GitHub repository},
howpublished = {\url{https://github.com/hjandlm/Chunk-Factory}},
}
Metadata
Release files for chunk-factory 0.1.5
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | Provenance | Actions |
|---|---|---|---|---|
| chunk_factory-0.1.5.tar.gz | 24.8 kB | Without provenance | Details for chunk_factory-0.1.5.tar.gz |
Release files / chunk_factory-0.1.5.tar.gz
| Download URL | chunk_factory-0.1.5.tar.gz |
|---|---|
| Size | 24.8 kB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
551dbf2dd8280ddffe0b6e7819662ebd657a3495b5d68eaa3079122205ae03b0
|
|
BLAKE2b-256 checksum How to use checksums |
9aa761f15f1ffa28b757547c265cafd6689518767bf32f19bdc4ddb9a3877948
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/6.1.0 CPython/3.10.16
|
Security for chunk-factory 0.1.5
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
No provenance found
None of the files in this release were uploaded with provenance metadata.
A note on package security
PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.
To learn more about protecting yourself from malicious actors, visit our package security documentation.