chunk-factory 0.1.0

An efficient chunking library that integrates traditional and advanced methods, with real-time evaluation of chunking results.

✨Chunk-Factory ✨

Chunk-Factory is a fast, efficient text chunking library with real-time evaluation.

Instroduction • Installation • Usage • Supported Methods • TODO • References • Citation

Instroduction

Chunk-Factory is a Python library that offers various text chunking methods, including both traditional approaches and state-of-the-art techniques. It not only provides efficient text chunking but also offers real-time evaluation metrics, allowing immediate assessment of chunking results. These features are crucial for retrieval-augmented tasks, helping to optimize context extraction and utilization in the retrieval process.

With Chunk-Factory, users can easily chunk text and evaluate its effectiveness, making it suitable for a wide range of natural language processing applications, particularly in scenarios that require fine-grained retrieval and document segmentation.

Note: Every time I do RAG, I have to chop up semantically coherent text into chunks and then have no clue whether it’s good or not. I can only guess based on the retrieval results, but can’t tell if it’s the retriever’s fault or the chunking’s fault. This library is here to solve that problem by evaluating the quality of the chunking first. Hopefully, it can help some people out of their misery—so annoying!

Installation

To install Chunk-Factory, simply run:

pip install chunk-factory

Usage

Here's a basic example to get you started:

from chunk_factory import Chunker

text = 'Chunk-Factory is a Python library that offers various text chunking methods, including both traditional approaches and state-of-the-art techniques. It not only provides efficient text chunking but also offers real-time evaluation metrics, allowing immediate assessment of chunking results. These features are crucial for retrieval-augmented tasks, helping to optimize context extraction and utilization in the retrieval process.'
language = 'en'

ck = Chunker(text,language)
text_chunks = ck.basechunk(chunk_size=20,chunk_overlap=5)
for i,chunk in enumerate(text_chunks):
    print(f'Number {i+1}: ', chunk)

Supported Methods

Chunk-Factory provides several chunkers to help you efficiently split your text for RAG tasks or other natural language processing tasks (such as information extraction). Here's a quick overview of the available chunkers:

• BaseChunker: Splits text based on words or tokens.

• DensexChunker: Splits text into propositions.

• LumberChunker: Splits text based on semantics using LLM.

• MspChunker: Splits text based on label probabilities from a small model to determine chunking.

• PPLChunker: Splits text based on perplexity.

TODO

• Add traditional text chunking methods

  • Add recursive chunking method

  • Add semantic chunking method

• Add retrieval evaluation methods

References

• MoC: Mixtures of Text Chunking Learners for Retrieval-Augmented Generation System [Paper]
• Dense X Retrieval: What Retrieval Granularity Should We Use? [Paper]
• LumberChunker: Long-Form Narrative Document Segmentation [Paper]
• Meta-chunking: Learning efficient text segmentation via logical perception [Paper]

Citation

If you use Chunk-Factory in your research, please cite it as follows:

@misc{chunkfactory2025,
  author = {Jie H},
  title = {Chunk-Factory: A toolkit with a variety of text chunking methods},
  year = {2025},
  publisher = {GitHub},
  journal = {GitHub repository},
  howpublished = {url{https://github.com/hjandlm/Chunk-Factory}},
}