Skip to main content

Chunk-Fcatory Logo

✨Chunk-Factory ✨

PyPI MIT License Documentation GitHub stars

Chunk-Factory is a fast, efficient text chunking library with real-time evaluation.

Instroduction • Installation • Usage • Supported Methods • TODO • References • Citation •

Instroduction

Chunk-Factory is a Python library that offers various text chunking methods, including both traditional approaches and state-of-the-art techniques. It not only provides efficient text chunking but also offers real-time evaluation metrics, allowing immediate assessment of chunking results. These features are crucial for retrieval-augmented tasks, helping to optimize context extraction and utilization in the retrieval process.

With Chunk-Factory, users can easily chunk text and evaluate its effectiveness, making it suitable for a wide range of natural language processing applications, particularly in scenarios that require fine-grained retrieval and document segmentation.

Note: Every time I do RAG, I have to chop up semantically coherent text into chunks and then have no clue whether it’s good or not. I can only guess based on the retrieval results, but can’t tell if it’s the retriever’s fault or the chunking’s fault. This library is here to solve that problem by evaluating the quality of the chunking first. Hopefully, it can help some people out of their misery—so annoying!

Installation

To install Chunk-Factory, simply run:

pip install chunk-factory

Usage

Here's a basic example to get you started:

from chunk_factory import Chunker

text = 'Chunk-Factory is a Python library that offers various text chunking methods, including both traditional approaches and state-of-the-art techniques. It not only provides efficient text chunking but also offers real-time evaluation metrics, allowing immediate assessment of chunking results. These features are crucial for retrieval-augmented tasks, helping to optimize context extraction and utilization in the retrieval process.'
language = 'en'

ck = Chunker(text,language)
text_chunks = ck.basechunk(chunk_size=20,chunk_overlap=5)
for i,chunk in enumerate(text_chunks):
    print(f'Number {i+1}: ', chunk)

Supported Methods

Chunk-Factory provides several chunkers to help you efficiently split your text for RAG tasks or other natural language processing tasks (such as information extraction). Here's a quick overview of the available chunkers:

  • BaseChunker: Splits text based on words or tokens.

  • SegmentChunker: Splits text into chunks based on sentence or paragraph boundaries.

  • DensexChunker: Splits text into propositions.

  • LumberChunker: Splits text based on semantics using LLM.

  • MspChunker: Splits text based on label probabilities from a small model to determine chunking.

  • PPLChunker: Splits text based on perplexity.

TODO

  • Add traditional text chunking methods

    • [✔️] Add segment chunking method

    • Add semantic chunking method

  • Add retrieval evaluation methods

References

  • MoC: Mixtures of Text Chunking Learners for Retrieval-Augmented Generation System [Paper]
  • Dense X Retrieval: What Retrieval Granularity Should We Use? [Paper]
  • LumberChunker: Long-Form Narrative Document Segmentation [Paper]
  • Meta-chunking: Learning efficient text segmentation via logical perception [Paper]

Citation

If you use Chunk-Factory in your research, please cite it as follows:

@misc{chunkfactory2025,
  author = {Jie H},
  title = {Chunk-Factory: A toolkit with a variety of text chunking methods},
  year = {2025},
  publisher = {GitHub},
  journal = {GitHub repository},
  howpublished = {\url{https://github.com/hjandlm/Chunk-Factory}},
}

Metadata

Release files for chunk-factory 0.1.5

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for chunk-factory 0.1.5
File Size Uploaded Provenance Actions
chunk_factory-0.1.5.tar.gz 24.8 kB Without provenance Details for chunk_factory-0.1.5.tar.gz

Release files / chunk_factory-0.1.5.tar.gz

Download URL chunk_factory-0.1.5.tar.gz
Size 24.8 kB
Tags Source
SHA-256 checksum
How to use checksums
551dbf2dd8280ddffe0b6e7819662ebd657a3495b5d68eaa3079122205ae03b0
BLAKE2b-256 checksum
How to use checksums
9aa761f15f1ffa28b757547c265cafd6689518767bf32f19bdc4ddb9a3877948
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via twine/6.1.0 CPython/3.10.16

Security for chunk-factory 0.1.5

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

No provenance found

None of the files in this release were uploaded with provenance metadata.

A note on package security

PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.

To learn more about protecting yourself from malicious actors, visit our package security documentation.

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page