smart-chunker 0.0.4

Smart-Chunker is a semantic chunker to prepare a long document for RAG

This smart chunker is a semantic chunker to prepare a long document for retrieval augmented generation (RAG).

Unlike a usual chunker, it does not split the text into identical groups of N tokens. Instead, it uses a cross-encoder to calculate the similarity function between neighboring sentences and divides the text based on the most significant boundaries of semantic transitions, i.e. minima in the above-mentioned similarity function.

The BAAI/bge-reranker-v2-m3, or any other model that supports the AutoModelForSequenceClassification interface, should be used as a cross encoder.

The smart chunker supports Russian and English.