This smart chunker is a semantic chunker to prepare a long document for retrieval augmented generation (RAG).
Unlike a usual chunker, it does not split the text into identical groups of N tokens. Instead, it uses a cross-encoder to calculate the similarity function between neighboring sentences and divides the text based on the most significant boundaries of semantic transitions, i.e. minima in the above-mentioned similarity function.
The BAAI/bge-reranker-v2-m3, or any other model that supports the AutoModelForSequenceClassification interface, should be used as a cross encoder.
The smart chunker supports Russian and English.
1 maintainer
Metadata
1 maintainer
Release files for smart-chunker 0.0.5
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | |
|---|---|---|---|
| smart_chunker-0.0.5.tar.gz | 17.3 kB | Details |
Built distribution (wheel)
| File | Interpreter | ABI | Platform | Reset |
|---|---|---|---|---|
| smart_chunker-0.0.5-py3-none-any.whl | Python 3 | none | any | Details |
Total release size: 27.1 kB
Release files / smart_chunker-0.0.5.tar.gz
| Download URL | smart_chunker-0.0.5.tar.gz |
|---|---|
| Size | 17.3 kB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
935a62daa60bafd31c3c7496d751f723fbfb289c42ee7aa85355589d14726804
|
|
BLAKE2b-256 checksum How to use checksums |
02c34c1307e29c9ab0d17df8d1e9600381147c138014a5415787f7d26ba0b943
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/6.2.0 CPython/3.10.18
|
Release files / smart_chunker-0.0.5-py3-none-any.whl
| Download URL | smart_chunker-0.0.5-py3-none-any.whl |
|---|---|
| Size | 9.8 kB |
| Tags | Python 3 |
|
SHA-256 checksum How to use checksums |
18dc6cff5f1da0fa13990883ed823a40395faced826db21f9cd2327318d36e1a
|
|
BLAKE2b-256 checksum How to use checksums |
a1029e9b63d8c4c20a8f39aa2c5ddaf2721f4242f95c7de7d1ef7e07bde18f41
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/6.2.0 CPython/3.10.18
|
