Skip to main content

Chonkie Logo

🦛 Chonkie ✨

PyPI version License Documentation Package size Downloads GitHub stars

The no-nonsense RAG chunking library that's lightweight, lightning-fast, and ready to CHONK your texts

Installation • Usage • Supported Methods • Benchmarks • Documentation • Citation

so i found myself making another RAG bot (for the 2342148th time) and meanwhile, explaining to my juniors about why we should use chunking in our RAG bots, only to realise that i would have to write chunking all over again unless i use the bloated software library X or the extremely feature-less library Y. WHY CAN I NOT HAVE SOMETHING JUST RIGHT, UGH?

Can't i just install, import and run chunking and not have to worry about dependencies, bloat, speed or other factors?

Well, with chonkie you can! (chonkie boi is a gud boi)

🚀 Feature-rich: All the CHONKs you'd ever need
✨ Easy to use: Install, Import, CHONK
⚡ Fast: CHONK at the speed of light! zooooom
🌐 Wide support: Supports all your favorite tokenizer CHONKS
🪶 Light-weight: No bloat, just CHONK
🦛 Cute CHONK mascot: psst it's a pygmy hippo btw
❤️ Moto Moto's favorite python library

What're you waiting for, just CHONK it!

Installation

To install chonkie, simply run:

pip install chonkie

Chonkie follows the rule to have minimal defualt installs, read the DOCS to know the installation for your required chunker, or simply install all if you don't want to think about it (not recommended).

pip install chonkie[all]

Usage

Here's a basic example to get you started:

# First import the chunker you want from Chonkie 
from chonkie import TokenChunker

# Import your favorite tokenizer library
# Also supports AutoTokenizers, TikToken and AutoTikTokenizer
from tokenizers import Tokenizer 
tokenizer = Tokenizer.from_pretrained("gpt2")

# Initialize the chunker
chunker = TokenChunker(tokenizer)

# Chunk some text
chunks = chunker("Woah! Chonkie, the chunking library is so cool! I love the tiny hippo hehe.")

# Access chunks
for chunk in chunks:
    print(f"Chunk: {chunk.text}")
    print(f"Tokens: {chunk.token_count}")

More example usages given inside the DOCS

Supported Methods

Chonkie provides several chunkers to help you split your text efficiently for RAG applications. Here's a quick overview of the available chunkers:

  • TokenChunker: Splits text into fixed-size token chunks.
  • WordChunker: Splits text into chunks based on words.
  • SentenceChunker: Splits text into chunks based on sentences.
  • SemanticChunker: Splits text into chunks based on semantic similarity.
  • SDPMChunker: Splits text using a Semantic Double-Pass Merge approach.

More on these methods and the approaches taken inside the DOCS

Benchmarks 🏃‍♂️

"I may be smol hippo, but I pack a punch!" 🦛

Here's a quick peek at how Chonkie performs:

Size📦

  • Default Install: 9.7MB (vs 80-171MB for alternatives)
  • With Semantic: Still lighter than the competition!

Speed⚡

  • Token Chunking: 33x faster than the slowest alternative
  • Sentence Chunking: Almost 2x faster than competitors
  • Semantic Chunking: Up to 2.5x faster than others

Check out our detailed benchmarks to see how Chonkie races past the competition! 🏃‍♂️💨

Acknowledgements

Chonkie would like to CHONK its way through a special thanks to all the users and contributors who have helped make this library what it is today! Your feedback, issue reports, and improvements have helped make Chonkie the CHONKIEST it can be.

And of course, special thanks to Moto Moto for endorsing Chonkie with his famous quote:

"I like them big, I like them chonkie." ~ Moto Moto

Citation

If you use Chonkie in your research, please cite it as follows:

@misc{chonkie2024,
  author = {Minhas, Bhavnick},
  title = {Chonkie: A Fast Feature-full Chunking Library for RAG Bots},
  year = {2024},
  publisher = {GitHub},
  journal = {GitHub repository},
  howpublished = {\url{https://github.com/bhavnick/chonkie}},
}

Metadata

Release files for chonkie 0.2.2

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for chonkie 0.2.2
File Size Uploaded Provenance Actions
chonkie-0.2.2.tar.gz 32.7 kB Without provenance Details for chonkie-0.2.2.tar.gz

Built distribution (wheel)

Table of built distributions (wheels) for chonkie 0.2.2
File Interpreter ABI Platform Actions
chonkie-0.2.2-py3-none-any.whl Python 3 none any Details for chonkie-0.2.2-py3-none-any.whl

Total release size: 71.9 kB

Release files / chonkie-0.2.2.tar.gz

Download URL chonkie-0.2.2.tar.gz
Size 32.7 kB
Tags Source
SHA-256 checksum
How to use checksums
7f13341ff7aab3999c60cd984f78bcd9036e5c2264e6c8a2bbf0a0fee94b5217
BLAKE2b-256 checksum
How to use checksums
c41a912b6a1409167b5c9c8254e89ffb03be7c0ab069cb5fc7b7d0658311b50b
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via twine/6.0.1 CPython/3.9.21

Release files / chonkie-0.2.2-py3-none-any.whl

Download URL chonkie-0.2.2-py3-none-any.whl
Size 39.2 kB
Tags Python 3
SHA-256 checksum
How to use checksums
c5be45cad5536a739dd0c80f75844d215d6859e2658c9870cee92f6ad8288370
BLAKE2b-256 checksum
How to use checksums
404db0c94b818d044999f3a21ed494b93be71cb58f87fbbe131a481110e147cd
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via twine/6.0.1 CPython/3.9.21

Security for chonkie 0.2.2

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

No provenance found

None of the files in this release were uploaded with provenance metadata.

A note on package security

PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.

To learn more about protecting yourself from malicious actors, visit our package security documentation.

Release history Release notifications | RSS feed

This release

0.2.2 This release

2 release files No provenance

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page