Skip to main content
Pre-release

This release is a pre-release and may not be stable for production use.

Chonkie Logo

🦛 Chonkie ✨

The no-nonsense ultra-light and lightning-fast chunking library that's ready to CHONK your texts!

PyPI version License Documentation Package size codecov Downloads Discord GitHub stars

Tired of making your gazillionth chunker? Sick of the overhead of large libraries? Want to chunk your texts quickly and efficiently? Chonkie the mighty hippo is here to help!

🚀 Feature-rich: All the CHONKs you'd ever need
✨ Easy to use: Install, Import, CHONK
⚡ Fast: CHONK at the speed of light! zooooom
🪶 Light-weight: No bloat, just CHONK
🌏 Wide support: CHONK with your favorite tokenizer, embedding model and APIs!
💬 ️Multilingual: Out-of-the-box support for 5+ language CHONKS (more coming 🔜)
☁️ Cloud-Ready: CHONK locally or in the Chonkie Cloud
🦛 Cute CHONK mascot: psst it's a pygmy hippo btw
❤️ Moto Moto's favorite python library

Chonkie is a chunking library that "just works" ✨

Installation

To install chonkie, run:

pip install chonkie

Chonkie follows the rule of minimum installs. Have a favorite chunker? Read our docs to install only what you need Don't want to think about it? Simply install all (Not recommended for production environments)

pip install chonkie[all]

Usage

Here's a basic example to get you started:

# First import the chunker you want from Chonkie
from chonkie import RecursiveChunker

# Initialize the chunker
chunker = RecursiveChunker()

# Chunk some text
chunks = chunker("Chonkie is the goodest boi! My favorite chunking hippo hehe.")

# Access chunks
for chunk in chunks:
    print(f"Chunk: {chunk.text}")
    print(f"Tokens: {chunk.token_count}")

Check out more usage examples in the docs!

Supported Methods

Chonkie provides several chunkers to help you split your text efficiently for RAG applications. Here's a quick overview of the available chunkers:

  • TokenChunker: Splits text into fixed-size token chunks.
  • SentenceChunker: Splits text into chunks based on sentences.
  • RecursiveChunker: Splits text hierarchically using customizable rules to create semantically meaningful chunks.
  • SemanticChunker: Splits text into chunks based on semantic similarity.
  • SDPMChunker: Splits text using a Semantic Double-Pass Merge approach.
  • LateChunker: Embeds text and then splits it to have better chunk embeddings.

More on these methods and the approaches taken inside the docs

Benchmarks 🏃‍♂️

"I may be smol hippo, but I pack a big punch!" 🦛

Chonkie is not just cute, it's also fast and efficient! Here's how it stacks up against the competition:

Size📦

  • Default Install: 15MB (vs 80-171MB for alternatives)
  • With Semantic: Still 10x lighter than the closest competition!

Speed⚡

  • Token Chunking: 33x faster than the slowest alternative
  • Sentence Chunking: Almost 2x faster than competitors
  • Semantic Chunking: Up to 2.5x faster than others

Check out our detailed benchmarks to see how Chonkie races past the competition! 🏃‍♂️💨

Contributing

Want to help grow Chonkie? Check out CONTRIBUTING.md to get started! Whether you're fixing bugs, adding features, or improving docs, every contribution helps make Chonkie a better CHONK for everyone.

Remember: No contribution is too small for this tiny hippo! 🦛

Acknowledgements

Chonkie would like to CHONK its way through a special thanks to all the users and contributors who have helped make this library what it is today! Your feedback, issue reports, and improvements have helped make Chonkie the CHONKIEST it can be.

And of course, special thanks to Moto Moto for endorsing Chonkie with his famous quote:

"I like them big, I like them chonkie." ~ Moto Moto

Citation

If you use Chonkie in your research, please cite it as follows:

@software{chonkie2025,
  author = {Minhas, Bhavnick AND Nigam, Shreyash},
  title = {Chonkie: A no-nonsense fast, lightweight, and efficient text chunking library},
  year = {2025},
  publisher = {GitHub},
  howpublished = {\url{https://github.com/chonkie-inc/chonkie}},
}

Metadata

Release files for chonkie 1.0.3a1

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for chonkie 1.0.3a1
File Size Uploaded Provenance Actions
chonkie-1.0.3a1.tar.gz 46.5 kB Without provenance Details for chonkie-1.0.3a1.tar.gz

Built distribution (wheel)

Table of built distributions (wheels) for chonkie 1.0.3a1
File Interpreter ABI Platform Actions
chonkie-1.0.3a1-py3-none-any.whl Python 3 none any Details for chonkie-1.0.3a1-py3-none-any.whl

Total release size: 101.7 kB

Release files / chonkie-1.0.3a1.tar.gz

Download URL chonkie-1.0.3a1.tar.gz
Size 46.5 kB
Tags Source
SHA-256 checksum
How to use checksums
e52a2bfdeb435666bc7d65549a0f55aa0d41ab469879986481a873bddffc9e19
BLAKE2b-256 checksum
How to use checksums
8135d3eb5daede7ccb9942e67985388bef05c691ae9440130655ae07dbf0332d
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via twine/6.1.0 CPython/3.9.22

Release files / chonkie-1.0.3a1-py3-none-any.whl

Download URL chonkie-1.0.3a1-py3-none-any.whl
Size 55.2 kB
Tags Python 3
SHA-256 checksum
How to use checksums
a6cdf1b0ad1d85974238bf5ee6c49dbfa0deba3f7fcaa82df74c98dea70202b0
BLAKE2b-256 checksum
How to use checksums
b40c50bdac9b25174fb9328ac46409dd9c28ca13acd1aef77b36def8738b17f0
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via twine/6.1.0 CPython/3.9.22

Security for chonkie 1.0.3a1

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

No provenance found

None of the files in this release were uploaded with provenance metadata.

A note on package security

PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.

To learn more about protecting yourself from malicious actors, visit our package security documentation.

Release history Release notifications | RSS feed

This release
Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page