Skip to main content

Chonkie Logo

🦛 Chonkie ✨

PyPI version License Documentation Package size codecov Downloads Discord GitHub stars

The no-nonsense ultra-light and lightning-fast chunking library that's ready to CHONK your texts!

Installation • Usage • Pipeline • Chunkers • Integrations • Benchmarks

Tired of making your gazillionth chunker? Sick of the overhead of large libraries? Want to chunk your texts quickly and efficiently? Chonkie the mighty hippo is here to help!

🚀 Feature-rich: All the CHONKs you'd ever need
✨ Easy to use: Install, Import, CHONK
⚡ Fast: CHONK at the speed of light! zooooom
🪶 Light-weight: No bloat, just CHONK
🌏 Wide support: CHONKie integrates with your favorite tokenizer, embedding model and APIs!
💬 ️Multilingual: Out-of-the-box support for 56 languages
☁️ Cloud-Ready: CHONK locally or in the Chonkie Cloud
🦛 Cute CHONK mascot: psst it's a pygmy hippo btw
❤️ Moto Moto's favorite python library

Chonkie is a chunking library that "just works" ✨

Installation

To install chonkie, run:

pip install chonkie

Chonkie follows the rule of minimum installs. Have a favorite chunker? Read our docs to install only what you need Don't want to think about it? Simply install all (Not recommended for production environments)

pip install chonkie[all]

Basic Usage

Here's a basic example to get you started:

# First import the chunker you want from Chonkie
from chonkie import RecursiveChunker

# Initialize the chunker
chunker = RecursiveChunker()

# Chunk some text
chunks = chunker("Chonkie is the goodest boi! My favorite chunking hippo hehe.")

# Access chunks
for chunk in chunks:
    print(f"Chunk: {chunk.text}")
    print(f"Tokens: {chunk.token_count}")

Check out more usage examples in the docs!

The Chonkie Pipeline

Chonkie processes text using a pipeline approach to transform raw documents into refined, usable chunks. This allows for flexibility and efficiency in handling different chunking strategies. We call this pipeline CHOMP (short for 'CHOnkie's Multi-step Pipeline').

Here's a conceptual overview of the pipeline, as illustrated in the diagram:

🤖 CHOMP pipeline diagram

The main stages are:

  1. 📄 Document: The starting point – your input text data. It can be in any format!
  2. 👨‍🍳 Chef: This stage handles initial text preprocessing. It might involve cleaning, normalization, or other preparatory steps to get the text ready for chunking. While this is optional, it is recommended to use the Chef stage to clean your text before chunking.
  3. 🦛 Chunker: The core component you select (e.g., RecursiveChunker, SentenceChunker). It applies its specific logic to split the preprocessed text into initial chunks based on the chosen strategy and parameters.
  4. 🏭 Refinery: After initial chunking, the Refinery performs post-processing. This can include merging small chunks based on overlap, adding embeddings, or adding additional context to the chunks. It helps ensure the quality and consistency of the output. You can have multiple Refineries to apply different post-processing steps.
  5. 🤗 Friends: The pipeline's produces the final results which can be either exported to be saved or ingested into your vector database. Chonkie offers Porters to export the chunks and Handshakes to ingest the chunks into your vector database.
    • 🐴 Porters: Porters can save the chunks to a file or a database. Currently, only JSON is supported for exporting the chunks.
    • 🤝 Handshakes: Handshakes provide a unified interface for ingesting the chunks into your preferred vector databases.

This modular pipeline allows Chonkie to be both powerful and easy to configure for various text chunking needs.

Chunkers

Chonkie provides several chunkers to help you split your text efficiently for RAG applications. Here's a quick overview of the available chunkers:

Name Alias Description
TokenChunker token Splits text into fixed-size token chunks.
SentenceChunker sentence Splits text into chunks based on sentences.
RecursiveChunker recursive Splits text hierarchically using customizable rules to create semantically meaningful chunks.
SemanticChunker semantic Splits text into chunks based on semantic similarity. Inspired by the work of Greg Kamradt.
SDPMChunker sdpm Splits text using a Semantic Double-Pass Merge approach.
LateChunker late Embeds text and then splits it to have better chunk embeddings.
CodeChunker code Splits code into structurally meaningful chunks.
NeuralChunker neural Splits text using a neural model.
SlumberChunker slumber Splits text using an LLM to find semantically meaningful chunks. Also known as "AgenticChunker".

More on these methods and the approaches taken inside the docs

Integrations

Chonkie boasts 19+ integrations across tokenizers, embedding providers, LLMs, porters, and vector databases, ensuring it fits seamlessly into your existing workflow.

🪓 Slice 'n' Dice! Chonkie supports 5+ ways to tokenize!

Choose from supported tokenizers or provide your own custom token counting function. Flexibility first!

Name Description Optional Install
character Basic character-level tokenizer. Default tokenizer. default
word Basic word-level tokenizer. default
tokenizers Load any tokenizer from the Hugging Face tokenizers library. default
tiktoken Use OpenAI's tiktoken library (e.g., for gpt-4). chonkie[tiktoken]
transformers Load tokenizers via AutoTokenizer from HF transformers. chonkie[transformers]

default indicates that the feature is available with the default pip install chonkie.

To use a custom token counter, you can pass in any function that takes a string and returns an integer! Something like this:

def custom_token_counter(text: str) -> int:
    return len(text)

chunker = RecursiveChunker(tokenizer_or_token_counter=custom_token_counter)

You can use this to extend Chonkie to support any tokenization scheme you want!

🧠 Embed like a boss! Chonkie links up with 7+ embedding pals!

Seamlessly works with various embedding model providers. Bring your favorite embeddings to the CHONK party! Use AutoEmbeddings to load models easily.

Provider / Alias Class Description Optional Install
model2vec Model2VecEmbeddings Use Model2Vec models. chonkie[model2vec]
sentence-transformers SentenceTransformerEmbeddings Use any sentence-transformers model. chonkie[st]
openai OpenAIEmbeddings Use OpenAI's embedding API. chonkie[openai]
cohere CohereEmbeddings Use Cohere's embedding API. chonkie[cohere]
gemini GeminiEmbeddings Use Google's Gemini embedding API. chonkie[gemini]
jina JinaEmbeddings Use Jina AI's embedding API. chonkie[jina]
voyageai VoyageAIEmbeddings Use Voyage AI's embedding API. chonkie[voyageai]
🧞‍♂️ Power Up with Genies! Chonkie supports 2+ LLM providers!

Genies provide interfaces to interact with Large Language Models (LLMs) for advanced chunking strategies or other tasks within the pipeline.

Genie Name Class Description Optional Install
gemini GeminiGenie Interact with Google Gemini APIs. chonkie[gemini]
openai OpenAIGenie Interact with OpenAI APIs. chonkie[openai]

You can also use the OpenAIGenie to interact with any LLM provider that supports the OpenAI API format, by simply changing the model, base_url, and api_key parameters. For example, here's how to use the OpenAIGenie to interact with the Llama-4-Maverick model via OpenRouter:

from chonkie import OpenAIGenie

genie = OpenAIGenie(model="meta-llama/llama-4-maverick",
                    base_url="https://openrouter.ai/api/v1",
                    api_key="your_api_key")
🐴 Exporting CHONKs! Chonkie supports 1+ Porter!

Porters help you save your chunks easily.

Porter Name Class Description Optional Install
json JSONPorter Export chunks to a JSON file. default
🤝 Shake hands with your DB! Chonkie connects with 4+ vector stores!

Handshakes provide a unified interface to ingest chunks directly into your favorite vector databases.

Handshake Name Class Description Optional Install
chroma ChromaHandshake Ingest chunks into ChromaDB. chonkie[chroma]
qdrant QdrantHandshake Ingest chunks into Qdrant. chonkie[qdrant]
pgvector PgvectorHandshake Ingest chunks into PostgreSQL with pgvector. chonkie[pgvector]
turbopuffer TurbopufferHandshake Ingest chunks into Turbopuffer. chonkie[turbopuffer]

With Chonkie's wide range of integrations, you can easily plug it into your existing infrastructure and start CHONKING!

Benchmarks

"I may be smol hippo, but I pack a big punch!" 🦛

Chonkie is not just cute, it's also fast and efficient! Here's how it stacks up against the competition:

Size📦

  • Default Install: 15MB (vs 80-171MB for alternatives)
  • With Semantic: Still 10x lighter than the closest competition!

Speed⚡

  • Token Chunking: 33x faster than the slowest alternative
  • Sentence Chunking: Almost 2x faster than competitors
  • Semantic Chunking: Up to 2.5x faster than others

Check out our detailed benchmarks to see how Chonkie races past the competition! 🏃‍♂️💨

Contributing

Want to help grow Chonkie? Check out CONTRIBUTING.md to get started! Whether you're fixing bugs, adding features, or improving docs, every contribution helps make Chonkie a better CHONK for everyone.

Remember: No contribution is too small for this tiny hippo! 🦛

Acknowledgements

Chonkie would like to CHONK its way through a special thanks to all the users and contributors who have helped make this library what it is today! Your feedback, issue reports, and improvements have helped make Chonkie the CHONKIEST it can be.

And of course, special thanks to Moto Moto for endorsing Chonkie with his famous quote:

"I like them big, I like them chonkie." ~ Moto Moto

Citation

If you use Chonkie in your research, please cite it as follows:

@software{chonkie2025,
  author = {Minhas, Bhavnick AND Nigam, Shreyash},
  title = {Chonkie: A no-nonsense fast, lightweight, and efficient text chunking library},
  year = {2025},
  publisher = {GitHub},
  howpublished = {\url{https://github.com/chonkie-inc/chonkie}},
}

Metadata

Release files for chonkie 1.1.0

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for chonkie 1.1.0
File Size Uploaded Provenance Actions
chonkie-1.1.0.tar.gz 265.7 kB Without provenance Details for chonkie-1.1.0.tar.gz

Built distributions (wheels)

Table of built distributions (wheels) for chonkie 1.1.0
File Actions
chonkie-1.1.0-cp313-cp313-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl CPython 3.13 CPython 3.13 Linux glibc 2.28+ x86-64, Linux glibc 2.17+ x86-64 Details for chonkie-1.1.0-cp313-cp313-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl
chonkie-1.1.0-cp313-cp313-macosx_10_13_x86_64.whl CPython 3.13 CPython 3.13 macOS 10.13+ x86-64 Details for chonkie-1.1.0-cp313-cp313-macosx_10_13_x86_64.whl
chonkie-1.1.0-cp312-cp312-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl CPython 3.12 CPython 3.12 Linux glibc 2.17+ x86-64, Linux glibc 2.28+ x86-64 Details for chonkie-1.1.0-cp312-cp312-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl
chonkie-1.1.0-cp312-cp312-macosx_10_13_x86_64.whl CPython 3.12 CPython 3.12 macOS 10.13+ x86-64 Details for chonkie-1.1.0-cp312-cp312-macosx_10_13_x86_64.whl
chonkie-1.1.0-cp311-cp311-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl CPython 3.11 CPython 3.11 Linux glibc 2.17+ x86-64, Linux glibc 2.28+ x86-64 Details for chonkie-1.1.0-cp311-cp311-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl
chonkie-1.1.0-cp311-cp311-macosx_10_9_x86_64.whl CPython 3.11 CPython 3.11 macOS 10.9+ x86-64 Details for chonkie-1.1.0-cp311-cp311-macosx_10_9_x86_64.whl
chonkie-1.1.0-cp310-cp310-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl CPython 3.10 CPython 3.10 Linux glibc 2.17+ x86-64, Linux glibc 2.28+ x86-64 Details for chonkie-1.1.0-cp310-cp310-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl
chonkie-1.1.0-cp310-cp310-macosx_10_9_x86_64.whl CPython 3.10 CPython 3.10 macOS 10.9+ x86-64 Details for chonkie-1.1.0-cp310-cp310-macosx_10_9_x86_64.whl
chonkie-1.1.0-cp39-cp39-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl CPython 3.9 CPython 3.9 Linux glibc 2.17+ x86-64, Linux glibc 2.28+ x86-64 Details for chonkie-1.1.0-cp39-cp39-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl
chonkie-1.1.0-cp39-cp39-macosx_10_9_x86_64.whl CPython 3.9 CPython 3.9 macOS 10.9+ x86-64 Details for chonkie-1.1.0-cp39-cp39-macosx_10_9_x86_64.whl

Total release size: 5.2 MB

Release files / chonkie-1.1.0.tar.gz

Download URL chonkie-1.1.0.tar.gz
Size 265.7 kB
Tags Source
SHA-256 checksum
How to use checksums
a3c145f7b94820fd3bee366714996abb0900a2302baa89ec7f948173c6ba4257
BLAKE2b-256 checksum
How to use checksums
84fd0f4a795c347b3a3b18b87fbe40096d3617188e4761e65df55bf60e0e198b
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via twine/6.1.0 CPython/3.9.23

Release files / chonkie-1.1.0-cp313-cp313-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl

Download URL chonkie-1.1.0-cp313-cp313-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl
Size 650.4 kB
Tags CPython 3.13 Linux glibc 2.17+ x86-64 Linux glibc 2.28+ x86-64
SHA-256 checksum
How to use checksums
f3dba3bbb3fefdf093f4c6d7d8a1dfdc85cd6d709de2dea1fed209808e7fbda2
BLAKE2b-256 checksum
How to use checksums
f27600484b27cb9ef3fbc2f078c65326656c7c27cfa98d4d4a59d371f51608f0
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via twine/6.1.0 CPython/3.9.23

Release files / chonkie-1.1.0-cp313-cp313-macosx_10_13_x86_64.whl

Download URL chonkie-1.1.0-cp313-cp313-macosx_10_13_x86_64.whl
Size 348.6 kB
Tags CPython 3.13 macOS 10.13+ x86-64
SHA-256 checksum
How to use checksums
9ba0cc92587546591e4f80ecdeaa1141f943b644b47e1979ce53d3ad039f4f11
BLAKE2b-256 checksum
How to use checksums
54af8d963794650969569857969f6afbe04ace9b81780268d8b81dd55c66536a
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via twine/6.1.0 CPython/3.9.23

Release files / chonkie-1.1.0-cp312-cp312-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl

Download URL chonkie-1.1.0-cp312-cp312-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl
Size 656.6 kB
Tags CPython 3.12 Linux glibc 2.17+ x86-64 Linux glibc 2.28+ x86-64
SHA-256 checksum
How to use checksums
dda97475be316bf119e665e2990b8fbcbae69220c92f495c3ac395c1bc81d482
BLAKE2b-256 checksum
How to use checksums
039557b8891ea5d31a4f226ddc7cf3ce47bfac8d2750cc94e362348a90f5a4a4
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via twine/6.1.0 CPython/3.9.23

Release files / chonkie-1.1.0-cp312-cp312-macosx_10_13_x86_64.whl

Download URL chonkie-1.1.0-cp312-cp312-macosx_10_13_x86_64.whl
Size 349.6 kB
Tags CPython 3.12 macOS 10.13+ x86-64
SHA-256 checksum
How to use checksums
cd7beb1013b9e93fab78d450972e46fec9781bf8330757da51f138bb89c23f14
BLAKE2b-256 checksum
How to use checksums
1db6631a2d48d76eb8ddebaaa9fef244d2b48233738283cbf1f9b02b2b01021a
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via twine/6.1.0 CPython/3.9.23

Release files / chonkie-1.1.0-cp311-cp311-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl

Download URL chonkie-1.1.0-cp311-cp311-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl
Size 631.9 kB
Tags CPython 3.11 Linux glibc 2.17+ x86-64 Linux glibc 2.28+ x86-64
SHA-256 checksum
How to use checksums
ab542a485d7caef6a452bb8df9033a0e83e5710a631a5f49606b805487f5465a
BLAKE2b-256 checksum
How to use checksums
2a41807dfd2bca810437d4a81c5abc64d762b38c3593a5ad9556cfa689ac420f
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via twine/6.1.0 CPython/3.9.23

Release files / chonkie-1.1.0-cp311-cp311-macosx_10_9_x86_64.whl

Download URL chonkie-1.1.0-cp311-cp311-macosx_10_9_x86_64.whl
Size 349.6 kB
Tags CPython 3.11 macOS 10.9+ x86-64
SHA-256 checksum
How to use checksums
31307dbfafb8cbfb807535500eed7e4849d2d391d33df53e0f03ebaeb220a021
BLAKE2b-256 checksum
How to use checksums
8e68f9ead21553b084d13ffa707a231ec6c653a01a0fe38972ca2fc32c219573
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via twine/6.1.0 CPython/3.9.23

Release files / chonkie-1.1.0-cp310-cp310-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl

Download URL chonkie-1.1.0-cp310-cp310-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl
Size 610.5 kB
Tags CPython 3.10 Linux glibc 2.17+ x86-64 Linux glibc 2.28+ x86-64
SHA-256 checksum
How to use checksums
75138736da5bafbee8c47b5c538904e928ef217254c635b561ca9740472ede18
BLAKE2b-256 checksum
How to use checksums
4a035d829f52912d94ab70a2511647348bbe4466e5a550462d6271fdfef728ea
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via twine/6.1.0 CPython/3.9.23

Release files / chonkie-1.1.0-cp310-cp310-macosx_10_9_x86_64.whl

Download URL chonkie-1.1.0-cp310-cp310-macosx_10_9_x86_64.whl
Size 348.8 kB
Tags CPython 3.10 macOS 10.9+ x86-64
SHA-256 checksum
How to use checksums
8de0021904b952f879a35b6d1d498b1a260dbac3c68b4b445dc4bfebee7ed975
BLAKE2b-256 checksum
How to use checksums
ed637cb683051f4cb26cc7714249cb760d5aa7b00228555d4f559d5774f6fd1c
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via twine/6.1.0 CPython/3.9.23

Release files / chonkie-1.1.0-cp39-cp39-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl

Download URL chonkie-1.1.0-cp39-cp39-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl
Size 616.3 kB
Tags CPython 3.9 Linux glibc 2.17+ x86-64 Linux glibc 2.28+ x86-64
SHA-256 checksum
How to use checksums
802732c16c308145334cd6808f8b562d98b1fb219bd2e94420343242260b0ce7
BLAKE2b-256 checksum
How to use checksums
a7b4ab590de3cace1c653b51d13fe26c2e2d9ee7d4b08b12120745803b8b14a8
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via twine/6.1.0 CPython/3.9.23

Release files / chonkie-1.1.0-cp39-cp39-macosx_10_9_x86_64.whl

Download URL chonkie-1.1.0-cp39-cp39-macosx_10_9_x86_64.whl
Size 349.3 kB
Tags CPython 3.9 macOS 10.9+ x86-64
SHA-256 checksum
How to use checksums
241cf2f634325b61661414348e9cc59741fae09fd3c923bf9a6667e3f87c995b
BLAKE2b-256 checksum
How to use checksums
931907b07f8bb602b888c82bc79be04f7776a2c8b83a38a21e64c397fe1fad20
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via twine/6.1.0 CPython/3.9.23

Security for chonkie 1.1.0

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

No provenance found

None of the files in this release were uploaded with provenance metadata.

A note on package security

PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.

To learn more about protecting yourself from malicious actors, visit our package security documentation.

Release history Release notifications | RSS feed

This release

1.1.0 This release

11 release files No provenance

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page