Chunking is a crucial step for RAG systems and LLM workflows. Most of the time, chunk quality is rarely evaluated even though it can have a significant impact on the performance of the system. Judie makes it super easy to evaluate your chunks!
📦 Installation
Installation is super easy! Just run the following command in your terminal:
pip install judie
🧑⚖️ Usage
Judie works together with chonkie to evaluate your chunks. It supports all the chunkers that chonkie supports, and as long as any chunker can be wrapped in a chonkie.BaseChunker wrapper, Judie will support it.
Here's a simple example of how to use Judie:
from judie import Evaluator
from chonkie import RecursiveChunker
chunker = RecursiveChunker(chunk_size=1000, chunk_overlap=200)
# Load your dataset
dataset = ["...", "..."]
# Initialize the evaluator
evaluator = Evaluator(chunker=chunker,
dataset=dataset,
embedding_model=embedding_model)
# Evaluate your chunks
metrics = evaluator.evaluate()
# Print the metrics
print(metrics['Recall@10'])
🧩 Available Benchmarks
| Benchmark | Description |
|---|---|
| 🧸 Gacha | 🧸 Gacha is a corpus of 100 most popular textbooks from Gutenberg Corpus and numerous NIAH-style questions for evaluating chunking algorithms! |
📚 Citation
If you use Judie in your work, please cite it as follows:
@software{judie2025,
author = {Bhavnick Minhas and Shreyash Nigam},
title = {🦉 Judie: Evaluating chunk quality with LLM Judges},
url = {https://github.com/chonkie-inc/judie},
version = {0.1.0},
year = {2025},
}
Metadata
Release files for judie 0.0.2
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | Provenance | Actions |
|---|---|---|---|---|
| judie-0.0.2.tar.gz | 17.4 kB | Without provenance | Details for judie-0.0.2.tar.gz |
Built distribution (wheel)
| File | Interpreter | ABI | Platform | Actions Reset |
|---|---|---|---|---|
| judie-0.0.2-py3-none-any.whl | Python 3 | none | any | Details for judie-0.0.2-py3-none-any.whl |
Total release size: 36.3 kB
Release files / judie-0.0.2.tar.gz
| Download URL | judie-0.0.2.tar.gz |
|---|---|
| Size | 17.4 kB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
3bef2c591dacb322cc512ebde044e7fb557453d70035dccddd878685e72ba828
|
|
BLAKE2b-256 checksum How to use checksums |
a6766393e0dcb6e678a021db2389254c15e67a442267a3b8762cbd818d34dbf5
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/6.1.0 CPython/3.9.23
|
Release files / judie-0.0.2-py3-none-any.whl
| Download URL | judie-0.0.2-py3-none-any.whl |
|---|---|
| Size | 18.9 kB |
| Tags | Python 3 |
|
SHA-256 checksum How to use checksums |
4515deea84ddd8302b7eb9eb482f02da0d455da5605c151b01ee681708986c23
|
|
BLAKE2b-256 checksum How to use checksums |
65c9b1261e03571464ff0c7cd05cca9ee043b8d8cc1d22e4180847d97381cb12
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/6.1.0 CPython/3.9.23
|
Security for judie 0.0.2
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
No provenance found
None of the files in this release were uploaded with provenance metadata.
A note on package security
PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.
To learn more about protecting yourself from malicious actors, visit our package security documentation.