Mini Judge
Simple implementation of LLM-As-Judge for pairwise evaluation of Q&A models.
Installation
Install the package using pip:
pip install mini-judge
Usage
First, set the OPENAI_API_KEY environment variable to your OpenAI API key.
Then, you can run the following command to evaluate the candidate answers in candidate_answers_path against the reference answers in ref_answers_path using judge_model as the judge model.
mini-judge \
--judge_model <judge_model> \
--questions_path <questions_path> \
--candidate_answers_path <candidate_answers_path> \
--ref_answers_path <ref_answers_path> \
--output_path <output_path>
To run a quick demo, use the following command to evaluate the candidate answers in example_data/candidate_answers.jsonl against the reference answers in example_data/ref_answers.jsonl using GPT-4 as the judge model.
mini_judge --output_path <output_path>
Data Format
All input data files are presumed to be in jsonl format.
The candidate_answers and ref_answers files should have each line as a json with an answer tag.
Similarly, the questions file should have json lines with a question tag.
All other tags will be ignored.
References
Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric. P Xing, Hao Zhang, Joseph E. Gonzalez, & Ion Stoica. (2023). Judging LLM-as-a-judge with MT-Bench and Chatbot Arena ArXiv.
Metadata
Release files for mini-judge 0.4.1
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | Provenance | Actions |
|---|---|---|---|---|
| mini_judge-0.4.1.tar.gz | 17.2 kB | Without provenance | Details for mini_judge-0.4.1.tar.gz |
Built distribution (wheel)
| File | Interpreter | ABI | Platform | Actions Reset |
|---|---|---|---|---|
| mini_judge-0.4.1-py3-none-any.whl | Python 3 | none | any | Details for mini_judge-0.4.1-py3-none-any.whl |
Total release size: 36.1 kB
Release files / mini_judge-0.4.1.tar.gz
| Download URL | mini_judge-0.4.1.tar.gz |
|---|---|
| Size | 17.2 kB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
5e9f69630cc626fc89dea9bfe06a3165e071b9189c6deb0237340ebdfa02416a
|
|
BLAKE2b-256 checksum How to use checksums |
4cf15df2d5a170e7d7633faa5bd1253eba3150471e4e1b8a1a34e52e3ee84de9
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
poetry/1.3.2 CPython/3.11.5 Linux/6.2.0-1012-azure
|
Release files / mini_judge-0.4.1-py3-none-any.whl
| Download URL | mini_judge-0.4.1-py3-none-any.whl |
|---|---|
| Size | 18.8 kB |
| Tags | Python 3 |
|
SHA-256 checksum How to use checksums |
d5d62cb3a6d5ca98eac146b6718151ea20566ed07b6e3b8ac8aeac84662fbcbe
|
|
BLAKE2b-256 checksum How to use checksums |
86e7f678d2ae43132b24a6f29daaa248410269477651afd576e162e0b69abd6a
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
poetry/1.3.2 CPython/3.11.5 Linux/6.2.0-1012-azure
|
Security for mini-judge 0.4.1
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
No provenance found
None of the files in this release were uploaded with provenance metadata.
A note on package security
PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.
To learn more about protecting yourself from malicious actors, visit our package security documentation.