juror

A zero-cost LLM evaluation toolkit. Score model outputs against a rubric using a local model as the judge ("LLM-as-judge"), and track how consistent a model is across repeated runs and across different models — running entirely on Ollama with no external API cost.

Status: v0.1.0. Scoring engine, batch evaluation, drift/consistency metrics, latency tracking, SQLite storage, and a Streamlit dashboard are all built and tested — 35 tests passing, including a real end-to-end run against a local model.

What it does

You give juror:

1. a rubric — a set of criteria like correctness, clarity, follows instructions, and
2. a response from some model,

and it uses a second local model to score that response against each criterion, with a short reason for every score. Because it runs on Ollama, evaluating costs nothing and needs no API key.

It also measures things you can only measure when the model runs on your own machine: run-to-run consistency, model-to-model drift, and real latency.

Quick start

# 1. Install Ollama (https://ollama.com), then pull a model:
ollama pull llama3.2

# 2. Install juror:
pip install juror        # or, from source:  pip install -e ".[dev]"

from juror import Criterion, Rubric, score

rubric = Rubric(
    name="qa",
    criteria=[
        Criterion("correctness", "Is the answer factually correct?"),
        Criterion("clarity", "Is the answer clear and easy to understand?"),
    ],
)

result = score(
    prompt="What is the capital of France?",
    response="The capital of France is Paris.",
    rubric=rubric,
    model="llama3.1",
)

for s in result.scores:
    print(s.criterion, s.score, "--", s.reasoning)
print("overall:", result.overall())

See examples/quickstart.py for a runnable version.

How the LLM-as-judge scoring works

juror builds a judge prompt that contains your rubric, the original prompt, and the response to score, and asks the model to return JSON with a score and one-sentence reason per criterion. Scores are parsed, clamped to each criterion's range, and returned as a structured JudgeResult.

The judge is decoupled from the model behind a tiny ModelClient interface, which is what makes it (a) unit-testable with a fake client and (b) swappable between models like Llama 3.1 and Mistral.

Known limitations (read this)

• The judge can share blind spots with the model being judged. If both models are wrong in the same way, the score can look confidently fine. LLM-as-judge is a useful, scalable signal — not ground truth.
• Judges have biases (e.g. rewarding longer or more confident answers). Part of this project is measuring how consistent the judge is, precisely because a single score can't be fully trusted.
• Small local models are noisier judges than large hosted ones. That trade-off — cost and privacy vs. raw judge quality — is the point, not a bug.

Architecture

prompt + response ──► judge prompt ──► local model (Ollama) ──► JSON scores
                                                                    │
                                                        parse + clamp to rubric
                                                                    │
                                                              JudgeResult
                                                          (─► SQLite ─► dashboard)

Sample results

All numbers below are real output from evaluating the 18 built-in starter prompts on two local models — llama3.2 (3B) and gemma2:2b — on a 16 GB laptop (CPU), with each model judging its own answers ("self-eval"). 84 grades in total. Reproduce with examples/generate_demo_data.py.

Model comparison — quality vs. speed

The two are close on speed, but gemma scores its own answers noticeably higher.

Why median latency, not mean? The first grade on the freshly-pulled gemma model took 1148 s — a one-time cost to load the model into memory. Mean latency would report a misleading ~43 s; median (16.6 s) reflects steady-state performance. You can only see this because the model runs on your machine — hosted APIs hide the infrastructure.

A judge-bias caveat, made visible

Because these are self-evals, a higher score doesn't prove a better model — it can mean a more lenient judge. gemma gave itself a perfect 5.00 on completeness, faithfulness, and format; llama gave itself 3.00 on format and 3.11 on conciseness. LLM-as-judge produces relative signals, not absolute truth — exactly the limitation flagged above.

Run-to-run consistency

Same prompt, 5 runs each at temperature 0.8 — std dev of the overall score (lower = steadier):

llama wobbles most on classification — the borderline, subjective tasks. gemma looks steadier, but largely because it's a lenient grader parked near the top of the scale (a ceiling effect): lower wobble isn't automatically "better." Reading consistency and calibration together is the whole point.

Dashboard

juror-dashboard opens this local Streamlit view — score-by-criterion, model comparison, per-prompt consistency, score history, and a raw-data table — read live from your SQLite results.

Design decisions

• Why LLM-as-judge instead of human labels? Human labeling doesn't scale cheaply; LLM-as-judge is an increasingly standard pattern. The trade-off (shared blind spots) is documented above rather than hidden.
• Why local models instead of a hosted API? Zero cost, no rate limits, full reproducibility (you control the exact weights), and data privacy.
• Why SQLite? Zero setup, file-based, perfect for building up a history of eval runs you can chart over time.

Development

pip install -e ".[dev]"
pytest

License

MIT — see LICENSE.