slmjury 0.1.0

💡 What is SLMJury?

SLMJury is a comprehensive framework that investigates whether Small Language Models (0.6B–14B parameters) can serve as reliable judges across both closed-ended (accuracy-based) and open-ended (correlation-based) evaluation paradigms. The project explores six evaluation modes: individual judging, persona-based evaluation, majority-vote ensembles, multi-agent debate, human agreement scoring (SummEval), and LLM agreement scoring (MT-Bench).

🌟 Key Highlights

⚡ Installation

🔧 From Source (Recommended)

git clone https://github.com/anishh15/SLMJury.git
cd SLMJury
pip install -e .

🚀 Quick Start

💻 CLI Scripts

# Step 1: Run student model inference
python scripts/run_student.py --model qwen2.5-32b --datasets gsm8k math

# Step 2: Run judge evaluations
python scripts/run_judge.py --judge qwen3-4b --max-tokens 10 8192

# Step 3: Evaluate all judgements and generate summaries
python scripts/run_evaluation.py

🐍 Python API

from slmjury.core.solver import StudentSolver
from slmjury.core.judge import JudgeModel
from slmjury.core.evaluator import JudgeEvaluator

# Step 1: Solve problems with a student model
solver = StudentSolver("qwen2.5-32b")
results = solver.solve_batch(problems, "gsm8k")
solver.save_results(results, "gsm8k")
solver.cleanup()

# Step 2: Judge the solutions
judge = JudgeModel("qwen3-4b")
judgements = judge.evaluate_batch(results, max_tokens=10)
judge.save_results(judgements, "qwen2.5-32b", "gsm8k", 10)
judge.cleanup()

# Step 3: Evaluate judge accuracy
evaluator = JudgeEvaluator("qwen3-4b", "qwen2.5-32b", "gsm8k", 10, judgements)
summary = evaluator.evaluate()

# Majority voting ensemble
from slmjury.strategies.ensemble import run_majority_voting
run_majority_voting(
    judge_keys=["qwen3-4b", "phi4mi-3.8b", "qwen2.5-3b"],
    student_results=results,
    max_tokens=10,
)

# Multi-agent debate (3 judges, RCR prompting)
from slmjury.strategies.debate import run_debate
run_debate(
    combo_models=["qwen3-4b", "phi4mi-3.8b", "qwen2.5-3b"],
    combo_temps=[0, 0, 0],
    student_results=results,
    dataset_name="gsm8k",
)

# Persona effects (6 system prompts × all judges)
from slmjury.strategies.persona import run_persona_evaluation
run_persona_evaluation("qwen3-4b", results, max_tokens=10)

# Score SummEval with a single judge
python scripts/run_scoring_judge.py \
  --judge qwen3-4b --dataset summeval

# Score MT-Bench with a single judge
python scripts/run_scoring_judge.py \
  --judge qwen3-4b --dataset mtbench \
  --oracle-scores results/mtbench_oracle/

from slmjury.core.scoring_judge import ScoringJudge

judge = ScoringJudge("qwen3-4b", output_dir="results/scoring")

# Score SummEval (4-dimension scoring)
summeval_data = load_dataset("summeval")
results = judge.score_summeval(summeval_data, max_tokens=8192)
judge.save_results(results, "summeval")
judge.cleanup()

🤖 Supported Models

_{*Phi-4 Reasoning/Plus/Mini-Reasoning always use thinking mode and skip quick verdict (t=10) evaluation.}

📊 Datasets

Closed-ended (verdict: Correct/Incorrect):

Open-ended (scoring: 1–5):

🏗️ Project Structure

SLMJury/
├── slmjury/                  # Python package
│   ├── configs/              # Centralized YAML model configurations
│   ├── data/                 # Dataset loaders (HuggingFace → local JSON)
│   ├── parsers/              # Answer extraction, normalization, verdict/score parsing
│   ├── core/                 # Pipeline: solver → judge → evaluator + scoring
│   └── strategies/           # Ensemble voting, multi-agent debate, personas
├── scripts/                  # CLI entry-points (student, judge, oracle, scoring)
├── bash/                     # Bash wrappers for full experiment runs
├── tests/                    # Unit & integration tests (pytest)
├── website/                  # React leaderboard (Vite + Tailwind)
├── assets/                   # SVG banner and logo
├── pyproject.toml            # Package config (pip install -e .)
└── README.md

🏆 Leaderboard

Explore full results on the interactive leaderboard:

📖 Citation

If you use SLMJury in your research, please cite:

@misc{laddha2026slmjury,
      title={SLMJury: Can Small Language Models Judge as Well as Large Language Models?},
      author={Anish Laddha and Nitesh Pradhan and Gaurav Srivastava},
      year={2026},
}

📄 License

Apache License 2.0 — see LICENSE for details.