py-openjudge 0.1.7

OpenJudge: A Next-Generation Evaluation System for AI Model Assessment

Holistic Evaluation, Quality Rewards: Driving Application Excellence

Documentation | Contributing | 中文

News

• 2025-10-20 - Auto-Rubric: Learning to Extract Generalizable Criteria for Reward Modeling - We released a new paper on learning generalizable reward criteria for robust modeling.
• 2025-10-17 - Taming the Judge: Deconflicting AI Feedback for Stable Reinforcement Learning - We introduced techniques to align judge feedback and improve RL stability.
• 2025-07-09 - Released OpenJudge v0.1.0 on PyPI

Evaluation and reward signals are the cornerstones of application excellence. Holistic evaluation enables the systematic analysis of shortcomings to drive rapid iteration, while high-quality rewards provide the essential foundation for advanced optimization and fine-tuning. Open-Judge unifies reward signals and evaluation metrics into one Grader interface—with pre-built graders, flexible customization, and seamless framework integration.

Key Features

• Systematic & Quality-Assured Grader Library: Access N+ production-ready graders featuring a comprehensive taxonomy, rigorously validated for reliable performance.

  • Multi-Scenario Coverage: Extensive support for diverse domains including Agent, text, code, math, and multimodal tasks via specialized graders.
  • Holistic Agent Evaluation: Beyond final outcomes, we assess the entire lifecycle—including trajectories and specific components (Memory, Reflection, Tool Use).
  • Quality Assurance: Built for reliability. Every grader comes with benchmark datasets and pytest integration for immediate quality validation.

• Flexible Grader Building Methods: Choose the build method that fits your requirements:

  • Customization: Easily extend or modify pre-defined graders to fit your specific needs.
  • Data-Driven Rubrics: Have a few examples but no clear rules? Use our tools to automatically generate white-box evaluation criteria (Rubrics) based on your data.
  • Trainable Judge Models: For high-scale scenarios, train dedicated Judge models as Graders. We support SFT, Bradley-Terry models, and Reinforcement Learning workflows.

• Easy Integration: Seamlessly connect with mainstream evaluation platforms (e.g., LangSmith, LangFuse) and training frameworks (e.g., VERL) using our comprehensive tutorials and flexible APIs.

Installation

pip install open_judge

More installation methods can be found in the here.

Quickstart

import asyncio
from open_judge.models import OpenAIChatModel
from open_judge.graders.common.relevance import RelevanceGrader


# step1 create model client
model = OpenAIChatModel(model="qwen3-32b")

# step2 choose and initialize proper grader
grader = RelevanceGrader(model=model)

# step3 Prepare data

data = {
    "query": "What is machine learning?",
    "response": "Machine learning is a subset of AI that enables computers to learn from data.",
}

# step 4 Evaluate using the data
result = await grader.aevaluate(**data)

print(f"Score: {result.score}")  # Score: 5
print(f"Reason: {result.reason}")

Complete Quickstart can be found in here.

Integrations

Integration	Documentation
LangSmith	LangSmith
LangFuse	LangFuse
Arize Phoenix	Arize Phoenix

Contributing

We welcome contributions from the community!

1. Raise and comment on Issues.
2. Open a PR - Whether you're fixing bugs, adding new features, improving documentation, or sharing ideas, your contributions help make Open-Judge better for everyone. See Contributing for more details.

Citation

If you use Open-Judge in your research, please cite:

@software{
title = {OpenJudge: XXXX},
author = {The Open-Judge Team},
url = {https://github.com/modelscope/Open-Judge},
month = {07},
year = {2025}
}