openrubricrl 0.1.0

Open-source pipeline that converts human-written rubrics into LLM-based reward functions for RL and RLHF training

OpenRubricRL

An open-source pipeline that converts human-written rubrics into LLM-based reward functions for RL and RLHF training.

🎯 Problem It Solves

Current RLHF pipelines require expensive human labelers to score outputs. Labs want LLM-based reward models to scale scoring — but they need high-quality rubrics to make that work. No open standard exists for turning a rubric into a reusable, consistent reward function.

OpenRubricRL fills this gap by providing:

• 📋 A standard JSON/YAML schema for defining evaluation rubrics
• 🤖 Automatic conversion of rubrics into LLM scoring prompts
• 🔌 Ready-to-use API and CLI tools for scoring model outputs
• 🧪 Integration with popular RL libraries (RLlib, TRL, CleanRL)

🚀 Quick Start

Installation

pip install openrubricrl

For development with all features:

pip install openrubricrl[all]

Basic Usage

1. Create a Rubric

openrubricrl create-template my_rubric --domain code

This creates my_rubric.json with a basic template. Edit it to define your criteria:

{
  "name": "code_quality_basic",
  "version": "1.0.0",
  "description": "Basic code quality evaluation",
  "domain": "code",
  "scale": {"min": 0.0, "max": 10.0},
  "criteria": [
    {
      "name": "correctness",
      "description": "Does the code solve the problem correctly?",
      "weight": 0.4,
      "examples": {
        "excellent": [
          {
            "input": "Write a function to reverse a string",
            "output": "def reverse_string(s): return s[::-1]",
            "score": 9.0,
            "explanation": "Correct and efficient implementation"
          }
        ]
      }
    },
    {
      "name": "readability", 
      "description": "Is the code clean and readable?",
      "weight": 0.6
    }
  ]
}

2. Score Model Outputs

Command Line:

export OPENAI_API_KEY="your-key-here"

openrubricrl score my_rubric.json \
  "Write a function to add two numbers" \
  "def add(a, b): return a + b"

Python API:

from openrubricrl import Rubric, create_openai_scorer

# Load rubric
rubric = Rubric.from_file("my_rubric.json")

# Create scorer
scorer = create_openai_scorer(rubric, api_key="your-key")

# Score an output
result = await scorer.score(
    task_input="Write a function to add two numbers",
    model_output="def add(a, b): return a + b"
)

print(f"Score: {result.overall_score}/10")
print(f"Explanation: {result.overall_explanation}")

REST API:

# Start server
openrubricrl serve --rubrics-dir ./rubrics

# Score via HTTP
curl -X POST "http://localhost:8000/score/my_rubric" \
  -H "Content-Type: application/json" \
  -d '{
    "task_input": "Write a function to add two numbers",
    "model_output": "def add(a, b): return a + b"
  }'

🏗️ Architecture

Core Components

1. Rubric Schema (rubric_schema.json): JSON schema defining the standard format
2. Prompt Builder (prompt_builder.py): Converts rubrics into LLM prompts
3. Scorer (scorer.py): Handles LLM API calls and response parsing
4. API Server (server.py): FastAPI-based REST API
5. CLI (cli.py): Command-line interface

Supported LLM Providers

• ✅ OpenAI (GPT-4, GPT-3.5)
• ✅ Anthropic (Claude)
• 🔄 Local models via vLLM (coming soon)

📖 Examples

See the examples/ directory for complete examples:

• code_evaluation.py - Scoring code generation
• dialogue_quality.py - Evaluating chatbot responses
• creative_writing.py - Scoring creative content
• batch_scoring.py - Processing multiple outputs

🔗 Integrations

Reinforcement Learning Libraries

# RLlib integration example
from openrubricrl.integrations.rllib import RubricRewardFunction

reward_fn = RubricRewardFunction(
    rubric_path="my_rubric.json",
    provider="openai"
)

# Use in your RL training loop
reward = reward_fn(state, action, context)

Hugging Face Transformers

from openrubricrl.integrations.transformers import RubricCallback

trainer = Trainer(
    model=model,
    callbacks=[RubricCallback(rubric_path="my_rubric.json")],
    # ... other args
)

🧪 Development

Setup

git clone https://github.com/openrubricrl/openrubricrl.git
cd openrubricrl
pip install -e ".[dev]"

Run Tests

pytest tests/ -v

Code Quality

black src/ tests/
isort src/ tests/
flake8 src/ tests/
mypy src/

📚 Documentation

• Rubric Schema Reference
• API Documentation
• Integration Guide
• Contributing Guidelines

🗓️ Roadmap

Phase 1 - Foundation ✅

• JSON/YAML schema for rubrics
• Rubric → prompt converter
• Minimal scoring API with OpenAI/Anthropic
• CLI tool for local scoring

Phase 2 - Community & Repository (Q2 2024)

• Open Rubric Hub (Git repo with curated rubrics)
• Templates for common domains (code, dialogue, writing)
• Contribution guidelines and review process

Phase 3 - Integrations & Scaling (Q3 2024)

• RLlib / TRL integration examples
• Hybrid reward module (LLM + automated metrics)
• Bias/drift detection module
• Local model support via vLLM

Phase 4 - Sustainability (Q4 2024)

• Hosted API service (optional paid tier)
• Enterprise features and support
• Dataset hosting for scoring logs

🤝 Contributing

We welcome contributions! Please see CONTRIBUTING.md for guidelines.

Early Adopters & Target Users

• 🔬 Small RL research teams without budget for large-scale human feedback
• 🏆 AI hackathon participants who want reward shaping quickly
• 🚀 Startups doing RLHF in niche domains (customer service bots, educational tutors)
• 🎓 Academics studying automated evaluation methods

📜 License

MIT License - see LICENSE for details.

🙏 Acknowledgments

• Inspired by the need for standardized evaluation in RLHF
• Built on top of excellent libraries: FastAPI, Pydantic, Click
• Thanks to the open-source RL and NLP communities

---

🔗 Links:

• GitHub Repository
• Documentation
• PyPI Package
• Discord Community