llm-judge-panel

A flexible, concurrent framework for evaluating concept matching in text datasets, using an LLM-as-a-Judge voting consensus. Chain multiple model providers and evaluate text rows across custom voting criteria simultaneously.

Features

• Two-stage Workflow: Run Stage 1 (Concept Verification) and Stage 2 (Candidate Matching) independently or combined.
• Parallel Multi-Criteria Evaluation: Evaluate your custom voting rules (e.g., strict and majority agreements) simultaneously in a single run.
• Granular Outputs: View aggregated execution metrics instantly and export row-level details directly to JSON, dictionaries, or pandas.
• Async Execution with Live Progress Tracking: Leverages asyncio to execute judge pools concurrently, with real-time progress tracking.

Installation

It's simple!

pip install llm-judge-panel

Getting Started

1. Configure Environment Variables

Store your respective API keys in your environment variables, namely for the LLM provider(s) you would like to use. We currently support OpenAI, Anthropic, Google Gemini, and Together AI.

export OPENAI_API_KEY="your-key"
export ANTHROPIC_API_KEY="your-key"
export GEMINI_KEY="your-key"
export TOGETHER_API_KEY="your-key"

2. Define Your Concept

"Concepts" are a quantity of interest. These can be anything!

from llm_judge_panel import Concept

skill_concept = Concept(
        name="skill",
        definition="The ability to perform a specific task and apply knowledge in a work context."
    )

3. Define the Voting Criterion

Decide how you would like to resolve the individual LLM votes. You can choose one or multiple (current strict/unanimous, majority, and at_least_m).

from llm_judge_panel import VotingCriterion

evaluation_strategies = {
        "strict": VotingCriterion(rule="strict"),
        "majority": VotingCriterion(rule="majority")
    }

4. Set Up the Panel and Run!

A panel consists of one or more LLM judges. Instantiate the panel with something like the following, and run on your data:

from llm_judge_panel import Panel
from llm_judge_panel.providers import *

async with Panel(criteria=evaluation_strategies) as panel:
    panel.add_judge(OpenAIProvider("gpt-4o-mini"))
    panel.add_judge(AnthropicProvider("claude-sonnet-4-5"))

    # tuples of the form (extracted sentence, match candidate)
    dataset = [
        ("Seeking an expert in distributed Go microservices.", "Go language"),
        ("...", "...")
    ]

    run_result = await panel.judge_full_pipeline(skill_concept, dataset)

    df = run_result.to_dataframe()
    df.to_csv("results.csv", index=False)

Optional: Separate Stages

The function judge_full_pipeline runs both Stage 1 and 2 validation in order. If you would like only to run on of these, do the following:

# setup as before

run_result = await panel.judge_stage1(skill_concept, LIST_OF_LABELED_TEXT_TUPLES) # i.e., (TEXT, 1/0)
# OR
run_result = await panel.judge_stage2(skill_concept, LIST_OF_TUPLES) # i.e., like the above example

Contributing

For more providers or more flexible functionality, feel free to open up an issue!