Twelve Angry LLMs

Panel-based reliability annotation for preference data.

Modern preference datasets are built by asking a single strong judge to score or rank candidate responses, then collapsing its verdict into (chosen, rejected) pairs for DPO or reward-model training. That collapse throws away any notion of how contestable each label was.

twelve-angry-llms recovers that signal: it runs a panel of diverse LLM judges over each (prompt, K responses) datapoint and measures how much the judges agree with each other — the inter-judge agreement (IJA), computed as average pairwise Kendall's τ-b over the judges' rankings. High IJA means the label is trustworthy; low IJA means it is shaky and likely to inject noise into training. The result exports straight into TRL's (prompt, chosen, rejected) schema with the reliability signals attached, so you can filter, down-weight, or soft-label your data and run your existing training setup unchanged.

Installation

pip install twelve-angry-llms          # core: OpenRouter + any OpenAI-compatible provider
pip install "twelve-angry-llms[data]"  # + raw UltraFeedback / Nectar loaders

Quickstart

The easiest way to run a cross-family panel is a single OpenRouter key — one account, hundreds of models:

from twelve_angry_llms import (
    Judge, Panel, PreferenceDatapoint, ScoringProtocol,
    OpenRouterClient, to_records, to_jsonl,
)

openrouter = OpenRouterClient()  # reads OPENROUTER_API_KEY
panel = Panel([
    Judge(model="openai/gpt-4o-2024-08-06", client=openrouter),
    Judge(model="anthropic/claude-sonnet-4.5", client=openrouter),
    Judge(model="qwen/qwen-2.5-72b-instruct", client=openrouter),
])

datapoints = [
    PreferenceDatapoint(
        prompt="What causes tides?",
        responses=(
            "The gravitational pull of the moon and sun.",
            "Mostly wind patterns over the ocean.",
            "The moon's gravity, with a smaller solar contribution.",
        ),
    ),
]

results = panel.annotate_sync(datapoints, ScoringProtocol())
print(results[0].ija)              # panel agreement on this datapoint, in [-1, 1]
print(panel.diagnostics(results))  # Krippendorff's alpha, judge-judge correlations

to_jsonl(to_records(results), "annotated.jsonl")  # TRL schema + IJA columns

Judges aren't tied to OpenRouter: OpenAICompatibleClient (which OpenRouterClient is a preconfigured version of) speaks to any OpenAI-compatible endpoint — OpenAI directly, Together, Groq, a local vLLM or Ollama server, ... — via its base_url. Keys are read from environment variables; pass api_key_env to point at a different variable per provider.

from twelve_angry_llms import OpenAICompatibleClient

openai = OpenAICompatibleClient()  # OPENAI_API_KEY
local = OpenAICompatibleClient(base_url="http://localhost:11434/v1", api_key="-")

Each exported record carries:

Two elicitation protocols

• ScoringProtocol — every judge scores each response 1–5 against a shared rubric (the UltraFeedback style). Also yields per-judge score margins for free.
• RankingProtocol — every judge orders all K responses best-to-worst in one pass (the Nectar style).

Both parse into per-response utilities, so IJA and all exports work identically. Judges receiving one shared guideline per protocol is the point: any disagreement then reflects the judges, not prompt wording.

Command line

tal annotate --input data.jsonl --panel panel.yaml --output annotated.jsonl
tal export   --input annotated.jsonl --output pairs.jsonl

data.jsonl rows need prompt and responses; panel.yaml declares the judges and (optionally) a response cache:

metric: kendall
cache: .tal/cache.sqlite
judges:
  - model: openai/gpt-4o-2024-08-06
    provider: openrouter                 # the default
  - model: anthropic/claude-sonnet-4.5
    provider: openrouter
  - model: qwen/qwen-2.5-72b-instruct
    provider: openrouter
  - model: llama3.1:70b                 # any OpenAI-compatible endpoint also works
    provider: openai
    base_url: http://localhost:11434/v1

With the data extra you can annotate the raw (non-binarized) datasets directly: tal annotate --dataset ultrafeedback --limit 1000 ....

Reproducible OpenRouter runs

One caveat with aggregators: they load-balance open-weight models across hosts with different quantizations, which can silently change judge behavior between runs. Pin the routing with extra_body (merged into every request, and part of the response-cache key):

  - model: qwen/qwen-2.5-72b-instruct
    provider: openrouter
    extra_body:
      provider: {order: [together], allow_fallbacks: false, quantizations: [fp16]}

Use exact-versioned model slugs (not ~latest aliases) for the same reason. See research/configs/panel.example.yaml for a fully pinned five-judge panel.

Reproducibility

• Temperature defaults to 0 everywhere.
• CachedClient / the cache: key stores every raw response in SQLite, keyed on (model, sampling settings, messages) — re-runs never re-bill.
• Clients track token usage (client.usage) so runs can report cost.

The research

This library is the instrument for an ongoing study of IJA as a label-reliability signal — hypotheses, judging protocol, and experimental phases live in research/RESEARCH_PLAN.md. Working title: "Twelve Angry LLMs: Judge Agreement as a Label-Reliability Signal for Preference Data."

License

MIT