llm-arbiter 0.1.1

A/B testing for LLMs. Statistical proof, not vibes.

Verdict

A/B testing for LLMs. Statistical proof, not vibes.

Verdict is a Python framework for systematically comparing large language models with statistical rigor. Instead of picking models based on gut feeling, verdict helps you make data-driven decisions using rigorous experimental design and statistical analysis.

Features

• 🧪 Parallel A/B Testing — Run multiple model variants concurrently on the same queries
• 📊 Statistical Analysis — Built-in significance testing using scipy
• 💰 Cost Tracking — Monitor and compare inference costs across models
• 🎯 LLM Judge Scoring — Automatic evaluation using GPT-4o-mini with custom criteria
• ⚡ Performance Metrics — Track latency, tokens, and costs
• 📈 Variant Comparison — Side-by-side results with statistical significance

Installation

pip install llm-arbiter

Quick Start

from verdict import Experiment, Variant, Judge

# Define model variants to test
variants = [
    Variant(
        name="gpt-4o",
        fn=call_model,
        config={"model": "gpt-4o"},
        model="gpt-4o"
    ),
    Variant(
        name="gpt-4o-mini",
        fn=call_model,
        config={"model": "gpt-4o-mini"},
        model="gpt-4o-mini"
    ),
]

# Create an experiment
experiment = Experiment(
    queries=["What is X?", "Explain Y"],
    variants=variants,
)

# Run the experiment
results = experiment.run()

# Analyze results with optional LLM judge
judge = Judge(criteria=[
    "Does it cite specific numbers?",
    "Is the analysis correct?",
])
analysis = experiment.analyze(judge=judge)
print(analysis.summary())

Core Components

Variant

Represents a single model configuration being tested.

Variant(
    name="gpt-4o",
    fn=call_model,  # (config, query) → str or dict
    config={"model": "gpt-4o"},  # Passed to fn
    model="gpt-4o",  # For pricing lookup
    pricing={"input": 2.50/1_000_000, "output": 10.00/1_000_000}  # Optional override
)

Experiment

Defines and runs the test across all variants.

• queries: List of queries to test
• variants: List of Variant objects
• num_workers: Parallel execution threads (default: auto)

Judge

Scores model outputs using GPT-4o-mini based on custom criteria.

judge = Judge(
    criteria=["Is it accurate?", "Does it cite sources?"],
    model="gpt-4o-mini"  # Can override
)

Analysis

Statistical results after running the experiment.

• RunResult: Individual execution data (latency, tokens, cost, score)
• summary(): Display results with significance testing

Supported Models & Pricing

Pre-configured pricing for OpenAI models:

• gpt-4o
• gpt-4o-mini
• gpt-4.1
• gpt-4.1-mini
• gpt-4.1-nano
• o3-mini

Custom pricing can be specified per variant.

Example: Market Research Analysis

See test.py for a complete example comparing models on market research questions.

Requirements

• Python 3.9+
• numpy >= 1.24
• scipy >= 1.10
• openai >= 1.0

Environment Variables

Set your OpenAI API key:

export OPENAI_API_KEY="sk-..."

License

MIT

Contributing

Contributions welcome! This project uses statistical rigor—pull requests should include validation.