llm-compare

LLM plugin to compare responses from multiple language models and evaluate their performance.

Installation

Install this plugin in the same environment as LLM:

llm install llm-compare

Usage

The plugin adds a new compare command that allows you to send the same prompt to multiple models and compare their responses:

llm compare "What are the key considerations for AI safety?"

This will:

1. Send the prompt to multiple models in parallel
2. Gather their responses
3. Use an evaluator model to analyze and compare the responses
4. Generate a synthesis of the best elements from each response

Options

• -m, --models: Models to include in comparison (can specify multiple)
• --evaluator: Model to use as evaluator (default: gpt-4)
• --score-threshold: Minimum score threshold (default: 0.7)
• --output: Save full results to a JSON file
• --stdin/--no-stdin: Read additional input from stdin (default: True)

Examples

Basic usage with default models:

llm compare "Explain quantum computing"

Using specific models:

llm compare "Explain blockchain technology" \
  -m gpt-4 \
  -m claude-3-sonnet \
  -m gemini-pro \
  --evaluator claude-3-opus-20240229

Reading from a file:

cat prompt.txt | llm compare "Analysis request:"

Saving full results:

llm compare "Complex technical question" \
  -m gpt-4 \
  -m claude-3-opus \
  --output results.json

Output Format

The command outputs:

1. Individual responses from each model
2. Detailed comparison and analysis
3. Scores for each response
4. Key points of agreement and disagreement
5. A synthesized best answer

Development

To set up this plugin locally:

git clone https://github.com/yourusername/llm-compare
cd llm-compare
python -m venv venv
source venv/bin/activate
pip install -e '.[test]'

Run tests:

pytest

License

This project is licensed under the Apache License 2.0.