Features

• Multi-Provider Support: OpenAI, Anthropic, Google Gemini, Groq, Mistral, Cohere, Together AI, Azure, and more (via litellm).
• Rich Terminal Output: Real-time progress bars with ETA.
• Detailed Metrics: Pass rates, P95 latency, TTFT, tokens/sec, and cost.
• Flexible Validation: Regex patterns, custom Python validators, JSON Schema, LLM judges.
• ROUGE-L Scoring: Compare outputs against reference texts.
• Concurrent Execution: Fast parallel testing with rate limiting.
• Caching: Saves money on repeated runs.
• Export: HTML, CSV, or JSON reports with interactive charts.
• Comparison: Compare benchmark results across runs.
• External Data: Load test cases from CSV or JSONL files.
• Interactive Setup: Config wizard to get started quickly.
• CI/CD Friendly: Quiet mode, no-color output, fail-fast, and dry-run support.

Installation

pip install llm-bench-cli

Or install from source:

git clone https://github.com/abdulbb/llm-bench-cli.git
cd llm-bench-cli
pip install -e .

Setting Up API Keys

Set your API keys as environment variables before running benchmarks:

export OPENAI_API_KEY="sk-..."
export ANTHROPIC_API_KEY="sk-ant-..."
export GEMINI_API_KEY="..."
export GROQ_API_KEY="gsk_..."
export OPENROUTER_API_KEY="sk-or-..."

To make these persistent across terminal sessions, add them to your ~/.bashrc, ~/.zshrc, or use a tool like direnv.

For more detailed setup instructions, see Provider Setup.

Documentation

• CLI Reference: All commands and options.
• Configuration Guide: Learn how to write bench.config.yaml.
• Real-World Examples: Start Here! Code generation, summarization, and data extraction scenarios.
• Evaluation & Validation: How results are judged (Regex, Custom Validators, JSON Schema, LLM Judge).
• Exporting Results: Details on HTML, CSV, and JSON output.
• Provider Setup: How to setup API keys for OpenAI, Gemini, Groq, etc.
• Development: How to contribute to this project.

Quick Start

1. Create a new benchmark configuration:

# Interactive wizard
llm-bench init

# Or generate a template
llm-bench init --non-interactive

2. Validate your configuration:

llm-bench validate --config bench.config.yaml

3. Preview without making API calls:

llm-bench run --config bench.config.yaml --dry-run

4. Run the benchmark:

llm-bench run --config bench.config.yaml

Usage Examples

Run a Standard Benchmark:

llm-bench run --config code_gen.yaml

Compare Models on the Fly:

llm-bench run --model openai/gpt-4o --model anthropic/claude-3-5-sonnet-20241022

Generate an Interactive Report:

llm-bench run --config summarization.yaml --export html --output report.html

Set Safety Limits:

llm-bench run --config expensive_test.yaml --max-cost 1.0

Stop on First Failure:

llm-bench run --config ci_tests.yaml --fail-fast

CI/CD Mode (quiet, no colors):

llm-bench run --config bench.config.yaml --quiet --no-color

List Available Models:

llm-bench models
llm-bench models --provider openai
llm-bench models --provider groq

Manage Cache:

llm-bench cache info
llm-bench cache clear

Compare Benchmark Results:

llm-bench compare run1.json run2.json --export html --output comparison.html

Enable Shell Completions:

# Install completions for your shell
llm-bench --install-completion

# Show completion script without installing
llm-bench --show-completion

See CLI Reference for all commands and options, and Real-World Examples for full configuration files.

Example Configuration

name: "Code Generation Benchmark"
system_prompt: "You are a Python coding assistant."
models:
  - "openai/gpt-4o"
  - "anthropic/claude-3-5-sonnet-20241022"
  - "groq/llama-3.1-70b-versatile"

validators_file: "validators.py"

test_cases:
  - input: "Write a function to calculate factorial"
    regex_pattern: "def factorial"
    validator: "is_valid_python"
    
  - input: "What is 2+2?"
    expected: 4

License

MIT