ai-prompt-benchmark 0.1.0

Measure cost, quality, and performance of LLM API interactions

llm-bench

Stop guessing which AI model to use. Start measuring.

Most teams pick AI providers based on marketing claims or gut feeling. llm-bench gives you evidence — real numbers on cost, quality, and speed — so you can make decisions grounded in data.

from llm_bench import Suite, Scenario
from llm_bench.quality import contains, max_length
from llm_bench.providers import OpenAIProvider, AnthropicProvider

suite = Suite(
    scenarios=[
        Scenario(
            prompt="Summarize this support ticket in one sentence.",
            providers=["openai/gpt-4o-mini", "anthropic/claude-haiku-4-5"],
            assertions=[contains("customer"), max_length(200)],
        )
    ],
    providers={
        "openai": OpenAIProvider(api_key="..."),
        "anthropic": AnthropicProvider(api_key="..."),
    },
)

suite.run().save("results")

openai/gpt-4o-mini      cost=$0.000089  quality=2/2  latency=743ms
anthropic/claude-haiku  cost=$0.000104  quality=2/2  latency=921ms

---

Why this exists

AI APIs cost real money. Response quality varies across models, prompts, and providers. Latency affects user experience. Yet most teams have no systematic way to compare any of this — they rely on vibes.

llm-bench treats AI evaluation as an engineering problem: define what "good" means, run the benchmark, read the numbers.

---

What it measures

Cost

Token-level cost tracking with a bundled pricing table (GPT-4o, GPT-4o-mini, Claude Haiku, Sonnet, Opus, and more). Every run reports input_usd, output_usd, and total_usd. No surprises on your bill.

Quality

Two complementary layers:

Deterministic assertions — fast, free, exact. Did the response contain the right word? Match a pattern? Stay within a length limit? Return valid JSON? These run on every call with zero extra cost.

from llm_bench.assertions import contains, not_contains, matches, max_length, valid_json, json_schema

assertions=[
    contains("Paris"),
    not_contains("I don't know"),
    matches(r"\d{4}-\d{2}-\d{2}"),   # date format
    max_length(500),
    valid_json(),
]

LLM-as-judge (opt-in) — a second model scores the response against a rubric you define, on a scale you choose. Judge cost is tracked separately and never blended into deterministic scores.

from llm_bench import Judge

judge=Judge(
    provider="openai",
    model="gpt-4o-mini",
    rubric="Is this explanation clear and accurate for a non-technical audience?",
    scale=10,
)

Performance

• Latency (total_ms) — full round-trip time
• Throughput (tokens_per_sec) — output tokens per second
• Reliability — retry count and final error message, with automatic exponential backoff on rate limits and transient errors (up to 3 retries: 1s, 2s, 4s)

---

Output

Results export to JSON (full detail) and CSV (flat summary), both from one call:

suite.run().save("results")          # writes results.json + results.csv
suite.run().save("results.json")     # JSON only
suite.run().save("results.csv")      # CSV only

JSON includes assertion reasons, raw provider payloads, and per-call metadata. CSV is ready for Excel, Pandas, or CI artifact dashboards.

---

Installation

# OpenAI only
pip install llm-bench[openai]

# Anthropic only
pip install llm-bench[anthropic]

# Both
pip install llm-bench[all]

Python 3.11+. MIT license.

---

Extending

New providers implement one method:

from llm_bench.providers.base import Provider
from llm_bench.scenario import CallConfig, LLMResponse

class MyProvider(Provider):
    def call(self, prompt: str, config: CallConfig) -> LLMResponse:
        response = my_sdk.complete(prompt, model=config.model)
        return LLMResponse(
            text=response.text,
            input_tokens=response.usage.input,
            output_tokens=response.usage.output,
            model=config.model,
            raw=response.raw_dict,
        )

New assertions implement one callable:

def my_assertion(text: str) -> AssertionResult:
    passed = "expected phrase" in text
    return AssertionResult(passed=passed, reason="Found" if passed else "Missing")

No registration, no subclassing beyond these — just pass instances in.

---

Architecture

Seven focused modules, no circular imports, each with one responsibility:

llm_bench/
├── scenario.py      # All core dataclasses (Scenario, Result, Cost, Performance, ...)
├── suite.py         # Orchestration — runs the scenario × provider matrix
├── cost.py          # Token counting + USD calculation from pricing table
├── quality.py       # Deterministic assertions + QualityChecker
├── performance.py   # Latency and throughput tracking
├── report.py        # Aggregation + JSON/CSV serialization
├── assertions.py    # Re-export of quality assertion factories
└── providers/
    ├── base.py      # Abstract Provider interface
    ├── openai.py    # OpenAI adapter
    └── anthropic.py # Anthropic adapter

Key design decision: QualityChecker never makes API calls. All provider calls — including judge calls — are centralized in Suite. This keeps dependencies explicit and modules independently testable.

---

Testing

51 unit tests, zero API calls required:

pip install llm-bench[dev]
pytest tests/unit/ -v

Integration tests hit real APIs and are opt-in:

RUN_INTEGRATION=1 OPENAI_API_KEY=... ANTHROPIC_API_KEY=... pytest tests/integration/

CI runs unit tests on every pull request (Python 3.11 and 3.12). Integration tests run on a weekly schedule.

---

Examples

# Side-by-side provider comparison
OPENAI_API_KEY=... ANTHROPIC_API_KEY=... python examples/basic_comparison.py

# With LLM-as-judge scoring
OPENAI_API_KEY=... python examples/with_judge.py

---

Roadmap

• More providers (Gemini, Cohere, Mistral, local models via Ollama)
• Parallel execution (v2)
• Streaming support + TTFT metric
• CLI interface

Contributions welcome. Open an issue before starting large changes.

---

MIT License · Built with evidence, not vibes.