ParametricBench

Catch LLM and VLM regressions by running the same evaluation cases across configurable prompts, models, and generation settings.

ParametricBench is a lightweight, provider-independent test runner for validating model behavior with plain Python. Use it to compare configurations, test prompt changes, validate text or multimodal responses, and run repeatable model checks locally or in CI.

Installation

pip install parametricbench

Usage

Run several checks in one benchmark while keeping case-specific expectations and settings together:

from parametricbench import benchmark


def model(prompt, temperature=0):
    return prompt.upper()


def contains_expected(response, case):
    return case["expected"] in response


cases = [
    {"name": "uppercase-greeting", "input": "hello", "expected": "HELLO"},
    {
        "name": "uppercase-question",
        "input": "how are you?",
        "expected": "HOW ARE YOU?",
        "params": {"temperature": 0.2},
    },
    {
        "name": "contains-keyword",
        "input": "parametric benchmarks",
        "expected": "BENCHMARKS",
    },
]

report = benchmark(
    target=model,
    cases=cases,
    evaluators=[contains_expected],
    params={"temperature": 0},
)

print(f"Passed: {report['passed_count']}/{report['total']}")
for result in report["results"]:
    print(result["name"], result["passed"], result["score"])

Why ParametricBench?

Model behavior can change when prompts, providers, models, retrieved context, or generation settings change. Manual checks are difficult to reproduce and easy to forget. ParametricBench turns them into reusable Python benchmarks for development, automated tests, production checks, and CI.

It is intentionally smaller than a complete evaluation platform:

• No provider lock-in or required model SDK
• No server, database, dashboard, or configuration language
• No runtime dependencies or infrastructure to deploy

Use it when an ad hoc script is too fragile but a full evaluation platform is unnecessary.

Compare configurations

report = benchmark(
    target=model,
    cases=cases,
    variants=[
        {"name": "deterministic", "params": {"temperature": 0}},
        {"name": "creative", "params": {"temperature": 0.8}},
    ],
    evaluators=[contains_expected],
)

Each case runs once with every named variant. Parameters are merged without mutation in this order: global, then variant, then case-specific parameters, with later values taking precedence.

Custom evaluators

Evaluators receive the response and complete benchmark case:

def exact_match(response, case):
    return response == case["expected"]

They may return a boolean or a numeric score from 0.0 to 1.0. Multiple scores are averaged. Without evaluators, a successful target call scores 1.0; a target error scores 0.0. Target and evaluator errors are captured without stopping later cases.

LLM and VLM inputs

Inputs can be any Python value:

case = {
    "input": {"image": "traffic-light.png", "prompt": "Which object is shown?"},
    "expected": "traffic light",
}

ParametricBench passes the value to your target without loading or interpreting it.

Use cases

• Detect regressions after prompt or model changes
• Compare model versions, providers, temperatures, or environments
• Validate structured, text, and multimodal responses
• Reuse the same checks in development, production, and CI

Features

• Any Python callable and arbitrary inputs
• Global, variant, and case parameters
• Boolean or numeric evaluators and configurable thresholds
• Ordered dictionary reports with timing and concise errors
• No runtime dependencies

Issues

Report issues at https://github.com/edujbarrios/parametricbench.

Author

Eduardo J. Barrios — edujbarrios@outlook.com

License

Mozilla Public License 2.0