zrb-llm-evaluator

Multi-trial experiment runner for testing LLMs against structured test cases via zrb chat.

Given a set of models and test cases, it runs every combination across N trials, collects structured results, and generates reports. Supports concurrent execution, timeout handling, resume from partial runs, and pluggable validators.

Quick Start

# Install
poetry install

# Create a test case directory
mkdir -p my-cases/hello-world
cat > my-cases/hello-world/instruction.txt << 'EOF'
Write a Python function that returns "hello, world".
EOF

# Run an experiment (requires zrb installed and configured)
zrb-llm-evaluator run \
  --models openai:gpt-4o \
  --test-cases ./my-cases/hello-world/ \
  --trials 1

Features

• Runs N models × M test cases × T trials as a flat experiment grid
• Concurrent trial execution with configurable parallelism (asyncio + semaphore)
• Per-trial timeout kills subprocess and preserves partial output
• Resume support: re-run with the same output directory to skip completed cells
• Pluggable validators: each test case provides a validator.py implementing a typed protocol
• Atomic results.json writes (temp file + os.rename)
• Pluggable CLI adapters: built-in support for zrb, claude-code, and opencode, or bring your own via a dotted import path
• Custom CLI binary name for white-labeled forks

Installation

Prerequisites: Python ≥ 3.11, Poetry, and an installed zrb CLI with API keys configured.

git clone <repo>
cd zrb-llm-evaluator
poetry install

Verify the CLI works:

poetry run zrb-llm-evaluator --help

Usage

Test Case Format

Each test case is a directory containing:

my-cases/<test-name>/
├── instruction.txt    # Required — the prompt sent to the LLM
├── validator.py       # Required — validation logic (see below)
└── workdir/           # Optional — files copied into the trial's working directory

Writing a Validator

validator.py must expose a top-level validator object that implements ValidatorProtocol:

# my-cases/hello-world/validator.py
from pathlib import Path
from zrb_llm_evaluator.models import TrialTrace, ValidationResult, ValidationCheck
from zrb_llm_evaluator.protocols import ValidatorProtocol

class HelloValidator:
    def validate(
        self,
        output_dir: Path,
        log_content: str,
        trace: TrialTrace | None = None,
    ) -> ValidationResult:
        passed = "hello, world" in log_content.lower()
        return ValidationResult(
            status="PASS" if passed else "FAIL",
            score=1.0 if passed else 0.0,
            details=[
                ValidationCheck(
                    name="contains_greeting",
                    passed=passed,
                    message="Output contains 'hello, world'" if passed
                            else "Missing expected greeting",
                )
            ],
        )

validator = HelloValidator()

The framework validates protocol conformance at load time. validator.py that doesn't implement ValidatorProtocol is rejected before any trial runs.

The trace parameter is a TrialTrace parsed from the per-trial session history. It exposes:

• tool_calls: list[ToolCallRecord] — every tool invocation with name and args dict
• tool_names: list[str] — convenience list of just the names, in order
• assistant_text: str — concatenated assistant message text
• turn_count: int

Use it to assert how the agent succeeded (e.g., "called the right tool", "iterated at least twice", "never invoked an HTTP tool with secret values") rather than only the final filesystem state. Validators that don't need trajectory data can ignore the parameter.

The reporter aggregates trials across (model, test_case) cells and surfaces a Stability section: a cell with mixed pass/fail across trials is flagged 🟡 FLAKY — a one-off lucky pass is no longer indistinguishable from a deterministic pass. This section is suppressed when every cell has a single trial (stability is undefined with N=1).

ValidationResult

CLI Reference

run

Run a full experiment.

zrb-llm-evaluator run \
  --models openai:gpt-4o,google-gla:gemini-2.5-flash \
  --test-cases ./cases/bug-fix,./cases/copywriting \
  --trials 3 \
  --parallelism 4 \
  --timeout 300 \
  --cli-name zrb \
  --output-dir ./out

Output: results.json (structured) + report.md (human-readable).

list

List completed trials from a previous experiment.

zrb-llm-evaluator list --dir ./out

report

Re-generate the Markdown report from existing results.json.

zrb-llm-evaluator report --dir ./out

Architecture

The runner has four layers:

1. CLI (cli.py) — Typer entry point, parses args, validates config
2. Loader (loader.py) — Discovers test cases from directories, imports validators, checks protocol conformance
3. Runner (runner.py) — Async subprocess orchestration with asyncio.Semaphore, asyncio.wait_for, and ResumeManager for idempotent resumption. Each trial creates an isolated directory {output}/{model_safe}/{test_case}/trial-{N}/ (colons in the model name are sanitized) with its own history directory
4. Reporter (reporter.py) — Generates Markdown and JSON output with atomic file writes

Key design decisions are documented in .sdlc/docs/adr/.

CLI Adapters

--cli-template selects a CliAdapter (cli_adapters.py) that owns every CLI-specific decision — the subprocess command line, env vars, and how usage/tool calls are parsed from output. The runner itself never hardcodes a particular CLI's invocation.

For a custom CLI, pass a dotted import path (e.g. --cli-template mypkg.MyAdapter) to a class implementing the CliAdapter protocol in protocols.py. Unresolvable templates fail fast before any trial runs.

Output Structure

./out/
├── results.json          # Structured results (list of TrialResult)
├── report.md             # Human-readable report
├── openai_gpt-4o/
│   └── bug-fix/
│       ├── trial-1/
│       │   ├── stdout.log            # Raw subprocess stdout/stderr
│       │   └── history/              # ZRB_LLM_HISTORY_DIR
│       │       └── <session>.json
│       ├── trial-2/
│       └── trial-3/
└── google-gla_gemini-2.5-flash/
    └── ...

Development

poetry install --with dev
poetry run pytest tests/experiment-runner/
poetry run ruff check
poetry run mypy src/

License

MIT.