RetryBench

Evaluate whether an LLM's first recovery decision after a tool timeout is permitted, and whether replaying that decision achieves the intended state.

Install and run

Python 3.11+ for model inference:

pip install "retrybench[inspect]"
retrybench run --model mockllm/retrybench-defer

The local mock runs all 240 cases without credentials or inference charges. It always defers: unsupported retries 0/240, permitted progress 0/80, and unresolved outcomes 480/480. These are synthetic fixture results.

For a real model, set its provider's API credentials and pass a provider/model identifier supported by Inspect AI:

retrybench run --model "$MODEL_ID"

The inspect extra includes the OpenAI and Anthropic clients. Other providers may require their own SDK; follow Inspect's provider setup instructions.

This makes up to 240 billable inference calls with a 256-token output limit per call and no runner retries. --dry-run shows the call count without inference; --limit 4 runs a diagnostic subset and reports the remaining cases as missing. Temperature and reasoning effort use provider defaults unless supplied. Each run prints its result directory and writes predictions, per-case scores, and a report. Existing runs are never overwritten. Partial runs exit with code 2.

Use any inference framework

The core scorer supports Python 3.9+ with no third-party runtime dependencies:

pip install retrybench
retrybench export --output cases.jsonl
# Generate one prediction per visible case using your model or framework.
retrybench score --predictions predictions.jsonl --model my-model

Predictions are JSONL records such as:

{"case_id":"<exact exported case_id>","action":"unresolved","arguments":{"idempotency_key":null}}

Use retrybench run without --model for the deterministic policy baselines. The earlier retrybench eval --model ... command remains supported.

Data and interpretation

retry-permission-v1 freezes 240 visible decisions and 480 correlated replayed outcomes across four reused software tasks. Reports separate unsupported retry permission, permitted progress, protocol validity, duplicates, omissions, and unresolved recovery, with exact denominators and missing/invalid accounting. The labeled suite is public and is not a held-out generalization test. Each model receives only visible prompts and output schemas; keep scoring labels outside inference. No real side-effecting tool is executed.

The wheel includes the complete, hash-checked suite. A clone, dataset download, and the paper's internal validation history are unnecessary for evaluation. The separate research archive supports reanalysis of historical paper studies.

Source · Evaluation guide · Benchmark card · Citation

Code and suite are distributed under PolyForm Noncommercial 1.0.0.

Version archive DOI: 10.5281/zenodo.24044371.