turbobench-cli 2.0.6

⚖️ Matched environments. Measured fairly. ⚖️

turbobench is a local Python CLI for reinforcement-learning environment authors, researchers, and provider maintainers who need fair performance comparisons between compatible implementations and standardized cross-provider parity checks. Each immutable workload profile contains both its performance settings and its parity contract, so the same profile ID drives both operations. Runs produce portable, self-verifying evidence.

Optional comparison videos replay the same locked providers and semantic action trajectory. Only valid, conclusive evidence can produce unmarked promotional media; diagnostic output is clearly watermarked.

Install

Install turbobench-cli 2.0.6 from PyPI:

uv tool install turbobench-cli==2.0.6

Alternatively, install it in an active virtual environment with python -m pip install turbobench-cli==2.0.6. The installed command and Python import remain turbobench.

For a development checkout:

git clone https://github.com/tsilva/turbobench.git
cd turbobench
uv sync --frozen --group dev

Run turbobench profiles list and turbobench providers list to choose a compatible profile and provider pair. Prefix CLI commands with uv run when working from a development checkout.

Commands

turbobench doctor vizdoom/basic-v1       # check the host, tools, and profile assets
turbobench profiles list                 # list immutable workloads
turbobench providers list                # list built-in and registered providers

turbobench parity supermario/world1-v1 \
  --candidate env-supermariobrosnes-turbo-emu@checkout:/absolute/path/to/repo \
  --allow-dirty --quick                         # test current work diagnostically

turbobench parity vizdoom/basic-v1 \
  --candidate env-vizdoom-turbo@artifact:/absolute/path/to/final.whl \
  --output turbobench-parity/vizdoom            # certify the exact final wheel

turbobench verify-parity turbobench-parity/vizdoom --require-canonical

turbobench compare vizdoom/basic-v1 \
  --left env-vizdoom-turbo@1.3.0.post27 \
  --right vizdoom@1.3.0 \
  --output turbobench-results/vizdoom            # create a result bundle

turbobench compare breakout/start-v1 \
  --left env-breakoutatari2600-turbo-native@VERSION \
  --right stable-retro@1.0.1 \
  --output turbobench-results/breakout-vs-stable-retro

turbobench compare breakout/start-v1 \
  --left env-breakoutatari2600-turbo-native@VERSION \
  --right env-stableretro-turbo@VERSION \
  --output turbobench-results/breakout-vs-stable-retro-turbo

# Add one receipt for an authority/candidate pair, or two receipts against the
# same authority to reuse transitive Turbo-versus-Turbo correctness evidence.
turbobench compare breakout/start-v1 \
  --left env-breakoutatari2600-turbo-native@VERSION \
  --right env-stableretro-turbo@VERSION \
  --parity-receipt /external/evidence/native-receipt \
  --parity-receipt /external/evidence/stable-retro-turbo-receipt

turbobench verify turbobench-results/vizdoom  # verify integrity and consistency
turbobench report turbobench-results/vizdoom  # print the generated report
turbobench promo turbobench-results/vizdoom --diagnostic

uv run --frozen ruff check .                    # lint the project
uv run --frozen pytest -m "not acceptance"      # run tests without proprietary assets

Long-running commands write progress to standard error and reserve standard output for their final machine-readable JSON.

Notes

• The controller supports Python 3.11 and newer. Provider runtimes default to CPython 3.14. uv, FFmpeg, and FFprobe are required.
• Current workload profiles are supermario/world1-v1, breakout/start-v1, and vizdoom/basic-v1. The same ID is used with compare and parity. compare --quick performs selected-pair correctness and two shape-1 timing pairs. Full compare starts with those same two shape-1 pairs, continues to seven pairs, and then measures shapes 16 and 32. It records the first two pairs as light statistics.
• Provider references accept provider, provider@latest, provider@VERSION, provider@artifact:/absolute/path.whl, and provider@checkout:/absolute/path. latest excludes prereleases, yanked releases, incompatible artifacts, and releases still inside the seven-day quarantine.
• Set TURBOBENCH_ROM_PATH, TURBOBENCH_ASSET_ROOT, or RETRO_DATA_PATH to locate required local game payloads. ROMs and local paths are never written to portable bundles; only canonical digests are recorded.
• The breakout/start-v1 profile compares the Atari 2600 Breakout Start workload against either original Stable Retro or Stable Retro Turbo. Replace VERSION with an exact release, use @latest, or select a clean checkout with @checkout:/absolute/path.
• Every official result must pass provider compatibility, matched correctness, system-load, alternating paired-measurement, statistical uncertainty, provenance, and asset gates. Quick runs and explicit overrides remain diagnostic.
• Exact release parity accepts only the final local wheel on the chosen canonical host. Checkout snapshots include tracked edits and nonignored untracked source, so developers do not need to commit before running a quick diagnostic check.
• Comparisons validate only their selected pair and never prepare an original authority unless that authority is itself selected. Compatible parity receipts may replace matching correctness shapes; missing shapes are checked directly during the comparison.
• Turbo providers are preflighted against the normative Turbo Vector API v2 contract. Contract reports are hash-bound into result bundles; malformed v2 providers stop before any workload, while historical v1 providers remain diagnostic-only.
• Result bundles contain the exact provider lock, shape-local statistics, report, chart, raw evidence, verification records, and optional media. manifest.json binds every portable file by size and SHA-256; turbobench does not upload or publish bundles.
• Official v1 hosts are Apple-silicon macOS and x86-64 Linux. Third-party providers can register through the turbobench.providers entry-point group.

Architecture

License

MIT