turbobench-cli 1.0.0

⚖️ Matched environments. Measured fairly. ⚖️

turbobench is a local Python CLI for reinforcement-learning environment authors, researchers, and provider maintainers who need fair performance comparisons between compatible implementations. It verifies that two providers produce matching transitions before timing them, then writes a portable, self-verifying evidence bundle. Run it with a built-in workload profile and two provider references.

Optional comparison videos replay the same locked providers and semantic action trajectory. Only valid, conclusive evidence can produce unmarked promotional media; diagnostic output is clearly watermarked.

Install

Install the published CLI from PyPI with uv tool install turbobench-cli or pip install turbobench-cli. The installed command and Python import remain turbobench.

For a development checkout:

git clone https://github.com/tsilva/turbobench.git
cd turbobench
uv sync --frozen --group dev

Run uv run turbobench profiles list and uv run turbobench providers list to choose a compatible profile and provider pair.

Commands

uv run turbobench doctor vizdoom/basic-v1       # check the host, tools, and profile assets
uv run turbobench profiles list                 # list immutable workloads
uv run turbobench providers list                # list built-in and registered providers

uv run turbobench compare vizdoom/basic-v1 \
  --left vizdoom-turbo@1.3.0.post23 \
  --right vizdoom@1.3.0 \
  --output turbobench-results/vizdoom            # create a result bundle

uv run turbobench verify turbobench-results/vizdoom  # verify integrity and consistency
uv run turbobench report turbobench-results/vizdoom  # print the generated report
uv run turbobench promo turbobench-results/vizdoom --diagnostic

uv run --frozen ruff check .                    # lint the project
uv run --frozen pytest -m "not acceptance"      # run tests without proprietary assets

Long-running commands write progress to standard error and reserve standard output for their final machine-readable JSON.

Notes

• The controller supports Python 3.11 and newer. Provider runtimes default to CPython 3.14. uv, FFmpeg, and FFprobe are required.
• Built-in profiles cover supermario/canonical-v1, breakout/start-v1, and vizdoom/basic-v1. Shapes 1, 16, and 32 are measured and reported independently.
• Provider references accept provider, provider@latest, provider@VERSION, and provider@checkout:/absolute/path. latest excludes prereleases, yanked releases, incompatible artifacts, and releases still inside the seven-day quarantine.
• Set TURBOBENCH_ROM_PATH, TURBOBENCH_ASSET_ROOT, or RETRO_DATA_PATH to locate required local game payloads. ROMs and local paths are never written to portable bundles; only canonical digests are recorded.
• Every official result must pass provider compatibility, matched correctness, system-load, alternating paired-measurement, statistical uncertainty, provenance, and asset gates. Quick runs and explicit overrides remain diagnostic.
• Result bundles contain the exact provider lock, shape-local statistics, report, chart, raw evidence, verification records, and optional media. manifest.json binds every portable file by size and SHA-256; turbobench does not upload or publish bundles.
• Official v1 hosts are Apple-silicon macOS and x86-64 Linux. Third-party providers can register through the turbobench.providers entry-point group.

Architecture

License

MIT