simready-benchmark 2026.6.5

SimReady Benchmark, the engine-agnostic core for validating SimReady USD assets

simready-benchmark

Engine-agnostic Python framework for benchmarking SimReady USD assets. Discovers applicable tests, executes them across one or more engine sessions in parallel, and produces HTML and JSON reports.

Part of the SimReady Python Library Suite alongside simready-validate and simready-search.

---

Installation

pip install simready-benchmark          # framework + CLI only
pip install simready-benchmark[kit]     # + Kit / Isaac Sim engine plugin

The bundled FET test suite (simready-benchmark-kit-suite) is part of the simready_foundations repository. Refer to the install runbook in the package documentation for the setup.

Python 3.11 or newer is required.

---

Quick Start

# Plan + run + report on every asset in the project, in one pass
simready-benchmark --project-config /path/to/project_config.toml

# Or pass paths ad-hoc, scoped to one feature
simready-benchmark --sr-specs /path/to/sr_specs \
              --assets /path/to/asset.usd \
              --features FET003

Outputs land in _testing/ next to the project config (or the current directory if no config is set):

• plan.json. The resolved plan
• state/events.jsonl. Live JSON-Lines event stream
• results/<asset-dir>/<asset>.usd. Stamped mirror copy
• results/<asset-dir>/.simready/validation.json. runtime_tests[] receipt
• results/<asset-dir>/.simready/runtime/<test>/result.json. Per-test artifacts
• logs/<engine>_<n>.log. Per-engine subprocess logs
• index.html (HTML report at the output-dir root) plus report/test_results_index.json. Aggregated reports

The original asset directory is never modified.

---

Key Concepts

Plan / Run / Report Pipeline

A single simready-benchmark invocation runs three phases by default:

1. Plan. Scan installed test packs and the active sr_specs; match assets to applicable tests; write plan.json.
2. Run. Execute the plan across one or more engine sessions in parallel; stream JSON events; write per-test result.json.
3. Report. Aggregate per-test results into HTML and JSON reports.

A final stamp phase (default-on, suppress with --no-stamp) writes benchmark metadata into the mirror-copy USDs and merges a runtime_tests[] array into .simready/validation.json. Refer to the stamp-results skill.

Engines and Test Packs

Engines are registered through the simready_benchmark.engines entry-point group; test packs through simready_benchmark.tests. The framework discovers both at import time. No per-install configuration is needed for installed wheels.

Out of the box, simready-benchmark registers an nvcf engine plugin for NVIDIA Cloud Functions execution. simready-benchmark-engine-kit adds the kit engine plugin for Kit / Isaac Sim. Third-party engines and test packs slot in cleanly through the same mechanism. Refer to the write-test-pack skill.

Engine Configuration

Scaffold a starter engines.toml and edit the placeholders:

simready-benchmark --init-engines-toml             # writes ~/.simready-benchmark/engines.toml
simready-benchmark --init-engines-toml ./          # writes ./engines.toml (per-project)

The framework reads engine paths from engines.toml using a 4-tier lookup (first match wins):

1. --engines-toml <FILE> CLI flag
2. $SIMREADY_ENGINES_TOML env var (missing file is a hard error)
3. engines.toml in the current working directory
4. ~/.simready-benchmark/engines.toml

A [engines.toml] <path> line goes to stderr one time per process so the user can identify which file is in effect (suppress with SIMREADY_QUIET_ENGINES_TOML=1).

[kit.isaac_sim]
executable_path = "C:/Program Files/NVIDIA/IsaacSim/2025.1/isaac-sim.bat"
version = "5.1.0"
tags = ["isaac", "kit"]

# Optional: fixed output location (otherwise <CWD>/_testing).
[paths]
output_dir = "/path/to/_testing"

In the common case engines.toml carries only the engine. sr_specs and the on-disk test packs come from project_config.toml (set SIMREADY_PROJECT_CONFIG or pass --project-config). engines.toml can also carry [paths].sr_specs and [tests].paths, but those are overrides for when there is no project config or a machine needs a different value; resolution is first-non-empty-wins in the order CLI flag, environment variable, engines.toml, then project_config.toml.

Verify with simready-benchmark --show-config. This prints the resolved engines.toml, sr_specs source, project_config, and discoverable test packs with [ok]/[missing] markers.

---

CLI

simready-benchmark is a single command with mutually-exclusive mode flags rather than subcommands. The default invocation runs the full pipeline.

Mode Flags

Flag	Behavior
(default)	Plan + run + report in one pass
--plan-only	Generate plan.json and stop
--edit-plan ...	Edit an existing plan.json (no run, no report)
--resume	Run + report against an existing plan.json
--report-only	Regenerate the report from existing per-test result.json files
--list-tests	Print the registered features and tests
--show-config	Print the resolved configuration (engines.toml + tier, sr_specs source, test packs discoverable) and exit
--init-engines-toml [PATH]	Run the setup wizard and write a complete engines.toml. Overwrites an existing file by default; pass --no-overwrite to refuse

Plus the stamp subcommand:

simready-benchmark stamp --output-dir _testing

Common Flags

Flag	Description
--project-config <PATH>	Load content roots, sr_specs, profiles, features, and [tests].paths from project_config.toml
--sr-specs <PATH>	Path to sr_specs (overrides project config and engines.toml)
--assets <PATH> [<PATH>...]	One or more files or directories of assets
--engines-toml <FILE>	Override the engines.toml lookup chain
--tests-path <PATH> [<PATH>...]	Extra directories scanned for @test-decorated functions
--output-dir <DIR>	Override the default _testing/ location
--features <ID> [<ID>...]	Match feature ID prefixes; also bypasses the validation gate for matches
--tests <NAME> [<NAME>...]	Exact test names to keep
--max-concurrent <N>	Up to N engine sessions in parallel (default 2)
--workers <url1,url2,...>	NVCF worker endpoints (additive to local Kit engines)
--no-local-engines	NVCF-only mode (use ONLY explicitly-constructed engines)
--runtime <NAME> [<NAME>...]	Pin execution to one or more named engines from engines.toml (for example --runtime isaac_sim); unknown names hard-fail
--no-stamp	Skip the stamp phase after a successful run

Exit Codes

Code	Meaning
0	All tests passed
1	One or more tests failed, or a runtime error
2	CLI usage error
3	--resume / --report-only invoked but no plan.json or results found
4	Setup or environment not ready (no engine could run part of the plan, or an engine failed preflight). 4 outranks 1.
130	Interrupted by user (Ctrl-C)

---

Python API (Advanced)

The CLI is the recommended entry point. The Python surfaces are low-level and intended for advanced integrations:

• simready_benchmark.Planner. Static-method class; Planner.plan(sr_specs_path, test_dirs, discovery, validation_source, ...) -> Plan.
• simready_benchmark.Runner. Static-method class; Runner.start(plan, output_dir, ...) -> RunHandle.
• simready_benchmark.Reporter. Static-method class; Reporter.generate(output_dir, fmt="both") -> dict.
• simready_benchmark.PlanEditor. Wraps a Plan; appends overrides. Load and save through simready_benchmark.core.models.load_plan / save_plan.
• simready_benchmark.test. The @test decorator. Functions MUST be async def. Required kwargs: features, requirement, name, description, expected_video, version.
• simready_benchmark.RunContext, simready_benchmark.SceneHandle. Runtime types used by tests.

Refer to the ten packaged skills under src/simready_benchmark/skills/ for ready-to-use recipes: configure-engines-toml, run-tests, edit-plan, stamp-results, embed-in-pipeline, diagnose-failures, remediate-assets, write-test-pack, write-engine-plugin, and write-online-runner.

---

Metadata Ownership

simready.test owns only the runtime_testing sub-key of SimReady_Metadata in USD customLayerData, and the runtime_tests[] array in .simready/validation.json. It never reads or writes asset_id, validation, or any other section.

customLayerData["SimReady_Metadata"]
├── asset_id          <- simready.create
├── validation        <- simready.validate
└── runtime_testing   <- simready.test

.simready/validation.json
├── schema_version, profile, passed, issues, ...   <- simready.validate
└── runtime_tests[]                                 <- simready.test

---

License

Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. SPDX-License-Identifier: Apache-2.0