simready-benchmark

Engine-agnostic Python framework for benchmarking SimReady USD assets. Discovers applicable tests, executes them across one or more engine sessions in parallel, and produces HTML and JSON reports.

Part of the SimReady Python Library Suite alongside simready-validate and simready-search.

---

Requirements

• Python 3.12 exactly. The supported engine, NVIDIA Isaac Sim, requires Python 3.12, so the virtual environment must use 3.12.
• A 64-bit x86 machine (x86_64, also called AMD64) running Windows 10 or 11, or Linux (Ubuntu). usd-core, a core dependency, publishes no ARM (aarch64) wheels, so pip install fails while resolving usd-core on ARM machines, including ARM cloud instances. On Linux, confirm your architecture with uname -m, which must report x86_64.
• A CUDA-capable NVIDIA GPU and about 30 GB of free disk, primarily for Isaac Sim, the engine that runs the tests.

---

Installation

Always work inside a dedicated virtual environment, because the framework and the Isaac packages can conflict with other USD or Omniverse installations.

Windows

mkdir C:\simready
cd C:\simready
python -m venv .venv
.\.venv\Scripts\Activate.ps1
pip install "simready-benchmark[kit]"

If PowerShell reports that running scripts is disabled, run Set-ExecutionPolicy -ExecutionPolicy RemoteSigned -Scope CurrentUser one time, then retry the activate command.

Linux (Ubuntu)

Ubuntu 22.04 ships Python 3.10 and has no python3.12 package, so add the deadsnakes PPA first:

sudo apt install -y software-properties-common
sudo add-apt-repository -y ppa:deadsnakes/ppa
sudo apt update
sudo apt install -y python3.12 python3.12-venv

Then create the environment and install:

mkdir -p ~/simready
cd ~/simready
python3.12 -m venv .venv
source .venv/bin/activate
python -m pip install --upgrade pip
pip install "simready-benchmark[kit]"

Do not install python-is-python3 as a substitute; it only aliases python to the system Python (3.10 on Ubuntu 22.04), which Isaac Sim rejects.

Verify

simready-benchmark --version

These commands respect the default index, mirror, and credentials already configured for pip. To install a local repository build, run repo.bat python_package from the simready-explorer root and install the simready_benchmark-*.whl and simready_benchmark_engine_kit-*.whl files from _build/packages/dist/.

The [kit] extra pulls in simready-benchmark-engine-kit, the Kit and Isaac Sim engine plugin, along with usd-core, numpy, and Pillow. Install simready-benchmark without the extra for the engine-agnostic core and CLI only.

Specification tiers may also be installed as Python packages:

pip install simready-foundation-tier-core

Benchmark discovers their catalogs and optional runtime tests through the simready.tier descriptor. Independent test-only wheels use simready_benchmark.tests. An explicit Foundation checkout remains available for source-tree development:

simready-benchmark --foundations-path /path/to/simready_foundations --list-tests

This discovers the checkout's tier catalogs and runtime-test package. The flag applies to that command only; it is not required for an installed-tier workflow.

To use the same checkout on every command without repeating --foundations-path, create or update ~/.simready-benchmark/engines.toml (%USERPROFILE%\.simready-benchmark\engines.toml on Windows) and persist the checkout's project configuration:

[paths]
project_config = "<foundations>/sample_content/project_config.toml"

Use forward slashes in the TOML path on Windows. Benchmark then obtains the checkout's content root, tier catalogs, and tier-owned runtime tests from that single project configuration, so subsequent commands need neither --foundations-path nor --project-config:

simready-benchmark --show-config
simready-benchmark --list-tests
simready-benchmark --assets /path/to/assets

SIMREADY_PROJECT_CONFIG provides the same project-config selection through an environment variable. Explicit CLI arguments take precedence over environment and engines.toml defaults.

---

Setup

Install a tier with its Benchmark extra, then validate the environment:

pip install "simready-foundation-tier-core[benchmark]"

The tier advertises catalogs and its optional runtime_tests_path through simready.tier. Independent test-only distributions use simready_benchmark.tests. Setup checks those installed providers and the active Python 3.12 environment. Do not install pip Isaac beside Benchmark's usd-core: Isaac's converter dependencies bring usd-exchange, and the two distributions install overlapping pxr files. --setup --install-isaac refuses that unsafe combination so it cannot silently break usd-validation-nvidia.

Create a separate Python 3.12 venv containing only pip Isaac, then point engines.toml at its isaacsim launcher and Python executable:

[kit.isaac_sim]
executable_path = "/path/to/isaac-env/bin/isaacsim"
python_executable = "/path/to/isaac-env/bin/python"
version = "6.1.0.0"
tags = ["isaac", "kit"]
runtime = "PhysX"

Add a second stanza with runtime = "Newton" and experience = "isaacsim.exp.full.newton" for Newton. On Windows, use the environment's Scripts/isaacsim.exe and Scripts/python.exe. See the installation guide's Separate Isaac environment section for complete commands and configuration.

Setup creates no source checkout, does not persist --foundations-path, and does not create a configuration file. A source-checkout-only workflow does not use --setup to register the checkout; select it per command with --foundations-path, or persist its sample_content/project_config.toml under [paths] in engines.toml as shown above.

When upgrading from the retired simready-benchmark-kit-suite or simready-foundation-runtime-tests-kit distributions, uninstall both and then force-reinstall simready-foundation-tier-core. The old and new distributions otherwise claim the same test-package files; --setup rejects that ambiguous state with an actionable migration message.

Pip Isaac in the active environment is discovered automatically as two runtimes: isaac_sim for PhysX and isaac_sim_newton for Newton. Use engines.toml for the recommended separate Isaac environment, a custom/standalone executable, custom experience, or remote worker.

Verify the Configuration

simready-benchmark --show-config

This prints installed tier catalogs, runtime-test packs, automatic engines, and any explicit project or engines.toml override.

---

First Test

List the discovered tests, then run the full pipeline (plan, run, and report) against one demo asset. With an installed tier, or after persisting the Foundation project config in engines.toml, run:

simready-benchmark --list-tests
simready-benchmark --assets <foundations>/sample_content/common_assets/props_general/apple_a01

For an unconfigured source checkout, include --foundations-path on both commands:

simready-benchmark --foundations-path <foundations> --list-tests
simready-benchmark --foundations-path <foundations> \
  --assets <foundations>/sample_content/common_assets/props_general/apple_a01

Replace <foundations> with your clone path, for example <drive>:\path\to\simready_foundations on Windows or /path/to/simready_foundations on Linux. Isaac Sim launches and the applicable tests run against the apple asset. Benchmark prints the allocated run directory and the exact HTML report path when it finishes. Open that printed index.html; do not assume a fixed _testing directory because an existing requested directory causes Benchmark to allocate a fresh benchmark_testing[_N] child.

The exit code is 0 when every test passes and 1 when any test fails. Refer to Exit Codes for the full list.

Known Issues (Isaac Sim First Run)

• The first run on a machine where Isaac Sim has never launched can be very slow or appear to fail until Isaac Sim compiles its shader cache. Warm the cache once: launch Isaac Sim directly, wait for the main viewport, open any USD file (the sample apple works), then close it. Re-run the benchmark. After one successful run the cache is warm and later runs complete normally.
• A test can fail with an entirely white (blank) comparison frame. Isaac Sim and Kit sometimes do not render a frame in time, which fails the comparison. This is an engine rendering issue, not an asset defect. Re-run only the failed tests with simready-benchmark --rerun failed; if they pass on the second run, the asset is good. If your original run used a custom --output-dir or --project-config, pass the same flag to --rerun failed so it finds the results.
• The separate Isaac Sim install stalls or fails on a network read timeout. Re-run the pip install with --timeout 120 --retries 5 inside the Isaac venv. Do not install it into the validation venv as a retry.
• An installed provider or Isaac is missing. Run simready-benchmark --show-config. If Isaac is missing, verify the separate venv's executable_path and python_executable entries in engines.toml.

---

Day-to-Day Use

The virtual environment is per-terminal. In every new terminal, activate it before running the tool:

# Windows
cd C:\simready
.\.venv\Scripts\activate

# Linux
cd ~/simready
source .venv/bin/activate

To refresh the test content and check for engine and framework upgrades, run simready-benchmark --update.

---

Output Layout

The requested output location resolves in this order: --output-dir, engines.toml [paths].output_dir, <project_root>/_testing, then <cwd>/_testing. A new run allocates a fresh benchmark_testing[_N] child when that requested location already exists. The resolved directory contains:

• plan.json. The resolved plan
• run_summary.json. Terminal run, readiness, and failure summary
• state/events.jsonl. Live JSON-Lines event stream
• state/work_pool.json. Work-item and per-test execution state
• results/<asset-dir>/<asset>. Root-layer report copy; writable layers are stamped, while USDZ packages remain byte-identical
• results/<asset-dir>/.simready/validation.json. Wrapped SimReady_Metadata.runtime_testing receipt
• results/<asset-dir>/.simready/runtime/<test-output-name>/<engine>/result.json. Unique tests keep their raw name; provider qualification and a deterministic digest make overlapping names collision-safe. Per-engine test result and media artifacts
• logs/<engine>_<n>.log. Per-engine subprocess logs
• index.html plus assets/<slug>.html (HTML index and per-asset details) and report/test_results_index.json. Aggregated reports

The original asset directory is never modified. This is intentional: source assets may be read-only, remote, version controlled, or require a separate checkout/publish workflow before they can be changed.

The results tree mirrors the source asset's relative path to prevent output name collisions; it does not mirror the complete asset directory. Stamping copies only the root USD layer and writes the benchmark receipt beside it. Referenced layers, payloads, materials/, textures/, and other relative dependencies are not copied or rewritten. Consequently, the stamped USD under results/ is a report/metadata artifact and may not open with complete composition or visual fidelity. Open the original asset for inspection, or use an authorized downstream packaging/publishing step if a portable stamped asset is required.

Benchmark never clears the requested output directory. It creates and marks a fresh exclusive run directory; when the requested directory already exists (including a filesystem or drive root), it writes only to a benchmark_testing[_N] child. It never writes run artifacts directly into the root or clears existing content. Read the resolved output and report paths from the final CLI events. The deprecated --keep-outputs flag is accepted for compatibility but does not change the always-preserve behavior.

---

Key Concepts

Plan / Run / Stamp / Report Pipeline

A single simready-benchmark invocation runs four phases by default:

1. Plan. Scan installed test packs and the active sr_specs; match assets to applicable tests; write plan.json.
2. Run. Execute the plan across one or more engine sessions in parallel; stream JSON events; write per-test result.json.
3. Stamp. Unless suppressed with --no-stamp, write matching benchmark metadata into writable root-layer mirrors and .simready/validation.json. USDZ mirrors are byte-identical and sidecar-only. These copies are receipts, not self-contained asset packages.
4. Report. Aggregate per-test results and stamp information into HTML and JSON reports.

Engines and Test Packs

Engines are registered through simready_benchmark.engines. Independent test packs use simready_benchmark.tests; tier-owned packs are advertised through a simready.tier descriptor's optional runtime_tests_path. No per-install path configuration is needed for installed providers.

Out of the box, simready-benchmark registers an experimental nvcf engine plugin for NVIDIA Cloud Functions execution. The NVCF plugin, remote-worker path, and local emulator are Technology Preview features and are not the supported release-validation path. They require a separately provisioned compatible worker endpoint and credentials; Benchmark does not provide an NVCF deployment, worker URL, API key, or test account. simready-benchmark-engine-kit adds the supported kit engine plugin for Kit and Isaac Sim. Third-party engines and test packs slot in through the same mechanism.

Engine Configuration

Pip Isaac in the active environment is discovered automatically as isaac_sim (PhysX) and isaac_sim_newton (Newton). No config file is needed.

An optional engines.toml overrides automatic local discovery for a standalone/custom runtime. The framework reads it using a 4-tier lookup (first match wins):

1. --engines-toml <FILE> CLI flag
2. $SIMREADY_ENGINES_TOML environment variable (a missing file is a hard error)
3. engines.toml in the current working directory
4. ~/.simready-benchmark/engines.toml

The run output uses its own precedence: --output-dir, then [paths].output_dir from the resolved engines.toml, then <project_root>/_testing, and finally <cwd>/_testing. Benchmark allocates a fresh benchmark_testing[_N] child when the requested location already exists.

For a standalone install, two engine configurations can share one executable and differ only in the Kit experience:

[kit.isaac_sim]
executable_path = "C:/simready/.venv/Scripts/isaacsim.EXE"
version = "6.0.1.0"
tags = ["isaac", "kit"]

[kit.isaac_sim_newton]
executable_path = "C:/simready/.venv/Scripts/isaacsim.EXE"
version = "6.0.1.0"
tags = ["isaac", "kit"]
experience = "isaacsim.exp.full.newton"

# Optional: fixed output location (otherwise <CWD>/_testing).
[paths]
output_dir = "/path/to/_testing"

Select an engine per run with --runtime: --runtime isaac_sim_newton runs Newton only, --runtime isaac_sim runs PhysX only, and a run with no --runtime runs every eligible test on all configured engines. Runtime-specific physics tests are eligible only when the asset validation record contains the matching feature variant, such as FET_003_NEWTON or FET_003_PHYSX; FET_003_STANDARD alone does not authorize either concrete backend when the test has runtime-specific variants. Runtime-neutral tests can still run on both engines. The report attributes each result to the feature matching its engine.

To launch the raw Isaac Sim application in Newton mode, without the benchmark, pass the experience as the first launcher argument: isaacsim.EXE isaacsim.exp.full.newton (isaacsim without the .EXE on Linux).

---

CLI

This README summarizes common operations. The project documentation's docs/cli-reference.md contains every public option and subcommand. The exact help for an installed build is available with simready-benchmark --help and simready-benchmark <subcommand> --help.

The default invocation runs the full pipeline. Mutually exclusive mode flags select alternate top-level workflows, and four subcommands provide staging, stamping, and preview-emulator operations.

Mode Flags

Flag	Behavior
(default)	Plan, run, and report in one pass
--plan-only	Generate plan.json and stop
--edit-plan ...	Edit an existing plan.json (no run, no report)
--rerun {plan,failed,report}	Operate on an existing run: plan runs and reports against the existing plan.json, failed re-runs only the tests that failed in a previous run, report regenerates the report from existing per-test result.json files
--list-tests	Print the registered features and tests
--show-config	Print the resolved configuration and exit
--setup	Verify installed tiers, runtime-test packs, and pip Isaac; add --install-isaac to install Isaac when absent
--update	Check pip Isaac for an update and report the Benchmark upgrade command

Subcommands are stamp, stamp-one, stage, and the Technology Preview nvcf-emulator. For example:

simready-benchmark stamp --output-dir _testing

Common Flags

Flag	Description
--project-config <PATH>	Load content roots, sr_specs, profiles, features, and [tests].paths from project_config.toml
--foundations-path <DIR>	Load tier catalogs and runtime tests from a simready_foundations source checkout; cannot be combined with --project-config
--sr-specs <PATH>	Path to sr_specs (overrides project config and engines.toml)
--assets <PATH> [<PATH>...]	One or more files or directories of assets
--engines-toml <FILE>	Override the engines.toml lookup chain
--tests-path <PATH> [<PATH>...]	Extra directories scanned for @test-decorated functions
--output <FILE>	Override the plan path; normally unnecessary because --output-dir writes <DIR>/plan.json automatically
--output-dir <DIR>	Override the default _testing/ location
--features <ID> [<ID>...]	Match feature ID prefixes; also bypasses the validation gate for matches
--tests <SELECTOR> [<SELECTOR>...]	Raw names keep every provider match; provider:name selects one exact provider
--max-concurrent <N>	Up to N engine sessions in parallel (default 2)
--session-mode {batch,per_asset}	Reuse sessions across assets, or start a fresh session for each asset
--workers <url1,url2,...>	Technology Preview: separately provisioned NVCF worker endpoints (additive to local Kit engines)
--no-local-engines	Skip local engines; with --workers, selects the Technology Preview NVCF-only path
--runtime <NAME> [<NAME>...]	Pin execution to one or more discovered engines (for example isaac_sim_newton or isaac_sim); with no flag every test runs on all matching engines. Unknown names hard-fail
--no-stamp	Skip the stamp phase after a successful run
--keep-outputs	Deprecated compatibility flag; existing output is always preserved
--format {html,json,both}	Select report output; default both

--setup accepts --install-isaac or --no-install-isaac. Run simready-benchmark --help for the complete list.

Exit Codes

Code	Meaning
0	All tests passed
1	One or more tests failed, or a runtime error
2	CLI usage error
3	A requested existing plan or result set was missing (--rerun plan, --rerun report, or --rerun failed)
4	Setup or environment not ready (no engine could run part of the plan, or an engine failed preflight). 4 outranks 1.
130	Interrupted by user (Ctrl-C)

---

Python API (Advanced)

The CLI is the recommended entry point. The Python surfaces are low-level and intended for advanced integrations:

• simready_benchmark.Planner. Static-method class; Planner.plan(sr_specs_path, test_dirs, discovery, validation_source, ...) -> Plan.
• simready_benchmark.Runner. Static-method class; Runner.start(plan, output_dir, ...) -> RunHandle.
• simready_benchmark.Reporter. Static-method class; Reporter.generate(output_dir, fmt="both") -> dict.
• simready_benchmark.PlanEditor. Wraps a Plan; appends overrides. Load and save through the public Planner.load and Planner.save methods.
• simready_benchmark.test. The @test decorator. Functions MUST be async def. Required keyword arguments: features, name, description, expected_video, and version. Optional: engine, config_defaults, max_duration (default 300), and enabled (default True).
• simready_benchmark.RunContext, simready_benchmark.SceneHandle. Runtime types used by tests.
• simready_benchmark.EngineConfig, EnginePlugin, EngineSession, and ValidationResult. Public engine-plugin contracts.

The package ships six skills under src/simready_benchmark/skills/ with ready-to-use recipes: install-environment, run-tests, embed-in-pipeline, diagnose-failures, write-test-pack, and write-engine-plugin.

---

Metadata Ownership

simready.test owns only the runtime_testing sub-key of SimReady_Metadata in USD customLayerData and .simready/validation.json. It never reads or writes asset_id, validation, or any other section.

customLayerData["SimReady_Metadata"]
├── asset_id          <- simready.create
├── validation        <- simready.validate
└── runtime_testing   <- simready.test

.simready/validation.json
├── schema_version, profile, passed, issues, ...   <- simready.validate
└── SimReady_Metadata.runtime_testing.tested_features
                                                    <- simready.test

---

License

Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. SPDX-License-Identifier: Apache-2.0