Nvbenjo

Nvbenjo is a utility for benchmarking inference of deep learning models on NVIDIA GPUs. It supports models in Onnx format as well as PyTorch models, including torch.compile and ahead-of-time (AOT) compiled models. Nvbenjo generates comprehensive benchmark results including:

• CSV file with all measurement data (latency, throughput, memory usage, etc.)
• Plots visualizing the benchmark results

Demo

Installing

pip install nvbenjo

If you need a specific version of PyTorch or want to benchmark Onnx models adapt your install:

pip install torch torchvision --index-url https://download.pytorch.org/whl/cu130
pip install onnx onnxruntime-gpu
pip install nvbenjo

Usage

# Specify models to run in the command line
nvbenjo \
"+nvbenjo.models={\
    efficientnet: {type_or_path: 'torchvision:efficientnet_b0',  shape:['B',3,224,224],  batch_sizes: [16,32]},\
    resnet:       {type_or_path: 'torchvision:wide_resnet101_2', shape: ['B',3,224,224], batch_sizes: [16,32]}\
}"

# or better, specify your own config (or one of the pre-defined config files)
nvbenjo -cn small
nvbenjo -cn="/my/config/path/myconfig.yaml"

# override single arguments of your config
nvbenjo -cn="/my/config/path/myconfig.yaml" nvbenjo.models.mymodel.num_batches=10

# show current config and help
nvbenjo -cn="/my/config/path/myconfig.yaml" --help

Take a look at the Documentation for more information as well as example configurations.

Docker

The image bundles nvbenjo, its CUDA dependencies and nvbandwidth.

# build (the Onnx Runtime GPU backend is included by default)
docker build -t nvbenjo .

# run
mkdir -p outputs
docker run --rm -it --gpus all \
    -v "$PWD/outputs:/outputs" \
    -v nvbenjo-torch-cache:/root/.cache/torch \
    nvbenjo \
    nvbenjo -cn small

nvbandwidth is on the PATH as well:

docker run --rm --gpus all nvbenjo nvbandwidth

Development

Example using uv:

uv sync --group dev --extra onnx-cpu # or gpu
uv run nvbenjo

# for a quick run
uv run nvbenjo -cn small

# tests
uv run pytest
uv run nox

# docs
uv run --extra onnx-cpu sphinx-build -b html docs docs/_build/html
google-chrome docs/_build/html/index.html