Benchmarking Report

A benchmarking report is a standard data format describing the cluster configuration, workload, and results of a benchmark run. The report acts as a common API for different benchmarking experiments. Each supported harness in llm-d-benchmark creates a benchmark report upon completion of a run, in addition to saving results in its native format.

There are two lines of the benchmark report, 0.1 and 0.2. All reports are generated by the llm-d-benchmark harness pod, but new applications consuming benchmark data should use the 0.2 line, whose current revision is 0.2.1. Reports that declare 0.2 were produced before that revision and still load: every valid 0.2 report is also a valid 0.2.1 report.

Installation

The library is distributed on PyPI as llmd-benchmark-report:

pip install llmd-benchmark-report

The canonical import name is llmd_benchmark_report. Two aliases exist for backwards compatibility and will be removed in a future release: the bare benchmark_report name (shipped in this distribution, used historically by the harness pod scripts) and llmdbenchmark.analysis.benchmark_report (a shim in the llmdbenchmark package). Both alias the canonical modules with identity preserved.

Versioning and compatibility

llmd-benchmark-report package versions track the Benchmark Report schema version: package 0.2.x implements BR v0.2.x (and still parses earlier report versions). Within a minor line, releases are strictly additive: new optional fields and new component kinds may be added, but existing fields are never removed, renamed, retyped, or given stricter validation that rejects previously valid reports. A report valid against 0.2.x validates against every later 0.2.y. Breaking changes require a new minor line. Each minor line has a single implementation: schema_v0_2.py and native_to_br0_2.py hold the latest 0.2.x revision, and a new revision changes them in place rather than adding modules. Packaging-only fixes with no schema change ship as post-releases (e.g. 0.2.1.post1). Consumers should depend on llmd-benchmark-report>=0.2.1,<0.3.

Releasing

Releases are published to PyPI by the br-release GitHub workflow, triggered by pushing a br-v* tag (e.g. br-v0.2.1) whose version matches version in this directory's pyproject.toml; the workflow fails on a mismatch. Publishing uses PyPI trusted publishing (OIDC), so no credentials are stored in the repository.

v0.2.x Format Description

A benchmark report describes the inference service configuration, workload, and performance results. Individual traces from single inference executions are not captured, but time-series or statistical results may be captured.

See br_v0_2_example.yaml for a dummy example report (values may be non-sensical). A JSON Schema for the current 0.2.x revision is at br_v0_2_json_schema.json.

The top-level fields for the benchmark report are version, run, scenario, and results.

version Field

The version field (0.2.1 for reports produced by this package, 0.2 for reports produced before that revision) indicates the schema revision for the benchmark report. Different versions may add or drop certain details, so there is no guarantee of "lossless" conversion between benchmark report versions.

run Field

The run field contains details about the benchmark run. Only the unique ID run.uid is required, with other fields including start/end times and duration, user name, and additional IDs for the experiment (to group benchmark reports), cluster, and pod.

Two fields are populated from the experiment ID (LLMDBENCH_RUN_EXPERIMENT_ID, or the results directory name for an archived run):

• run.eid groups every report of one experiment, as a UUID5 of the experiment ID. Stages of the same run share it; treatments of a sweep do not.
• run.description is the human-readable run label. Consumers such as llm-d-prism read it as the run label, so it defaults to <model> [<experiment id>] (for example Qwen/Qwen3-32B [inference-perf-conc32-1786024743-hipkpq]): the ID keeps the treatments of a sweep from collapsing into one entry, and the model keeps the label readable. When no experiment ID is available at all, both fields are omitted.

The schema documents run.description as submitter-provided, so the generated label is only a default. Set your own with --run-description, LLMDBENCH_DESCRIPTION_TEXT, or description.text under a scenario's common: block, and it is used verbatim. A submitter-supplied description and keywords are recorded even when no experiment ID resolves, since neither needs one to be meaningful.

run.keywords is populated only from --run-keywords, LLMDBENCH_DESCRIPTION_KEYWORDS, or description.keywords (comma-separated). It is never auto-populated -- see SUBMISSION_POLICY.md -- and the key is omitted entirely when the submitter sets nothing.

scenario Field

scenario describes the inference stack configuration (under scenario.stack) and workload details (under scenario.load). These are the inputs to a benchmark experiment, and ideally will be sufficiently populated to ensure the benchmark is reproducible and leaves no question as to what exactly is being measured.

scenario.stack consists of a list of components such as inference engine instances (vLLM, SGLang...), routers, and any other part of the system used to support the inference endpoint. No specific component is required, so this may be customized to the needs of a particular experiment.

Each component has a metadata, standardized, and native field. The native subsection can store any arguments, environment variables, and configuration details in the native format of a particular component. The standardized subsection stores details in a single standardized schema for that component, regardless of the specific vendor or version used. For example, an inference engine component could be vLLM, SGLang, etc. but they will always certain configuration options in common albeit defined in different ways. The standardized subsection for an inference engine contains common details like LLM model name, accelerator hardware model and count, and parallelisms used.

The workload details under scenario.load follow the same methodology as components, with metadata, standardized, and native subsections.

results Field

The results field contains outputs from a benchmarking experiment. This may include any one of request-level performance metrics, session-level performance metrics, observability data, stack component health, or profiling.

results.request_performance

Aggregate statistics for individual inference requests: latencies (TTFT, TPOT, ITL, E2E), throughput (input/output/total token rates, request rate), and request counts.

results.session_performance

Session-level statistics for multi-turn workloads. Populated from *_session_lifecycle_metrics.json files produced by inference-perf when multi-turn sessions are enabled. The session_performance.sessions object contains:

A session_performance report is generated alongside the standard request_performance report for each stage that has a corresponding session lifecycle file. The scenario.load.standardized.stage field identifies which stage the report covers, and scenario.load.standardized.multi_turn.enabled is set to true.

Additions in 0.2.1

Revision 0.2.1 adds optional multi-modal payload statistics for image, video, and audio workloads, plus optional treatment-grouping metadata on scenario.load.metadata (treatment, treatment_group, concurrent_with). concurrent_with names the treatments that ran at the same time against the same endpoint: when it is non-empty the metrics reflect contention from those workloads rather than this one in isolation. Every field introduced is optional, so any 0.2 report validates unchanged under 0.2.1 (enforced by tests/test_benchmark_report_v0_2_legacy.py).

The multi-modal additions, all derived from what the client can determine from the payloads it sent, are:

• results.request_performance.aggregate.requests.request_size (Statistics): total encoded request size in bytes, capturing the large payloads typical of multi-modal requests.
• results.request_performance.aggregate.requests.multimodal: a per-modality block (image, video, audio), each a distribution set over the media instances sent. image/video carry count, filesize, pixels, and aspect_ratio; video adds frames; audio carries count, filesize, and duration.
• results.request_performance.aggregate.throughput.{image,video,audio}_rate (Statistics): per-modality delivery rates (images/s, videos/s, audios/s).

New unit categories back these fields: pixels (quantity), ratio (for aspect ratio, distinct from a 0..1 portion), bytes (memory), and a media-throughput category (images/s, videos/s, audios/s) kept separate from the request-rate category so the existing request_rate guardrail is unaffected.

Server-side multi-modal metrics (vision token counts, encoding time, multimodal cache hit rates) are out of scope for this revision.

v0.1 Format Description

A benchmark report describes the inference service configuration, workload, and aggregate results. Individual traces from single inference executions are not captured, rather statistics from multiple traces of identical scenarios are combined to create a report.

See br_v0_1_example.yaml for a dummy example report (values may be non-sensical). The JSON Schema for the benchmark report version 0.1 is at br_v0_1_json_schema.json. The report has three top-level fields, version, scenario, and metrics.

While each of these fields is required, some subfields may be optional or not apply to the specific benchmark being performed. For example, some metrics may not be captured or supported by a certain benchmarking toolset. In cases where one desires to capture information that is not part of the standard benchmark report schema, a metadata field may be placed almost anywhere under scenario or metrics to add arbitrary data.

version Field

The version field (0.1) indicates the schema revision for the benchmark report. Different versions may add or drop certain details, so there is no guarantee of "lossless" conversion between benchmark report versions.

scenario Field

The scenario describes precisely what was measured. This includes details about the inference platform (full stack, including versions and the important runtime arguments), cluster configuration (like GPUs and parallelism utilized), and workload. The content in this field should be detailed enough that it is sufficient to launch a repeat benchmarking experiment that will yield similar results (within some reasonable bound of variability).

metrics Field

The metrics field contains all of the results for the report. This does not include individual trace details, rather statistics for all runs that were captured in benchmarking for a particular scenario. This includes request-level performance metrics (like latencies and throughput), details about the inference service (like request queue lengths and KV cache size), and hardware metrics (such as GPU compute and memory utilization). Some of the underlying fields, such as the hardware metrics, more traditionally fall in the category of "observability" rather than "benchmarking".

Implementation and Usage

The schema for a benchmarking report is defined through Python classes using Pydantic in schema_{version}.py. Instantiating an instance of a benchmark report class includes various checks, such as ensuring compliance with the schema, proper use of units, and defining all required entities.

Requirements

numpy>=2.3.1
pydantic>=2.11.7
PyYAML>=6.0.2

Creating a BenchmarkReport

A BenchmarkReport class may be instantiated from a JSON/YAML file with import_benchmark_report(), or from a JSON/YAML string with yaml_str_to_benchmark_report(). Both of these functions will detect the appropriate version of the benchmark report.

To generate a JSON or YAML string from a BenchmarkReport instance, use the get_json_str() and get_yaml_str() methods. To save as a JSON/YAML file, use the export_json() or export_yaml() methods.

import llmd_benchmark_report

# Instantiate a BenchmarkReport instance from a YAML file
br = llmd_benchmark_report.import_benchmark_report("llmd_benchmark_report/br_v0_2_example.yaml")

# Create a YAML string from the BenchmarkReport
br_yaml_str = br.get_yaml_str()

# Print the string
print(br_yaml_str)

# Save the BenchmarkReport as a YAML file
br.export_yaml("br_test.yaml")

An instance of BenchmarkReport may be created directly, for example:

import llmd_benchmark_report

br = llmd_benchmark_report.BenchmarkReportV02(
    **{
        "run": {
            "uid": "38b1f169ca178b756f7483523b17de61",
        },
        "scenario": {
            "stack": [
                {
                    "metadata": {
                        "schema_version": "0.0.1",
                        "label": "vllm-svc-0",
                        "cfg_id": "cc73fc6b51a1d3b8128f312d70476d7c",
                    },
                    "standardized": {
                        "kind": "inference_engine",
                        "tool": "llm-d",
                        "tool_version": "ghcr.io/llm-d/llm-d-cuda:0.3.1",
                        "role": "decode",
                        "replicas": 2,
                        "model": {
                            "name": "Qwen/Qwen3-0.6B",
                        },
                        "accelerator": {
                            "model": "NVIDIA-H100-80GB-HBM3",
                            "count": 8,
                            "parallelism": {
                                "tp": 8,
                            },
                        },
                    },
                    "native": {},
                },
            ],
            "load": {
                "metadata": {
                    "schema_version": "0.0.1",
                    "cfg_id": "a4e18f265cc33786a42b8a3f7ac2edcb",
                },
                "standardized": {
                    "tool": "inference-perf",
                    "tool_version": "0.3.0",
                    "input_seq_len": {
                        "distribution": "fixed",
                        "value": 2148,
                    },
                    "source": "sampled",
                },
                "native": {},
            },
        },
        "results": {
            "request_performance": {
                "aggregate": {
                    "latency": {
                        "request_latency": {
                            "units": "s",
                            "mean": 0.0368578234128654,
                        },
                    },
                },
            },
        },
    }
)

print(br.get_yaml_str())

The native formats returned by different harnesses may be converted to a benchmark report using functions in native_to_br0_2.py, or using the CLI defined in cli.py.

To run the converter CLI, execute the module using python -m from the root of this repository:

benchmark-report \
  --workload-generator <generator_name> \
  --br-version <version> \
  <path_to_raw_results_file> \
  [path_to_output_report_yaml]

Parameters Reference

• -w, --workload-generator: Specifies the harness generator. Must be one of: 'guidellm', 'inferencemax', 'inference-perf', 'vllm-benchmark', 'nop'.
• -b, --br-version: Target benchmark report version (defaults to 0.1; 0.2 and 0.2.1 both produce the current 0.2.x revision, 0.2.1).
• -f, --force: Overwrites the output file if it already exists.
• -i, --index: Convert only the benchmark at this index, for a results file holding several (see Multi-benchmark results files below). Omit it to convert all of them.
• results_file (Positional): Path to the raw native results file to convert. (e.g. For inference-perf, this must contain "stage_" in its filename, e.g., stage_0_lifecycle_metrics.json).
• output_file (Positional, Optional): Destination for the converted report. If omitted, the YAML output is printed directly to stdout.

Keep the two positionals together rather than separating them with a flag. On Python 3.11, argparse cannot match a positional that follows an optional, so output_file is rejected as an unrecognized argument; Python 3.12 and later parse it correctly. The harness pod image is Python 3.13, but requires-python is >=3.11, so a local checkout may hit this:

# Portable: positionals adjacent, either before or after the flags.
... cli -w guidellm -b 0.2 results.json benchmark_report.yaml
... cli results.json benchmark_report.yaml -w guidellm -b 0.2

# Python 3.11: "error: unrecognized arguments: benchmark_report.yaml"
... cli results.json -w guidellm -b 0.2 benchmark_report.yaml

The output is always YAML, whatever the output filename's extension. To write JSON, use export_json() from the Python API instead (Creating a BenchmarkReport); a report's YAML can also be converted after the fact with yq -o=json.

Example

To convert a raw stage_0_lifecycle_metrics.json results file from an inference-perf run and save the v0.2 report to benchmark_report_v02.yaml:

benchmark-report \
  -w inference-perf \
  -b 0.2 \
  ./stage_0_lifecycle_metrics.json \
  ./benchmark_report_v02.yaml

Multi-benchmark Results Files (GuideLLM)

GuideLLM writes a single results.json holding every benchmark of a run -- one per profile stage, so a constant profile with rate: [2, 4] yields two. A benchmark report describes one benchmark, so the CLI writes one report per stage, inserting the index before the extension of the name given:

# Writes benchmark_report_0.yaml and benchmark_report_1.yaml
benchmark-report \
  -w guidellm -b 0.2 results.json benchmark_report.yaml

With no output file, all of them are printed to stdout, each preceded by a # Benchmark <n> of <total> comment.

Pass -i / --index to convert a single stage instead. The report is then written to the given filename unchanged, with no index suffix:

# Writes benchmark_report.yaml, for the first stage only
benchmark-report \
  -w guidellm -b 0.2 -i 0 results.json benchmark_report.yaml

Which stage a report came from is recorded in scenario.load.standardized.stage, alongside the rate_qps or concurrency that stage actually ran at.

Session Lifecycle Files (Inference-Perf)

For inference-perf session lifecycle files (*_session_lifecycle_metrics.json), pass the -s / --session flag:

# Print to stdout
benchmark-report -w inference-perf -b 0.2 -s stage_0_session_lifecycle_metrics.json

# Save to file
benchmark-report -w inference-perf -b 0.2 -s stage_0_session_lifecycle_metrics.json benchmark_report_session.yaml

The -s flag selects the import_inference_perf_session converter, which populates results.session_performance instead of results.request_performance. The inference-perf-analyze_results.sh script detects *_session_lifecycle_metrics.json files automatically and passes -s without manual intervention.