Skip to main content


Benchmark-Runner: Running benchmarks

Actions Status PyPI Latest Release Container Repository on Quay Coverage Status Documentation Status python License

What is it?

benchmark-runner is a containerized Python lightweight and flexible framework for running benchmark workloads on Kubernetes/OpenShift runtype kinds Pod and VM.

This framework support the following embedded workloads:

** For hammerdb mssql must run once permission

Benchmark-runner grafana dashboard example:

Reference:

  • The benchmark-runner package is located in PyPi
  • The benchmark-runner container image is located in Quay.io

Documentation

Documentation is available at benchmark-runner.readthedocs.io

Table of Contents

Run workload using Podman or Docker

The following options may be passed via command line flags or set in the environment:

mandatory: KUBEADMIN_PASSWORD=$KUBEADMIN_PASSWORD

mandatory: $KUBECONFIG [ kubeconfig file path]

mandatory: WORKLOAD=$WORKLOAD

Choose one from the following list:

['stressng_pod', 'stressng_vm', 'uperf_pod', 'uperf_vm', 'sysbench_pod', 'sysbench_vm', 'hammerdb_pod_mariadb', 'hammerdb_vm_mariadb', 'hammerdb_pod_mariadb_lso', 'hammerdb_vm_mariadb_lso', 'hammerdb_pod_mariadb_ephemeral', 'hammerdb_vm_mariadb_ephemeral', 'hammerdb_pod_postgres', 'hammerdb_vm_postgres', 'hammerdb_pod_postgres_lso', 'hammerdb_vm_postgres_lso', 'hammerdb_pod_postgres_ephemeral', 'hammerdb_vm_postgres_ephemeral', 'hammerdb_pod_mssql', 'hammerdb_vm_mssql', 'hammerdb_pod_mssql_lso', 'hammerdb_vm_mssql_lso', 'hammerdb_pod_mssql_ephemeral', 'hammerdb_vm_mssql_ephemeral', 'vdbench_pod', 'vdbench_vm', 'vdbench_pod_ephemeral', 'vdbench_vm_ephemeral', 'fio_pod', 'fio_vm', 'fio_pod_ephemeral', 'fio_vm_ephemeral', 'clusterbuster', 'krknhub', 'bootstorm_vm', 'windows_vm', 'winmssql_vm', 'winfio_vm' ]

** clusterbuster workloads: cpusoaker, files, fio, uperf. for more details see ** For windows workloads: need to share windows qcow2 image by nginx ** For hammerdb mssql must run only once permission ** winmssql_vm: will run hammerdb inside windows server mssql 2022: for more details see

Not mandatory:

auto: NAMESPACE=benchmark-runner [ The default namespace is benchmark-runner ]

auto: ODF_PVC=True [ True=ODF PVC storage, False=Ephemeral storage, default True ]

auto: EXTRACT_PROMETHEUS_SNAPSHOT=True [ True=extract Prometheus snapshot into artifacts, false=don't, default True ]

auto: SYSTEM_METRICS=False [ True=collect metric, False=not collect metrics, default False ]

auto: RUNNER_PATH=/tmp [ The default work space is /tmp ]

optional: PIN_NODE1=$PIN_NODE1 [node1 selector for running the workload]

optional: PIN_NODE2=$PIN_NODE2 [node2 selector for running the workload, i.e. uperf server and client, hammerdb database and workload]

optional: ELASTICSEARCH=$ELASTICSEARCH [ elasticsearch service name]

optional: ELASTICSEARCH_PORT=$ELASTICSEARCH_PORT

optional: CLUSTER=$CLUSTER [ set CLUSTER='kubernetes' to run workload on a kubernetes cluster, default 'openshift' ]

optional:scale SCALE=$SCALE [For Vdbench/Bootstorm: Scale in each node]

optional:scale SCALE_NODES=$SCALE_NODES [For Vdbench/Bootstorm: Scale's node]

optional:scale REDIS=$REDIS [For Vdbench only: redis for scale synchronization]

optional: LSO_DISK_ID=$LSO_DISK_ID [LSO_DISK_ID='scsi-<replace_this_with_your_actual_disk_id>' For using LSO Operator in hammerdb]

optional: WORKER_DISK_IDS=$WORKER_DISK_IDS [WORKER_DISK_IDS For ODF/LSO workloads hammerdb/vdbench]

optional: WINDOWS_URL=$WINDOWS_URL [WINDOWS_URL for qcow2 image — HTTP URL or S3 path-style HTTPS URL (https://s3.{region}.amazonaws.com/{bucket}/{key})]

optional: CDI_SOURCE_TYPE=$CDI_SOURCE_TYPE [CDI DataVolume source type: http (default) or s3 for authenticated S3 access]

optional: CDI_SOURCE_S3_CRED=$CDI_SOURCE_S3_CRED [K8s secret name containing accessKeyId and secretKey for S3 authentication. Secret must exist in benchmark-runner namespace before workload runs. Create with: oc create secret generic <secret-name> --from-literal=accessKeyId=AKIA... --from-literal=secretKey=CLj4... -n benchmark-runner]

optional: VM_STORAGE_CLASS=$VM_STORAGE_CLASS [Storage class for all VM workload PVCs (Windows and Linux). Default: ocs-storagecluster-ceph-rbd-virtualization]

For example:

podman run --rm -e WORKLOAD="hammerdb_pod_mariadb" -e KUBEADMIN_PASSWORD="1234" -e PIN_NODE1="node_name-1" -e PIN_NODE2="node_name-2" -e log_level=INFO -v /root/.kube/config:/root/.kube/config --privileged quay.io/benchmark-runner/benchmark-runner:latest

or

docker run --rm -e WORKLOAD="hammerdb_vm_mariadb" -e KUBEADMIN_PASSWORD="1234" -e PIN_NODE1="node_name-1" -e PIN_NODE2="node_name-2" -e log_level=INFO -v /root/.kube/config:/root/.kube/config --privileged quay.io/benchmark-runner/benchmark-runner:latest

SAVE RUN ARTIFACTS LOCAL:

  1. add -e SAVE_ARTIFACTS_LOCAL='True' or --save-artifacts-local=true
  2. add -v /tmp/benchmark-runner-run-artifacts:/tmp/benchmark-runner-run-artifacts

Run vdbench workload in Pod using OpenShift

Run vdbench workload in Pod using Kubernetes

Run workload in Pod using Kubernetes or OpenShift

[TBD]

Grafana dashboards

There are 2 grafana dashboards templates:

  1. FuncCi dashboard
  2. PerfCi dashboard ** PerfCi dashboard is generated automatically in Build GitHub actions from main.libsonnet

** After importing json in grafana, you need to configure elasticsearch data source. (for more details: see HOW_TO.md)

Inspect Prometheus Metrics

The CI jobs store snapshots of the Prometheus database for each run as part of the artifacts. Within the artifact directory is a Prometheus snapshot directory named:

promdb-YYYY_MM_DDTHH_mm_ss+0000_YYYY_MM_DDTHH_mm_ss+0000.tar

The timestamps are for the start and end of the metrics capture; they are stored in UTC time (+0000). It is possible to run containerized Prometheus on it to inspect the metrics. Note that Prometheus requires write access to its database, so it will actually write to the snapshot. So for example if you have downloaded artifacts for a run named hammerdb-vm-mariadb-2022-01-04-08-21-23 and the Prometheus snapshot within is named promdb_2022_01_04T08_21_52+0000_2022_01_04T08_45_47+0000, you could run as follows:

$ local_prometheus_snapshot=/hammerdb-vm-mariadb-2022-01-04-08-21-23/promdb_2022_01_04T08_21_52+0000_2022_01_04T08_45_47+0000
$ chmod -R g-s,a+rw "$local_prometheus_snapshot"
$ sudo podman run --rm -p 9090:9090 -uroot -v "$local_prometheus_snapshot:/prometheus" --privileged prom/prometheus --config.file=/etc/prometheus/prometheus.yml --storage.tsdb.path=/prometheus --storage.tsdb.retention.time=100000d --storage.tsdb.retention.size=1000PB

and point your browser at port 9090 on your local system, you can run queries against it, e.g.

sum(irate(node_cpu_seconds_total[2m])) by (mode,instance) > 0

It is important to use the --storage.tsdb.retention.time option to Prometheus, as otherwise Prometheus may discard the data in the snapshot. And note that you must set the time bounds on the Prometheus query to fit the start and end times as recorded in the name of the promdb snapshot.

How to develop in benchmark-runner

see HOW_TO.md

benchmark-runner blog

open link

Metadata

Release files for benchmark-runner 1.0.1062

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for benchmark-runner 1.0.1062
File Size Uploaded Provenance Actions
benchmark_runner-1.0.1062.tar.gz 206.6 kB Without provenance Details for benchmark_runner-1.0.1062.tar.gz

Built distribution (wheel)

Table of built distributions (wheels) for benchmark-runner 1.0.1062
File Interpreter ABI Platform Actions
benchmark_runner-1.0.1062-py3-none-any.whl Python 3 none any Details for benchmark_runner-1.0.1062-py3-none-any.whl

Total release size: 518.4 kB

Release files / benchmark_runner-1.0.1062.tar.gz

Download URL benchmark_runner-1.0.1062.tar.gz
Size 206.6 kB
Tags Source
SHA-256 checksum
How to use checksums
3ddd9b4ce14e76aa73d41e82a0b9dc3facb0b7cfce6007d886f809fc747901b0
BLAKE2b-256 checksum
How to use checksums
7d98cd84e905960f660bb8317a0f2d6d2220db2d298e27e0d3eb0beb85175305
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via twine/7.0.0 CPython/3.15.1

Release files / benchmark_runner-1.0.1062-py3-none-any.whl

Download URL benchmark_runner-1.0.1062-py3-none-any.whl
Size 311.8 kB
Tags Python 3
SHA-256 checksum
How to use checksums
98d153b9211dc8bea971d75f92158657d93426fb7ed8cf7ae774439ef01f344f
BLAKE2b-256 checksum
How to use checksums
bb7a5e7cc0c6dd985965f0713aac4b76d8f6df18064f70c4aac9d6e974d7ecd1
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via twine/7.0.0 CPython/3.15.1

Security for benchmark-runner 1.0.1062

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

No provenance found

None of the files in this release were uploaded with provenance metadata.

A note on package security

PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.

To learn more about protecting yourself from malicious actors, visit our package security documentation.

Release history Release notifications | RSS feed

This release

1.0.1062 This release

2 release files No provenance

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page