Benchmark-Runner: Running benchmarks
What is it?
benchmark-runner is a containerized Python lightweight and flexible framework for running benchmark workloads on Kubernetes/OpenShift runtype kinds Pod and VM.
This framework support the following embedded workloads:
- hammerdb: running hammerdb workload on the following databases: MSSQL, Mariadb, Postgresql in Pod or VM with Configuration
- stressng: running stressng workload in Pod or VM Configuration
- sysbench: running sysbench workload in Pod or VM with Configuration
- uperf: running uperf workload in Pod or VM with Configuration
- vdbench: running vdbench workload in Pod or VM with Configuration
- fio: running fio storage benchmark workload in Pod or VM with Configuration
- winfio: running fio storage benchmark workload on Windows VM with Configuration
- bootstorm: calculate VMs boot load time Configuration
** For hammerdb mssql must run once permission
Benchmark-runner grafana dashboard example:
Reference:
- The benchmark-runner package is located in PyPi
- The benchmark-runner container image is located in Quay.io
Documentation
Documentation is available at benchmark-runner.readthedocs.io
Table of Contents
Run workload using Podman or Docker
The following options may be passed via command line flags or set in the environment:
mandatory: KUBEADMIN_PASSWORD=$KUBEADMIN_PASSWORD
mandatory: $KUBECONFIG [ kubeconfig file path]
mandatory: WORKLOAD=$WORKLOAD
Choose one from the following list:
['stressng_pod', 'stressng_vm', 'uperf_pod', 'uperf_vm', 'sysbench_pod', 'sysbench_vm', 'hammerdb_pod_mariadb', 'hammerdb_vm_mariadb', 'hammerdb_pod_mariadb_lso', 'hammerdb_vm_mariadb_lso', 'hammerdb_pod_mariadb_ephemeral', 'hammerdb_vm_mariadb_ephemeral', 'hammerdb_pod_postgres', 'hammerdb_vm_postgres', 'hammerdb_pod_postgres_lso', 'hammerdb_vm_postgres_lso', 'hammerdb_pod_postgres_ephemeral', 'hammerdb_vm_postgres_ephemeral', 'hammerdb_pod_mssql', 'hammerdb_vm_mssql', 'hammerdb_pod_mssql_lso', 'hammerdb_vm_mssql_lso', 'hammerdb_pod_mssql_ephemeral', 'hammerdb_vm_mssql_ephemeral', 'vdbench_pod', 'vdbench_vm', 'vdbench_pod_ephemeral', 'vdbench_vm_ephemeral', 'fio_pod', 'fio_vm', 'fio_pod_ephemeral', 'fio_vm_ephemeral', 'clusterbuster', 'krknhub', 'bootstorm_vm', 'windows_vm', 'winmssql_vm', 'winfio_vm' ]
** clusterbuster workloads: cpusoaker, files, fio, uperf. for more details see ** For windows workloads: need to share windows qcow2 image by nginx ** For hammerdb mssql must run only once permission ** winmssql_vm: will run hammerdb inside windows server mssql 2022: for more details see
Not mandatory:
auto: NAMESPACE=benchmark-runner [ The default namespace is benchmark-runner ]
auto: ODF_PVC=True [ True=ODF PVC storage, False=Ephemeral storage, default True ]
auto: EXTRACT_PROMETHEUS_SNAPSHOT=True [ True=extract Prometheus snapshot into artifacts, false=don't, default True ]
auto: SYSTEM_METRICS=False [ True=collect metric, False=not collect metrics, default False ]
auto: RUNNER_PATH=/tmp [ The default work space is /tmp ]
optional: PIN_NODE1=$PIN_NODE1 [node1 selector for running the workload]
optional: PIN_NODE2=$PIN_NODE2 [node2 selector for running the workload, i.e. uperf server and client, hammerdb database and workload]
optional: ELASTICSEARCH=$ELASTICSEARCH [ elasticsearch service name]
optional: ELASTICSEARCH_PORT=$ELASTICSEARCH_PORT
optional: CLUSTER=$CLUSTER [ set CLUSTER='kubernetes' to run workload on a kubernetes cluster, default 'openshift' ]
optional:scale SCALE=$SCALE [For Vdbench/Bootstorm: Scale in each node]
optional:scale SCALE_NODES=$SCALE_NODES [For Vdbench/Bootstorm: Scale's node]
optional:scale REDIS=$REDIS [For Vdbench only: redis for scale synchronization]
optional: LSO_DISK_ID=$LSO_DISK_ID [LSO_DISK_ID='scsi-<replace_this_with_your_actual_disk_id>' For using LSO Operator in hammerdb]
optional: WORKER_DISK_IDS=$WORKER_DISK_IDS [WORKER_DISK_IDS For ODF/LSO workloads hammerdb/vdbench]
optional: WINDOWS_URL=$WINDOWS_URL [WINDOWS_URL for qcow2 image — HTTP URL or S3 path-style HTTPS URL (https://s3.{region}.amazonaws.com/{bucket}/{key})]
optional: CDI_SOURCE_TYPE=$CDI_SOURCE_TYPE [CDI DataVolume source type: http (default) or s3 for authenticated S3 access]
optional: CDI_SOURCE_S3_CRED=$CDI_SOURCE_S3_CRED [K8s secret name containing accessKeyId and secretKey for S3 authentication. Secret must exist in benchmark-runner namespace before workload runs. Create with: oc create secret generic <secret-name> --from-literal=accessKeyId=AKIA... --from-literal=secretKey=CLj4... -n benchmark-runner]
optional: VM_STORAGE_CLASS=$VM_STORAGE_CLASS [Storage class for all VM workload PVCs (Windows and Linux). Default: ocs-storagecluster-ceph-rbd-virtualization]
For example:
podman run --rm -e WORKLOAD="hammerdb_pod_mariadb" -e KUBEADMIN_PASSWORD="1234" -e PIN_NODE1="node_name-1" -e PIN_NODE2="node_name-2" -e log_level=INFO -v /root/.kube/config:/root/.kube/config --privileged quay.io/benchmark-runner/benchmark-runner:latest
or
docker run --rm -e WORKLOAD="hammerdb_vm_mariadb" -e KUBEADMIN_PASSWORD="1234" -e PIN_NODE1="node_name-1" -e PIN_NODE2="node_name-2" -e log_level=INFO -v /root/.kube/config:/root/.kube/config --privileged quay.io/benchmark-runner/benchmark-runner:latest
SAVE RUN ARTIFACTS LOCAL:
- add
-e SAVE_ARTIFACTS_LOCAL='True'or--save-artifacts-local=true - add
-v /tmp/benchmark-runner-run-artifacts:/tmp/benchmark-runner-run-artifacts
Run vdbench workload in Pod using OpenShift
Run vdbench workload in Pod using Kubernetes
Run workload in Pod using Kubernetes or OpenShift
[TBD]
Grafana dashboards
There are 2 grafana dashboards templates:
- FuncCi dashboard
- PerfCi dashboard ** PerfCi dashboard is generated automatically in Build GitHub actions from main.libsonnet
** After importing json in grafana, you need to configure elasticsearch data source. (for more details: see HOW_TO.md)
Inspect Prometheus Metrics
The CI jobs store snapshots of the Prometheus database for each run as part of the artifacts. Within the artifact directory is a Prometheus snapshot directory named:
promdb-YYYY_MM_DDTHH_mm_ss+0000_YYYY_MM_DDTHH_mm_ss+0000.tar
The timestamps are for the start and end of the metrics capture; they
are stored in UTC time (+0000). It is possible to run containerized
Prometheus on it to inspect the metrics. Note that Prometheus
requires write access to its database, so it will actually write to
the snapshot. So for example if you have downloaded artifacts for a
run named hammerdb-vm-mariadb-2022-01-04-08-21-23 and the Prometheus
snapshot within is named
promdb_2022_01_04T08_21_52+0000_2022_01_04T08_45_47+0000, you could run as follows:
$ local_prometheus_snapshot=/hammerdb-vm-mariadb-2022-01-04-08-21-23/promdb_2022_01_04T08_21_52+0000_2022_01_04T08_45_47+0000
$ chmod -R g-s,a+rw "$local_prometheus_snapshot"
$ sudo podman run --rm -p 9090:9090 -uroot -v "$local_prometheus_snapshot:/prometheus" --privileged prom/prometheus --config.file=/etc/prometheus/prometheus.yml --storage.tsdb.path=/prometheus --storage.tsdb.retention.time=100000d --storage.tsdb.retention.size=1000PB
and point your browser at port 9090 on your local system, you can run queries against it, e.g.
sum(irate(node_cpu_seconds_total[2m])) by (mode,instance) > 0
It is important to use the --storage.tsdb.retention.time option to
Prometheus, as otherwise Prometheus may discard the data in the
snapshot. And note that you must set the time bounds on the
Prometheus query to fit the start and end times as recorded in the
name of the promdb snapshot.
How to develop in benchmark-runner
see HOW_TO.md
benchmark-runner blog
open link
1 maintainer
Metadata
1 maintainer
Release files for benchmark-runner 1.0.1049
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | |
|---|---|---|---|
| benchmark_runner-1.0.1049.tar.gz | 200.4 kB | Details |
Built distribution (wheel)
| File | Interpreter | ABI | Platform | Reset |
|---|---|---|---|---|
| benchmark_runner-1.0.1049-py3-none-any.whl | Python 3 | none | any | Details |
Total release size: 508.4 kB
Release files / benchmark_runner-1.0.1049.tar.gz
| Download URL | benchmark_runner-1.0.1049.tar.gz |
|---|---|
| Size | 200.4 kB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
2806915acc0c50dc16905484f3e21710523025f58a4a940732632db2e979c1e3
|
|
BLAKE2b-256 checksum How to use checksums |
c1ed1a3bd6c8a7bf3c0225ebc6c3bb0c3ea4b3e056b48102745f0be788cb6029
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/7.0.0 CPython/3.14.7
|
Release files / benchmark_runner-1.0.1049-py3-none-any.whl
| Download URL | benchmark_runner-1.0.1049-py3-none-any.whl |
|---|---|
| Size | 307.9 kB |
| Tags | Python 3 |
|
SHA-256 checksum How to use checksums |
2181d017b9eade7c0abcdf8738b5b94fd824feaad65f52796f6a601991b1b53d
|
|
BLAKE2b-256 checksum How to use checksums |
60b8218d76742c568fd101ef7b28f13d7aa1dd45c687e0766c387ef47988d6f6
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/7.0.0 CPython/3.14.7
|
