Bencher
Continuous Integration Status
Install
pip install holobench
Intro
Bencher is a tool to make it easy to benchmark the interactions between the input parameters to your algorithm and its resulting performance on a set of metrics. It calculates the cartesian product of a set of variables
Parameters for bencher are defined using the param library as a config class with extra metadata that describes the bounds of the search space you want to measure. You must define a benchmarking function that accepts an instance of the config class and return a dictionary with string metric names and float values.
Parameters are benchmarked by passing in a list N parameters, and an N-Dimensional tensor is returned. You can optionally sample each point multiple times to get back a distribution and also track its value over time. By default the data will be plotted automatically based on the types of parameters you are sampling (e.g, continuous, discrete), but you can also pass in a callback to customize plotting.
The data is stored in a persistent database so that past performance is tracked.
Assumptions
The input types should also be of one of the basic datatypes (bool, int, float, str, enum, datetime) so that the data can be easily hashed, cached and stored in the database and processed with seaborn and xarray plotting functions. You can use class inheritance to define hierarchical parameter configuration class types that can be reused in a bigger configuration classes.
Bencher is designed to work with stochastic pure functions with no side effects. It assumes that when the objective function is given the same inputs, it will return the same output +- random noise. This is because the function must be called multiple times to get a good statistical distribution of it and so each call must not be influenced by anything or the results will be corrupted.
Pseudocode of bencher
Enumerate a list of all input parameter combinations
for each set of input parameters:
pass the inputs to the objective function and store results in the N-D array
get unique hash for the set of inputs parameters
look up previous results for that hash
if it exists:
load historical data
combine latest data with historical data
store the results using the input hash as a key
deduce the type of plot based on the input and output types
return data and plot
Resource Management with sampling_context
If your benchmark holds external resources (DB pools, GPU handles, simulators) you
may want to release them before the interactive result viewer starts. Wrapping
the entire bn.run() call in a with block won't work — the context stays open
while the Panel/Bokeh server blocks:
# Anti-pattern: resources held during the entire viewing session
with gpu_context():
bn.run(my_bench, show=True)
Instead, pass the context manager as sampling_context. It wraps only the sampling
phase; its __exit__ runs before the server starts:
bn.run(my_bench, show=True, sampling_context=gpu_context())
save and publish still execute inside the context (during sampling), so results
are persisted before the resource is released.
Demo
if you have pixi installed you can run a demo example with:
pixi run demo
An example of the type of output bencher produces can be seen here:
https://blooop.github.io/bencher/
Examples
Most features are demonstrated in the auto-generated examples under bencher/example/generated/.
Run pixi run generate-docs to regenerate the full example gallery. Key sections include:
generated/N_float/— Parameter sweeps with 0–3 float inputs, with/without repeats and over-time trackinggenerated/plot_types/— All supported plot types (scatter, line, heatmap, surface, etc.)generated/result_types/— Result types: images, videos, strings, booleans, paths, datasetsgenerated/composable_containers/— Combining results with different composition strategiesgenerated/sampling/— Custom values, levels, uniform, int vs floatgenerated/optimization/— Single and multi-objective optimization with Optunagenerated/advanced/— Time events, caching, aggregation over timegenerated/regression/— Performance regression detectiongenerated/statistics/— Error bands, distributions, repeats comparison
A few hand-written examples remain for unique functionality:
example_simple_float.py— Minimal getting-started exampleexample_image.py/example_video.py— Image and video result typesexample_self_benchmark.py— Bencher self-introspectionexample_workflow.py— Multi-stage optimization workflow
Documentation
More documentation is needed for the examples and general workflow.
1 maintainer
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
Filter files by name, interpreter, ABI, and platform.
If you're not sure about the file name format, learn more about wheel file names.
Copy a direct link to the current filters
File details
Details for the file holobench-1.114.0.tar.gz.
File metadata
- Download URL: holobench-1.114.0.tar.gz
- Upload date:
- Size: 366.8 kB
- Tags: Source
- Uploaded using Trusted Publishing? No
- Uploaded via:
twine/6.2.0 CPython/3.12.13
File hashes
| Algorithm | Hash digest | |
|---|---|---|
| SHA256 |
398eaf2f046b9ebcca95d86638985f4c114a25395de03c7b857e4b7e7163978e
|
|
| MD5 |
cd2f35d280ec61330defc5e8af5d60ea
|
|
| BLAKE2b-256 |
f2a378b2e02f04d8954cee4917a0e361fbf70b5e90bfa343102ae1cb002b6074
|
File details
Details for the file holobench-1.114.0-py3-none-any.whl.
File metadata
- Download URL: holobench-1.114.0-py3-none-any.whl
- Upload date:
- Size: 597.1 kB
- Tags: Python 3
- Uploaded using Trusted Publishing? No
- Uploaded via:
twine/6.2.0 CPython/3.12.13
File hashes
| Algorithm | Hash digest | |
|---|---|---|
| SHA256 |
ce82d7a5b8a946bf2dc4ba5482061dbd82d055283b671369d75ec1cbab9b23d7
|
|
| MD5 |
4022e25817b0cc2beaf15057f1662f5b
|
|
| BLAKE2b-256 |
e34f630104f5ec1292860e055a2bc2ef605b6a5bccad3cdf8ea73fffcc1649d9
|
