Omnibenchmark core utilities: Setup and running of continous benchmarking modules as part of omnibenchmark
Project description
Omnibenchmark
Generate and manage omnibenchmark modules for open and continous benchmarking. Each module represents a single building block of a benchmark, e.g., dataset, method, metric, ... Omnibenchmark-py provides a structure to generate modules and automatically run them.
Installation
You can install omnibenchmark from PyPI:
pip install omnibenchmark
The module is supported on Python >= 3.8 and requires renku >= 1.4.0.
How to use omnibenchmark
Quick start
Omnibenchmark uses the renku
platform to run open and continous benchmarks. To contribute an independent module to one of the existing benchmarks please start by creating a new renku project.
All relevant information on how to run a specific module are collected as OmniObject
.
The simplest way to generate an instance of an OmniObject
is to build it from an config.yaml
file:
## modules
from omnibenchmark.utils.build_omni_object import get_omni_object_from_yaml
## Load object
omni_obj = get_omni_object_from_yaml('src/config.yaml')
The config.yaml
defines all module specific information as inputs, outputs, script to run the module, benchmark that the module belongs to and much more. A simple config.yaml
file could look like this. Please check section The config.yaml file for more details.
---
data:
name: "module-name"
title: "A new module"
description: "A new module for omnibenchmark, e.g., a dataset, method, metric,..."
keywords: ["module-type-key"]
script: "path/to/module_script"
outputs:
template: "data/${name}/${name}_${out_name}.${out_end}"
files:
counts:
end: "mtx.gz"
data_info:
end: "json"
meta:
end: "json"
feature:
end: "json"
benchmark_name: "an-omnibenchmark"
Once you have an instance of an OmniObject
you can check, if it looks as you expected like this:
## Check object
print(omni_obj.__dict__)
print(omni_obj.outputs.file_mapping)
print(omni_obj.command.command_line)
If all inputs, outputs and the command line call look as you expected you can run your module:
## create output dataset that stores all result/output files
omni_obj.create_dataset()
## Update inputs from other modules
omni_obj.update_obj()
## Run your script with all defined inputs and outputs.
## This also generates a workflow description (plan) and is tracked as activity.
omni_obj.run_renku()
## Link output files to output dataset
omni_obj.update_result_dataset()
## Save and commit to gitlab
renku_save()
Once these steps ran successfully and your outputs were generated the module is ready to be submitted to become a part of omnibenchmark.
What is renku?
Create a new renku project
The config.yaml file
Omnibenchmark classes
OmniObject
OmniInput
OmniOutput
OmniCommand
OmniPlan
Submit your module
Release History
- 0.1.1
- FIX:
- 0.1.0
- The first proper release
- CHANGE:
- 0.0.1
- First version of all main functionalities
Meta
Almut Lütge – @Almut30618742 Anthony Sonrel – @AnthonySonrel Mark Robinson – @markrobinsonca
Distributed under the Apache 2.0 license. See LICENSE
for more information.
https://github.com/almutlue/omnibenchmark-py
Contributing
- Fork it (https://github.com/almutlue/omnibenchmark-py/fork)
- Create your feature branch (
git checkout -b feature/fooBar
) - Commit your changes (
git commit -am 'Add some fooBar'
) - Push to the branch (
git push origin feature/fooBar
) - Create a new Pull Request
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
Hashes for omnibenchmark-0.0.1-py3-none-any.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 9fa9e2b433ac4599502d94609e7dcdb9cb208b53795324763fca1de7092c104c |
|
MD5 | 6c67e96a9ab38212dfd6d439172814af |
|
BLAKE2b-256 | d81e933837d86a838e69b4af6ae3090b2b56fcbe2c67e50d55f4f5f8a039b0f9 |