GBD Benchmark Database Tools: Maintenance of Benchmark Instances and their Attributes
Project description
GBD Benchmark Database Tools
GBD Tools consists of two python tools, one command-line interface gbd
and the micro-services and website gbd-server
.
GBD facilitates the management of features of benchmark instances and thus the differentiated analysis of runtime experiments.
GBD was created for analysis of experiments in SAT Solving and to distribute benchmark instances in DIMACS CNF. Internally, Benchmark Instances are identified using GBD-Hash which hashes the normalized version of a CNF file, such that benchmark instance recognition is invariant to comments, compression, filename, line-encoding, etc. GBD can be used in other domains, if GBD-Hash is replaced accordingly.
Programming Language
- Python 3
- SQLite
Installation
- Setup python3 and pip3
- Install GBD via
pip3 install gbd-tools
Configuration
- Download a database, e.g., http://gbd.iti.kit.edu/getdatabase/meta.db, and safe it as [path/file]
export GBD_DB=[path/file]
(and put it in your .bashrc)- GBD_DB can contain several colon-spararted paths (semi-colon on Windows)
- GBD_DB can be overriden by command-line argument
gbd --db=[path/other]
GBD Command Line Interface
An overview on the commands of the gbd command-line interface is generated by gbd-help:
gbd --help
gbd [command] --help
Accessing information with GBD
In order to get an overview over the features stored in your database:
gbd info
gbd info [feature]
To get the list of benchmark instances referenced in your database (assuming the features 'result', 'filename' and 'family' are present in the database you use):
gbd get
gbd get -r filename
gbd get -r result family
GBD Queries can be used to filter the instances:
gbd get "family = hardware-bmc" -r filename
gbd get "family = hardware-bmc and (filename like %ibm% or author = biere)" -r filename
gbd get "family = uniform-random and (clauses / variables) = 4.2" -r filename
In order to use gbd in your experimentation workflow, gbd has to store the paths to the locally available benchmark instances. As this involves unpacking and hashing of all your instances, we recommend using gbds parallelization support. Also, we recommend installing the python acceleration module https://github.com/Udopia/gbdhash. In case you have downloaded your database from another source, this database might reference paths which are not available on your disk. GBD will ask you to remove them, just do it, this will not affect other features stored in the database.
gbd -j[nof-cores] init local [path/to/cnf]
Once you initialized the database with your local paths, you can run gbd queries and resolve the result against your local paths. If you have duplicates, you might want to use -c to collapse multiple values into one.
gbd get "family = hardware-bmc" -r local
gbd get "family = hardware-bmc" -r local -c min
Having initialized the instance feature local
, gbd can be used for experimentation.
while read -r hash path; do ./solve $path > $hash.out; done < <(gbd get "competition_track = main_2020" -r local)
GBD Queries are not the only way to retrieve instance features. Given a list of hashes, you can use pipes to retrieve data about the benchmark instances.
echo $hashes | gbd get -r family
Since instances are sometimes shuffled from one competition to another, we provide means to recognize the equivalence class of shuffled instances. You can generate the degree-sequence-hash for each instance with another sub-command of the init-command. GBDs degree-sequence-hash is the sorted sequence of in- and out-degrees of the polarity-perserving clause-variable graph.
gbd -j[nof-cores] init degree_sequence_hash
GBD Queries by default generate a list of gbd-hashes which is subsequently joined with the requested features. You can override this default grouping-behaviour, and e.g. group by degree_sequence_hash to display the equivalence groups, or to filter for only one representative.
gbd get -g degree_sequence_hash -r hash
gbd get -g degree_sequence_hash -r hash local -c min
Maintenance of Benchmark Instance Data with GBD
In GBD all data-points are seen as features of instances. Like this also runtimes of solvers on an instance can be stored as a specific featres. GBD provides means to bootstrap default-features, to distribute databases storing features and to create your own features.
GBD Create, Rename, Delete
GBD Import
GBD Set
Evaluating Experiments
GBD Feaure Meta-Info and Types
GBD Server and Micro-Services
GBD-Server starts a Web-Service alongside a bunch of micro-services which facilitates the distribution of benchmark instances and their feature over the web.
gbd-server --help
.
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
Hashes for gbd_tools-3.7.4-py3-none-any.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 87c04a897c93afe52a979af544c99d6c0f59b4d59cf98b703d3826d709c41010 |
|
MD5 | 172c8c94946069bddf89739bf6263b29 |
|
BLAKE2b-256 | 631ef694c7da801cd93e5774b6c73ba50ec4bfdfe58f26597437643c9953ea34 |