Skip to main content
Help us improve PyPI by participating in user testing. All experience levels needed!

A Framework for Reliable Benchmarking and Resource Measurement.

Project description

Build Status Code Quality Test Coverage Issue Stats Apache 2.0 License PyPI version

News:

BenchExec provides three major features:

  • execution of arbitrary commands with precise and reliable measurement and limitation of resource usage (e.g., CPU time and memory), and isolation against other running processes
  • an easy way to define benchmarks with specific tool configurations and resource limits, and automatically executing them on large sets of input files
  • generation of interactive tables and plots for the results

Contrary to other benchmarking frameworks, it is able to reliably measure and limit resource usage of the benchmarked tool even if it spawns subprocesses. In order to achieve this, it uses the cgroups feature of the Linux kernel to correctly handle groups of processes. For proper isolation of the benchmarks, it uses (if available) Linux user namespaces and an overlay filesystem to create a container that restricts interference of the executed tool with the benchmarking host. BenchExec is intended for benchmarking non-interactive tools on Linux systems. It measures CPU time, wall time, and memory usage of a tool, and allows to specify limits for these resources. It also allows to limit the CPU cores and (on NUMA systems) memory regions, and the container mode allows to restrict filesystem and network access. In addition to measuring resource usage, BenchExec can verify that the result of the tool was as expected, and extract further statistical data from the output. Results from multiple runs can be combined into CSV and interactive HTML tables, of which the latter provide scatter and quantile plots (have a look at our demo table).

BenchExec works only on Linux and needs a one-time setup of cgroups by the machine’s administrator. The actual benchmarking can be done by any user and does not need root access.

BenchExec was originally developed for use with the software verification framework CPAchecker and is now developed as an independent project at the Software Systems Lab at the Ludwig-Maximilians-Universität München (LMU).

Users of BenchExec

BenchExec was successfully used for benchmarking in all seven instances of the International Competition on Software Verification (2012-2018) with a wide variety of benchmarked tools and hundreds of thousands benchmark runs.

The developers of the following tools use BenchExec:

If you would like to be listed here, contact us.

Project details


Release history Release notifications

This version
History Node

1.16

History Node

1.14

History Node

1.13

History Node

1.12

History Node

1.11

History Node

1.10

History Node

1.9

History Node

1.8

History Node

1.7

History Node

1.6

History Node

1.5

History Node

1.4

History Node

1.3

History Node

1.2

History Node

1.1

History Node

1.0

History Node

0.5

History Node

0.4

History Node

0.3

Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Filename, size & hash SHA256 hash help File type Python version Upload date
BenchExec-1.16-py2.7.egg (275.8 kB) Copy SHA256 hash SHA256 Egg 2.7 Jan 31, 2018
BenchExec-1.16-py3.5.egg (557.9 kB) Copy SHA256 hash SHA256 Egg 3.5 Jan 31, 2018
BenchExec-1.16-py3-none-any.whl (262.2 kB) Copy SHA256 hash SHA256 Wheel py3 Jan 31, 2018
BenchExec-1.16.tar.gz (876.4 kB) Copy SHA256 hash SHA256 Source None Jan 31, 2018

Supported by

Elastic Elastic Search Pingdom Pingdom Monitoring Google Google BigQuery Sentry Sentry Error logging CloudAMQP CloudAMQP RabbitMQ AWS AWS Cloud computing Fastly Fastly CDN DigiCert DigiCert EV certificate StatusPage StatusPage Status page