pytest-benchmark 2.3.0

py.test fixture for benchmarking code

A py.test fixture for benchmarking code.

• Free software: BSD license

Installation

pip install pytest-benchmark

Usage

def test_my_stuff(benchmark):
    @benchmark
    def result():
        # Code to be measured
        return time.sleep(0.000001)

    # Extra code, to verify that the run completed correctly.
    # Note: this code is not measured.
    assert result is None

py.test command-line options:

--benchmark-min-time=BENCHMARK_MIN_TIME

Minimum time per round. Default: 25.00us

--benchmark-max-time=BENCHMARK_MAX_TIME

Maximum time to spend in a benchmark. Default: 1.00s

--benchmark-min-rounds=BENCHMARK_MIN_ROUNDS

Minimum rounds, even if total time would exceed –max-time. Default: 5

--benchmark-sort=BENCHMARK_SORT

Column to sort on. Can be one of: ‘min’, ‘max’, ‘mean’ or ‘stddev’. Default: min

--benchmark-timer=BENCHMARK_TIMER

Timer to use when measuring time. Default: time.perf_counter

--benchmark-warmup

Runs the benchmarks two times. Discards data from the first run.

--benchmark-warmup-iterations=BENCHMARK_WARMUP_ITERATIONS

Max number of iterations to run in the warmup phase. Default: 100000

--benchmark-verbose

Dump diagnostic and progress information.

--benchmark-disable-gc

Disable GC during benchmarks.

--benchmark-skip

Skip running any benchmarks.

--benchmark-only

Only run benchmarks.

Setting per-test options:

@pytest.mark.benchmark(
    group="group-name",
    min_time=0.1,
    max_time=0.5,
    min_rounds=5,
    timer=time.time,
    disable_gc=True,
    warmup=False
)
def test_my_stuff(benchmark):
    @benchmark
    def result():
        # Code to be measured
        return time.sleep(0.000001)

    # Extra code, to verify that the run
    # completed correctly.
    # Note: this code is not measured.
    assert result is None

Documentation

https://pytest-benchmark.readthedocs.org/

Obligatory screenshot

Development

To run the all tests run:

tox

Credits

• Timing code and ideas taken from: https://bitbucket.org/haypo/misc/src/tip/python/benchmark.py

Changelog

2.3.0 (2014-12-27)

• Moved the warmup in the calibration phase. Solves issues with benchmarking on PyPy.

  Added a --benchmark-warmup-iterations option to fine-tune that.

2.2.0 (2014-12-26)

• Make the default rounds smaller (so that variance is more accurate).

• Show the defaults in the --help section.

2.1.0 (2014-12-20)

• Simplify the calibration code so that the round is smaller.

• Add diagnostic output for calibration code (--benchmark-verbose).

2.0.0 (2014-12-19)

• Replace the context-manager based API with a simple callback interface.

• Implement timer calibration for precise measurements.

1.0.0 (2014-12-15)

• Use a precise default timer for PyPy.

? (?)

• Readme and styling fixes (contributed by Marc Abramowitz)

• Lots of wild changes.