neurtu 0.3.0

A simple benchmarking tool

Simple performance measurement tool

neurtu is a Python package providing a common interface for multi-metric benchmarks (including time and memory measurements). It can can be used to estimate time and space complexity of algorithms, while pandas integration allows quick analysis and visualization of the results.

neurtu means “to measure / evaluate” in Basque language.

See the documentation for more details.

Installation

neurtu requires Python 2.7 or 3.4+, it can be installed with,

pip install neurtu

pandas >=0.20 is an optional (but highly recommended) dependency.

Quickstart

To illustrate neurtu usage, will will benchmark array sorting in numpy. First, we will generator of cases,

import numpy as np
import neurtu

def cases()
    rng = np.random.RandomState(42)

    for N in [1000, 10000, 100000]:
        X = rng.rand(N)
        tags = {'N' : N}
        yield neurtu.delayed(X, tags=tags).sort()

that yields a sequence of delayed calculations, each tagged with the parameters defining individual runs.

We can evaluate the run time with,

>>> df = neurtu.timeit(cases())
>>> print(df)
        wall_time
N
1000     0.000014
10000    0.000134
100000   0.001474

which will internally use timeit module with a sufficient number of evaluation to work around the timer precision limitations (similarly to IPython’s %timeit). It will also display a progress bar for long running benchmarks, and return the results as a pandas.DataFrame (if pandas is installed).

By default, all evaluations are run with repeat=1. If more statistical confidence is required, this value can be increased,

>>> neurtu.timeit(cases(), repeat=3)
       wall_time
            mean       max       std
N
1000    0.000012  0.000014  0.000002
10000   0.000116  0.000149  0.000029
100000  0.001323  0.001714  0.000339

In this case we will get a frame with a pandas.MultiIndex for columns, where the first level represents the metric name (wall_time) and the second – the aggregation method. By default neurtu.timeit is called with aggregate=['mean', 'max', 'std'] methods, as supported by the pandas aggregation API. To disable, aggregation and obtains timings for individual runs, use aggregate=False. See neurtu.timeit documentation for more details.

To evaluate the peak memory usage, one can use the neurtu.memit function with the same API,

>>> neurtu.memit(cases(), repeat=3)
        peak_memory
               mean  max  std
N
10000           0.0  0.0  0.0
100000          0.0  0.0  0.0
1000000         0.0  0.0  0.0

More generally neurtu.Benchmark supports a wide number of evaluation metrics,

>>> bench = neurtu.Benchmark(wall_time=True, cpu_time=True, peak_memory=True)
>>> bench(cases())
         cpu_time  peak_memory  wall_time
N
10000    0.000100          0.0   0.000142
100000   0.001149          0.0   0.001680
1000000  0.013677          0.0   0.018347

including psutil process metrics.

For more information see the documentation and examples.

License

neurtu is released under the 3-clause BSD license.