Getting started

Consider a setup to compare NumPy ufuncs - sum, prod, max on arrays varying in their sizes. To keep it simple, let’s consider 1D arrays. Thus, we would have :

>>> import numpy as np
>>> funcs = [np.sum,np.prod,np.max]
>>> inputs = [np.random.rand(i) for i in 10**np.arange(5)]

>>> import benchit
>>> t = benchit.timings(funcs, inputs)

It’s a dataframe-like object and as such we can plot it. It automatically adds in specs into the title area to convey all of available benchmarking info :

>>> t.plot(logy=True, logx=True)

Multiple arguments

Let’s consider a setup where functions accept more than one argument. Let’s take the case of computing euclidean distances between two 2D arrays. We will feed in arrays with varying number of rows and 3 columns to represent data in 3D Cartesian coordinate system and benchmark two commonly used functions in Python.

>>> from sklearn.metrics.pairwise import pairwise_distances
>>> from scipy.spatial.distance import cdist
>>> fns = [cdist, pairwise_distances]

>>> import numpy as np
>>> in_ = {n:[np.random.rand(n,3), np.random.rand(n,3)] for n in [10,100,500,1000,4000]}
>>> t = benchit.timings(fns, in_, multivar=True, input_name='Array-length')
>>> t.plot(logx=True)

Multiple arguments with groupings

We will extend the previous example to make the second argument a variable too and study the trend as we vary the number of columns, resulting in subplots.

# Get benchmarking object (dataframe-like) and plot results
>>> R = np.random.rand
>>> in_ = {(n,W):[R(n,W), R(n,W)] for n in [10, 100, 500, 1000] for W in [3, 20, 50, 100]}
>>> t = benchit.timings(fns, in_, multivar=True, input_name=['nrows', 'ncols'])
>>> t.plot(logx=True, sp_ncols=2, sp_argID=0, sp_sharey='g')

For plotting, we are using number of rows as the x-axis base.

Use sp_argID=1 to switch-over to use number of cols as the x-axis base instead.

Single argument with groupings

Let’s manufacture a simple forward-filling scheme based on indices of True values in a boolean-array :

# Functions
def repeat(b):
    idx = np.flatnonzero(np.r_[b,True])
    return np.repeat(idx[:-1], np.diff(idx))

def maxaccum(b):
    return np.maximum.accumulate(np.where(b,np.arange(len(b)), 0))

in_ = {(n,sf): np.random.rand(n)<(100-sf)/100. for n in [100,1000,10000,100000,1000000] for sf in [20, 40, 60, 80, 90, 95]}
t = benchit.timings([repeat, maxaccum], in_, input_name=['Array-length','Sparseness %'])
t.plot(logx=True, sp_ncols=2, save='singlegrp_id0_ffillmask_timings.png')