This library is intended to make it easy to write small benchmarks and view the results.

• Usage
• API
  • Benchmark
    • benchmark
    • run
    • print_results
    • print_stats
    • save_results
  • cleanup
  • default_args
    • bench
    • stats
    • save
  • Timer
  • Commit
• Developer Notes

Usage

See examples/basic.py for a full working example.

from pyarkbench import Benchmark, Timer, default_args

class Basic(Benchmark):
    def benchmark(self):
        with Timer() as m1:
            # Do some stuff
            pass

        with Timer() as m2:
            # Do some other stuff
            pass

        return {
            "Metric 1 (ms)": m1.ms_duration,
            "Metric 2 (ms)": m2.ms_duration,
        }

if __name__ == '__main__':
    # Initialize the benchmark and use the default command line args
    bench = Basic(*default_args.bench())

    # Run the benchmark (will run your code in `benchmark` many times, some to warm up and then some where the timer results are save)
    results = bench.run()

    # View the raw results
    bench.print_results(results)

    # See aggregate statistics about the results
    bench.print_stats(results, stats=default_args.stats())

    # Save the results to a JSON file named based on the benchmark class
    bench.save_results(results, out_dir=default_args.save())

API

Benchmark

Benchmark(self, num_runs: int = 10, warmup_runs: int = 1, quiet: bool = False, commit: pybench.benchmarking_utils.Commit = None)

Benchmarks should extend this class and implement the benchmark method.

benchmark

Benchmark.benchmark(self) -> Dict[str, float]

This method must be implemented in your subclass and returns a dictionary of metric name to the time captured for that metric.

run

Benchmark.run(self) -> Dict[str, Any]

This is the entry point into your benchmark. It will first run benchmark() self.warmup_runs times without using the resulting timings, then it will run benchmark() self.num_runs times and return the resulting timings.

print_results

Benchmark.print_results(self, results)

Pretty print the raw results by JSON dumping them.

print_stats

Benchmark.print_stats(self, results, stats=('mean', 'median', 'variance'))

Collects and prints statistics over the results.

save_results

Benchmark.save_results(self, results, out_dir, filename=None)

Save the results gathered from benchmarking and metadata about the commit to a JSON file named after the type of self.

cleanup

cleanup()

Churn through a bunch of data, run the garbage collector, and sleep for a second to "reset" the Python interpreter.

default_args

default_args(self, /, *args, **kwargs)

Adds a bunch of default command line arguments to make orchestrating benchmark runs more convenient. To see all the options, call default_args.init() and run the script with the --help option.

bench

default_args.bench()

Default arguments to be passed to a Benchmark object

stats

default_args.stats()

Default arguments to be passed to the Benchmark.print_stats method

save

default_args.save()

Default arguments to be passed to the Benchmark.save_results method

Timer

Timer(self, /, *args, **kwargs)

Context manager object that will time the execution of the statements it manages. self.start - start time self.end - end time self.ms_duration - end - start / 1000 / 1000

Commit

Commit(self, time, pr, hash)

Wrapper around a git commit

Developer Notes

To build this package locally, check it out and run

python setup.py develop

To rebuild these docs, run

pip install pydoc-markdown
pydocmd simple pybench.Benchmark+ pybench.cleanup pybench.default_args+ pybench.Timer pybench.Commit