MetricX

A library for managing, exploring, and analyzing benchmark data. Given a set of tasks and a set of models which can be evaluated on the tasks, MetricX provides a suite of features including:

• Monitoring and logging of modeling results.
• Export to CSV, matplotlib, bokeh, and more!
• Smart selection of the next model to evaluate.
• Interactive visualization in Jupyter notebooks.
• Interactive HTML reports (deployable via Github Pages).

Quick Start

Install MetricX

You can install the latest stable release

pip install metricx

Or you can install the development head

pip install git+https://github.com/k15z/MetricX.git

Define your task

First, you'll want to define your task(s). Every task has a name and a set of metrics.

from metricx import Metric, Task

task = Task(
    name="mnist-digits",
    metrics=[
        Metric(name="accuracy", is_higher_better=True),
        Metric(name="fit-time", is_higher_better=False),
        Metric(name="predict-time", is_higher_better=False),
    ],
)

Report your results

Then, you can report your results by providing (1) the name of the model being evaluated and (2) a results dictionary containing a value for each of the metrics specified earlier.

task.report("convnet", {
  "accuracy": 1.0, 
  "fit-time": 100.0,
  "predict-time": 3.0,
})

task.report("logistic-regression", {
  "accuracy": 0.6, 
  "fit-time": 10.0,
  "predict-time": 1.0,
})

Generate plots and rankings

The Task object provides numerous functionality from generating plots to ranking models.

task.to_bokeh() # generate a Bokeh plot
task.to_figure() # generate a matplotlib Figure
task.rank() # return a ranking of models

Combine multiple tasks

If you have multiple tasks, as is typical in a benchmarking scenario, you can use the TaskGrid to wrap them together and generated combined visualizations and more.

from metricx import TaskGrid

grid = TaskGrid([task])
grid.to_html("benchmark.html")

This will create a standalone HTML file which allows you to interactively explore the benchmark results.