metriculous 0.2.1

Very unstable library containing utilities to measure and visualize statistical properties of machine learning models.

metriculous

Unstable python library with utilities to measure, visualize and compare statistical properties of machine learning models. Breaking improvements to be expected.

Installation

$ pip install metriculous

Or, for the latest unreleased version:

$ pip install git+https://github.com/metriculous-ml/metriculous.git

Or, to avoid getting surprised by breaking changes:

$ pip install git+https://github.com/metriculous-ml/metriculous.git@YourFavoriteCommit

Usage

Comparing Regression Models

import numpy as np

# Mock the ground truth, a one-dimensional array of floats
ground_truth = np.random.random(300)

# Mock the output of a few models
perfect_model = ground_truth
noisy_model = ground_truth + 0.1 * np.random.randn(*ground_truth.shape)
random_model = np.random.randn(*ground_truth.shape)
zero_model = np.zeros_like(ground_truth)

import metriculous

metriculous.compare_regressors(
    ground_truth=ground_truth,
    model_predictions=[perfect_model, noisy_model, random_model, zero_model],
    model_names=["Perfect Model", "Noisy Model", "Random Model", "Zero Model"],
).save_html("comparison.html").display()

This will save an HTML file with common regression metrics and charts, and if you are working in a Jupyter notebook will display the output right in front of you:

Comparing Classification Models

For an example that evaluates and compares classifiers, please refer to the quickstart notebook for classification.

Development

Poetry

This project uses poetry to manage dependencies. Please make sure it is installed for the required python version. Then install the dependencies with poetry install.

Makefile

A Makefile is used to automate common development workflows. Type make or make help to see a list of available commands. Before commiting changes it is recommended to run make format check test.