permetrics 1.0.4

A framework of PERformance METRICS (PerMetrics) for machine learning models

A framework of PERformance METRICS (PerMetrics) for artificial intelligence models

---

"Knowledge is power, sharing it is the premise of progress in life. It seems like a burden to someone, but it is the only way to achieve immortality." --- Thieu Nguyen

---

Introduction

• PerMetrics is a python library for performance metrics of machine learning models.

• The goals of this framework are:

  • Combine all metrics for regression, classification and clustering models
  • Helping users in all field access to metrics as fast as possible
  • Perform Qualitative Analysis of models.
  • Perform Quantitative Analysis of models.

• Metrics

Problem	STT	Metric	Metric Fullname	Characteristics
Regression	1	EVS	Explained Variance Score	Larger is better (Best = 1)
	2	ME	Max Error	Smaller is better (Best = 0)
	3	MAE	Mean Absolute Error	Smaller is better (Best = 0)
	4	MSE	Mean Squared Error	Smaller is better (Best = 0)
	5	RMSE	Root Mean Squared Error	Smaller is better (Best = 0)
	6	MSLE	Mean Squared Log Error	Smaller is better (Best = 0)
	7	MedAE	Median Absolute Error	Smaller is better (Best = 0)
	8	MRE	Mean Relative Error	Smaller is better (Best = 0)
	9	MAPE	Mean Absolute Percentage Error	Smaller is better (Best = 0)
	10	SMAPE	Symmetric Mean Absolute Percentage Error	Smaller is better (Best = 0)
	11	MAAPE	Mean Arctangent Absolute Percentage Error	Smaller is better (Best = 0)
	12	MASE	Mean Absolute Scaled Error	Smaller is better (Best = 0)
	13	NSE	Nash-Sutcliffe Efficiency Coefficient	Larger is better (Best = 1)
	14	WI	Willmott Index	Larger is better (Best = 1)
	15	R	Pearson’s Correlation Index	Larger is better (Best = 1)
	16	CI	Confidence Index	Larger is better (Best = 1)
	17	R2	Coefficient of Determination	Larger is better (Best = 1)
	18	R2s	(Pearson’s Correlation Index) ^ 2	Larger is better (Best = 1)
	19	DRV	Deviation of Runoff Volume	Smaller is better (Best = 0)
	20	KGE	Kling-Gupta Efficiency	Larger is better (Best = 1)
	21
Single Loss	1	RE	Relative error	Smaller is better (Best = 0)
	2	AE	Absolute error	Smaller is better (Best = 0)
	3	SE	Squared error	Smaller is better (Best = 0)
	4	SLE	Squared log error	Smaller is better (Best = 0)
	5	LL	Log likelihood	Smaller is better (Best = 0)
	6
Classification	1	MLL	Mean Log Likelihood	Smaller is better (Best = 0)
	2
Clustering	1
	2

Dependencies

• Python (>= 3.6)
• Numpy (>= 1.15.1)

User installation

Install the current PyPI release:

pip install permetrics

Or install the development version from GitHub:

pip install git+https://github.com/thieu1995/permetrics

Example

• All you need to do is: (Make sure your y_true and y_pred is a numpy array)

* Simple example:

## For example with RMSE:

    from numpy import array
    from permetrics.regression import Metrics

    ## For 1-D array
    y_true = array([3, -0.5, 2, 7])
    y_pred = array([2.5, 0.0, 2, 8])

    obj1 = Metrics(y_true, y_pred)
    print(obj1.rmse_func(clean=True, decimal=5))

    ## For > 1-D array
    y_true = array([[0.5, 1], [-1, 1], [7, -6]])
    y_pred = array([[0, 2], [-1, 2], [8, -5]])

    multi_outputs = [None, "raw_values", [0.3, 1.2], array([0.5, 0.2]), (0.1, 0.9)]
    obj2 = Metrics(y_true, y_pred)
    for multi_output in multi_outputs:
        print(obj2.rmse_func(clean=False, multi_output=multi_output, decimal=5))

* Or run the simple:
    python examples/RMSE.py

* The more complicated tests in the folder: examples

The documentation includes more detailed installation instructions and explanations.

Changelog

• See the ChangeLog.md for a history of notable changes to permetrics.

Important links

• Official source code repo: https://github.com/thieu1995/permetrics

• Official documentation: https://permetrics.readthedocs.io/

• Download releases: https://pypi.org/project/permetrics/

• Issue tracker: https://github.com/thieu1995/permetrics/issues

• This project also related to my another projects which are "meta-heuristics" and "neural-network", check it here

  • https://github.com/thieu1995/opfunu
  • https://github.com/thieu1995/metaheuristics
  • https://github.com/chasebk

Contributions

Citation

• If you use permetrics in your project, please cite my works:

@software{thieu_nguyen_2020_3951205,
  author       = {Thieu Nguyen},
  title        = {A framework of PERformance METRICS (PerMetrics) for artificial intelligence models},
  month        = jul,
  year         = 2020,
  publisher    = {Zenodo},
  doi          = {10.5281/zenodo.3951205},
  url          = {https://doi.org/10.5281/zenodo.3951205}
}

@article{nguyen2019efficient,
  title={Efficient Time-Series Forecasting Using Neural Network and Opposition-Based Coral Reefs Optimization},
  author={Nguyen, Thieu and Nguyen, Tu and Nguyen, Binh Minh and Nguyen, Giang},
  journal={International Journal of Computational Intelligence Systems},
  volume={12},
  number={2},
  pages={1144--1161},
  year={2019},
  publisher={Atlantis Press}
}

Future works

Classification

• F1 score
• Multiclass log loss
• Lift
• Average Precision for binary classification
• precision / recall break-even point
• cross-entropy
• True Pos / False Pos / True Neg / False Neg rates
• precision / recall / sensitivity / specificity
• mutual information

HIGHER LEVEL TRANSFORMATIONS TO HANDLE

• GroupBy / Reduce
• Weight individual samples or groups

PROPERTIES METRICS CAN HAVE

• Min or Max (optimize through minimization or maximization)
• Binary Classification
  • Scores predicted class labels
  • Scores predicted ranking (most likely to least likely for being in one class)
  • Scores predicted probabilities
• Multiclass Classification
  • Scores predicted class labels
  • Scores predicted probabilities
• Regression (More)
• Discrete Rater Comparison (confusion matrix)