PerMetrics: A Framework of Performance Metrics for Machine Learning Models
Project description
PerMetrics is a python library for performance metrics of machine learning models. We aim to implement all performance metrics for problems such as regression, classification, clustering, ... problems. Helping users in all field access metrics as fast as possible
- Free software: GNU General Public License (GPL) V3 license
- Total metrics: 103 (47 regression metrics, 16 classification metrics, 40 clustering metrics)
- Documentation: https://permetrics.readthedocs.io/en/latest/
- Python versions: >= 3.7.x
- Dependencies: numpy, scipy
Notification
- Currently, there is a huge misunderstanding among frameworks around the world about the notation of R, R2, and R^2.
- Please read the file R-R2-Rsquared.docx to understand the differences between them and why there is such confusion.
- My recommendation is to denote the Coefficient of Determination as COD or R2, while the squared Pearson's Correlation Coefficient should be denoted as R^2 or RSQ (as in Excel software).
Installation
Install with pip
Install the current PyPI release:
$ pip install permetrics==1.4.2
Or installing from the source code, use:
$ git clone https://github.com/thieu1995/permetrics.git
$ cd permetrics
$ python setup.py install
Or install the development version from GitHub:
pip install git+https://github.com/thieu1995/permetrics
After installation, you can import Permetrics as any other Python module:
$ python
>>> import permetrics
>>> permetrics.__version__
Let's go through some examples. The more complicated test case in the folder: examples
The documentation includes more detailed installation instructions and explanations.
Example with Regression metrics
import numpy as np
from permetrics import RegressionMetric
## For 1-D array
y_true = [3, -0.5, 2, 7]
y_pred = [2.5, 0.0, 2, 8]
evaluator = RegressionMetric(y_true, y_pred, decimal=5)
print(evaluator.RMSE())
print(evaluator.MSE())
## For > 1-D array
y_true = np.array([[0.5, 1], [-1, 1], [7, -6]])
y_pred = np.array([[0, 2], [-1, 2], [8, -5]])
evaluator = RegressionMetric(y_true, y_pred, decimal=5)
print(evaluator.RMSE(multi_output="raw_values", decimal=5))
print(evaluator.MAE(multi_output="raw_values", decimal=5))
Example with Classification metrics
from permetrics import ClassificationMetric
## For integer labels or categorical labels
y_true = [0, 1, 0, 0, 1, 0]
y_pred = [0, 1, 0, 0, 0, 1]
# y_true = ["cat", "ant", "cat", "cat", "ant", "bird", "bird", "bird"]
# y_pred = ["ant", "ant", "cat", "cat", "ant", "cat", "bird", "ant"]
evaluator = ClassificationMetric(y_true, y_pred, decimal=5)
## Call specific function inside object, each function has 3 names like below
print(evaluator.f1_score())
print(evaluator.F1S(average="micro"))
print(evaluator.F1S(average="macro"))
print(evaluator.F1S(average="weighted"))
Example with Clustering metrics
import numpy as np
from permetrics import ClusteringMetric
# generate sample data
X = np.random.uniform(-1, 10, size=(500, 7)) # 500 examples, 7 features
y_true = np.random.randint(0, 4, size=500) # 4 clusters
y_pred = np.random.randint(0, 4, size=500)
evaluator = ClusteringMetric(y_true=y_true, y_pred=y_pred, X=X, decimal=5)
## Call specific function inside object, each function has 2 names (fullname and short name)
## + Internal metrics: Need X and y_pred and has suffix as index
## + External metrics: Need y_true and y_pred and has suffix as score
print(evaluator.ball_hall_index())
print(evaluator.BHI())
Metrics
Problem | STT | Metric | Metric Fullname | Characteristics |
---|---|---|---|---|
Regression | 1 | EVS | Explained Variance Score | Bigger is better (Best = 1), Range=(-inf, 1.0] |
**** | 2 | ME | Max Error | Smaller is better (Best = 0), Range=[0, +inf) |
**** | 3 | MBE | Mean Bias Error | Best = 0, Range=(-inf, +inf) |
**** | 4 | MAE | Mean Absolute Error | Smaller is better (Best = 0), Range=[0, +inf) |
**** | 5 | MSE | Mean Squared Error | Smaller is better (Best = 0), Range=[0, +inf) |
**** | 6 | RMSE | Root Mean Squared Error | Smaller is better (Best = 0), Range=[0, +inf) |
**** | 7 | MSLE | Mean Squared Log Error | Smaller is better (Best = 0), Range=[0, +inf) |
**** | 8 | MedAE | Median Absolute Error | Smaller is better (Best = 0), Range=[0, +inf) |
**** | 9 | MRE / MRB | Mean Relative Error / Mean Relative Bias | Smaller is better (Best = 0), Range=[0, +inf) |
**** | 10 | MPE | Mean Percentage Error | Best = 0, Range=(-inf, +inf) |
**** | 11 | MAPE | Mean Absolute Percentage Error | Smaller is better (Best = 0), Range=[0, +inf) |
**** | 12 | SMAPE | Symmetric Mean Absolute Percentage Error | Smaller is better (Best = 0), Range=[0, 1] |
**** | 13 | MAAPE | Mean Arctangent Absolute Percentage Error | Smaller is better (Best = 0), Range=[0, +inf) |
**** | 14 | MASE | Mean Absolute Scaled Error | Smaller is better (Best = 0), Range=[0, +inf) |
**** | 15 | NSE | Nash-Sutcliffe Efficiency Coefficient | Bigger is better (Best = 1), Range=(-inf, 1] |
**** | 16 | NNSE | Normalized Nash-Sutcliffe Efficiency Coefficient | Bigger is better (Best = 1), Range=[0, 1] |
**** | 17 | WI | Willmott Index | Bigger is better (Best = 1), Range=[0, 1] |
**** | 18 | R / PCC | Pearson’s Correlation Coefficient | Bigger is better (Best = 1), Range=[-1, 1] |
**** | 19 | AR / APCC | Absolute Pearson's Correlation Coefficient | Bigger is better (Best = 1), Range=[-1, 1] |
**** | 20 | RSQ/R2S | (Pearson’s Correlation Index) ^ 2 | Bigger is better (Best = 1), Range=[0, 1] |
**** | 21 | R2 / COD | Coefficient of Determination | Bigger is better (Best = 1), Range=(-inf, 1] |
**** | 22 | AR2 / ACOD | Adjusted Coefficient of Determination | Bigger is better (Best = 1), Range=(-inf, 1] |
**** | 23 | CI | Confidence Index | Bigger is better (Best = 1), Range=(-inf, 1] |
**** | 24 | DRV | Deviation of Runoff Volume | Smaller is better (Best = 1.0), Range=[1, +inf) |
**** | 25 | KGE | Kling-Gupta Efficiency | Bigger is better (Best = 1), Range=(-inf, 1] |
**** | 26 | GINI | Gini Coefficient | Smaller is better (Best = 0), Range=[0, +inf) |
**** | 27 | GINI_WIKI | Gini Coefficient on Wikipage | Smaller is better (Best = 0), Range=[0, +inf) |
**** | 28 | PCD | Prediction of Change in Direction | Bigger is better (Best = 1.0), Range=[0, 1] |
**** | 29 | CE | Cross Entropy | Range(-inf, 0], Can't give comment about this |
**** | 30 | KLD | Kullback Leibler Divergence | Best = 0, Range=(-inf, +inf) |
**** | 31 | JSD | Jensen Shannon Divergence | Smaller is better (Best = 0), Range=[0, +inf) |
**** | 32 | VAF | Variance Accounted For | Bigger is better (Best = 100%), Range=(-inf, 100%] |
**** | 33 | RAE | Relative Absolute Error | Smaller is better (Best = 0), Range=[0, +inf) |
**** | 34 | A10 | A10 Index | Bigger is better (Best = 1), Range=[0, 1] |
**** | 35 | A20 | A20 Index | Bigger is better (Best = 1), Range=[0, 1] |
**** | 36 | A30 | A30 Index | Bigger is better (Best = 1), Range=[0, 1] |
**** | 37 | NRMSE | Normalized Root Mean Square Error | Smaller is better (Best = 0), Range=[0, +inf) |
**** | 38 | RSE | Residual Standard Error | Smaller is better (Best = 0), Range=[0, +inf) |
**** | 39 | RE / RB | Relative Error / Relative Bias | Best = 0, Range=(-inf, +inf) |
**** | 40 | AE | Absolute Error | Best = 0, Range=(-inf, +inf) |
**** | 41 | SE | Squared Error | Smaller is better (Best = 0), Range=[0, +inf) |
**** | 42 | SLE | Squared Log Error | Smaller is better (Best = 0), Range=[0, +inf) |
**** | 43 | COV | Covariance | Bigger is better (No best value), Range=(-inf, +inf) |
**** | 44 | COR | Correlation | Bigger is better (Best = 1), Range=[-1, +1] |
**** | 45 | EC | Efficiency Coefficient | Bigger is better (Best = 1), Range=(-inf, +1] |
**** | 46 | OI | Overall Index | Bigger is better (Best = 1), Range=(-inf, +1] |
**** | 47 | CRM | Coefficient of Residual Mass | Smaller is better (Best = 0), Range=(-inf, +inf) |
**** | **** | **** | **** | **** |
Classification | 1 | PS | Precision Score | Bigger is better (Best = 1), Range = [0, 1] |
**** | 2 | NPV | Negative Predictive Value | Bigger is better (Best = 1), Range = [0, 1] |
**** | 3 | RS | Recall Score | Bigger is better (Best = 1), Range = [0, 1] |
**** | 4 | AS | Accuracy Score | Bigger is better (Best = 1), Range = [0, 1] |
**** | 5 | F1S | F1 Score | Bigger is better (Best = 1), Range = [0, 1] |
**** | 6 | F2S | F2 Score | Bigger is better (Best = 1), Range = [0, 1] |
**** | 7 | FBS | F-Beta Score | Bigger is better (Best = 1), Range = [0, 1] |
**** | 8 | SS | Specificity Score | Bigger is better (Best = 1), Range = [0, 1] |
**** | 9 | MCC | Matthews Correlation Coefficient | Bigger is better (Best = 1), Range = [-1, +1] |
**** | 10 | HS | Hamming Score | Bigger is better (Best = 1), Range = [0, 1] |
**** | 11 | CKS | Cohen's kappa score | Bigger is better (Best = +1), Range = [-1, +1] |
**** | 12 | JSI | Jaccard Similarity Coefficient | Bigger is better (Best = +1), Range = [0, +1] |
**** | 13 | GMS | Geometric Mean Score | Bigger is better (Best = +1), Range = [0, +1] |
**** | 14 | ROC-AUC | ROC-AUC | Bigger is better (Best = +1), Range = [0, +1] |
**** | 15 | LS | Lift Score | Bigger is better (No best value), Range = [0, +inf) |
**** | 16 | GINI | GINI Index | Smaller is better (Best = 0), Range = [0, +1] |
**** | **** | **** | **** | **** |
Clustering | 1 | BHI | Ball Hall Index | Smaller is better (Best = 0), Range=[0, +inf) |
**** | 2 | XBI | Xie Beni Index | Smaller is better (Best = 0), Range=[0, +inf) |
**** | 3 | DBI | Davies Bouldin Index | Smaller is better (Best = 0), Range=[0, +inf) |
**** | 4 | BRI | Banfeld Raftery Index | Smaller is better (No best value), Range=(-inf, inf) |
**** | 5 | KDI | Ksq Detw Index | Smaller is better (No best value), Range=(-inf, +inf) |
**** | 6 | DRI | Det Ratio Index | Bigger is better (No best value), Range=[0, +inf) |
**** | 7 | DI | Dunn Index | Bigger is better (No best value), Range=[0, +inf) |
**** | 8 | CHI | Calinski Harabasz Index | Bigger is better (No best value), Range=[0, inf) |
**** | 9 | LDRI | Log Det Ratio Index | Bigger is better (No best value), Range=(-inf, +inf) |
**** | 10 | LSRI | Log SS Ratio Index | Bigger is better (No best value), Range=(-inf, +inf) |
**** | 11 | SI | Silhouette Index | Bigger is better (Best = 1), Range = [-1, +1] |
**** | 12 | SSEI | Sum of Squared Error Index | Smaller is better (Best = 0), Range = [0, +inf) |
**** | 13 | DHI | Duda-Hart Index | Smaller is better (Best = 0), Range = [0, +inf) |
**** | 14 | BI | Beale Index | Smaller is better (Best = 0), Range = [0, +inf) |
**** | 15 | RSI | R-squared Index | Bigger is better (Best=1), Range = (-inf, 1] |
**** | 16 | DBCVI | Density-based Clustering Validation Index | Bigger is better (Best=0), Range = [0, 1] |
**** | 17 | HI | Hartigan Index | Bigger is better (best=0), Range = [0, +inf) |
**** | 18 | MIS | Mutual Info Score | Bigger is better (No best value), Range = [0, +inf) |
**** | 19 | NMIS | Normalized Mutual Info Score | Bigger is better (Best = 1), Range = [0, 1] |
**** | 20 | RaS | Rand Score | Bigger is better (Best = 1), Range = [0, 1] |
**** | 21 | FMS | Fowlkes Mallows Score | Bigger is better (Best = 1), Range = [0, 1] |
**** | 22 | HS | Homogeneity Score | Bigger is better (Best = 1), Range = [0, 1] |
**** | 23 | CS | Completeness Score | Bigger is better (Best = 1), Range = [0, 1] |
**** | 24 | VMS | V-Measure Score | Bigger is better (Best = 1), Range = [0, 1] |
**** | 25 | PrS | Precision Score | Bigger is better (Best = 1), Range = [0, 1] |
**** | 26 | ReS | Recall Score | Bigger is better (Best = 1), Range = [0, 1] |
**** | 27 | FmS | F-Measure Score | Bigger is better (Best = 1), Range = [0, 1] |
**** | 28 | CDS | Czekanowski Dice Score | Bigger is better (Best = 1), Range = [0, 1] |
**** | 29 | HGS | Hubert Gamma Score | Bigger is better (Best = 1), Range=[-1, +1] |
**** | 30 | JS | Jaccard Score | Bigger is better (Best = 1), Range = [0, 1] |
**** | 31 | KS | Kulczynski Score | Bigger is better (Best = 1), Range = [0, 1] |
**** | 32 | MNS | Mc Nemar Score | Bigger is better (No best value), Range=(-inf, +inf) |
**** | 33 | PhS | Phi Score | Bigger is better (No best value), Range = (-inf, +inf) |
**** | 34 | RTS | Rogers Tanimoto Score | Bigger is better (Best = 1), Range = [0, 1] |
**** | 35 | RRS | Russel Rao Score | Bigger is better (Best = 1), Range = [0, 1] |
**** | 36 | SS1S | Sokal Sneath1 Score | Bigger is better (Best = 1), Range = [0, 1] |
**** | 37 | SS2S | Sokal Sneath2 Score | Bigger is better (Best = 1), Range = [0, 1] |
**** | 38 | PuS | Purity Score | Bigger is better (Best = 1), Range = [0, 1] |
**** | 39 | ES | Entropy Score | Smaller is better (Best = 0), Range = [0, 1] |
**** | 40 | TS | Tau Score | Bigger is better (Best = 1), Range = [-1, 1] |
**** | **** | **** | **** | **** |
Support (questions, problems)
Official channels
-
Official source code repo: https://github.com/thieu1995/permetrics
-
Official document: https://permetrics.readthedocs.io/
-
Download releases: https://pypi.org/project/permetrics/
-
Issue tracker: https://github.com/thieu1995/permetrics/issues
-
Notable changes log: https://github.com/thieu1995/permetrics/blob/master/ChangeLog.md
-
Official chat group: https://t.me/+fRVCJGuGJg1mNDg1
-
This project also related to our another projects which are "optimization" and "machine learning", check it here:
Citation Request
Please include these citations if you plan to use this library:
@software{thieu_nguyen_2020_3951205,
author = {Nguyen Van Thieu},
title = {Permetrics: A framework of performance metrics for artificial intelligence models},
month = jul,
year = 2020,
publisher = {Zenodo},
doi = {10.5281/zenodo.3951205},
url = {https://doi.org/10.5281/zenodo.3951205}
}
@article{van2023mealpy,
title={MEALPY: An open-source library for latest meta-heuristic algorithms in Python},
author={Van Thieu, Nguyen and Mirjalili, Seyedali},
journal={Journal of Systems Architecture},
year={2023},
publisher={Elsevier},
doi={10.1016/j.sysarc.2023.102871}
}
Related Documents
- https://www.debadityachakravorty.com/ai-ml/cmatrix/
- https://neptune.ai/blog/evaluation-metrics-binary-classification
- https://danielyang1009.github.io/model-performance-measure/
- https://towardsdatascience.com/multi-class-metrics-made-simple-part-i-precision-and-recall-9250280bddc2
- http://cran.nexr.com/web/packages/clusterCrit/vignettes/clusterCrit.pdf
- https://torchmetrics.readthedocs.io/en/latest/
- http://rasbt.github.io/mlxtend/user_guide/evaluate/lift_score/
- https://www.baeldung.com/cs/multi-class-f1-score
- https://kavita-ganesan.com/how-to-compute-precision-and-recall-for-a-multi-class-classification-problem/#.YoXMSqhBy3A
- https://machinelearningmastery.com/precision-recall-and-f-measure-for-imbalanced-classification/
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
Hashes for permetrics-1.4.2-py3-none-any.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | ee6c44cbfa37be88cd21ad74055748ca7652f31f082db81d6f4b38ce0ad1f694 |
|
MD5 | 4a8f116f7d87f9997caf875c8f261cf8 |
|
BLAKE2b-256 | 2eb8eb91ee1127b8d8046b8654cc8c153ecb10d84a1e747c952eee84cbcd8135 |