A framework of PERformance METRICS (PerMetrics) for machine learning models
Project description
A framework of PERformance METRICS (PerMetrics) for artificial intelligence models
"Knowledge is power, sharing it is the premise of progress in life. It seems like a burden to someone, but it is the only way to achieve immortality." --- Thieu Nguyen
Introduction
-
PerMetrics is a python library for performance metrics of machine learning models.
-
The goals of this framework are:
- Combine all metrics for regression, classification and clustering models
- Helping users in all field access to metrics as fast as possible
- Perform Qualitative Analysis of models.
- Perform Quantitative Analysis of models.
-
Metrics
Problem | STT | Metric | Metric Fullname | Characteristics |
---|---|---|---|---|
Regression | 1 | EVS | Explained Variance Score | Larger is better (Best = 1) |
2 | ME | Max Error | Smaller is better (Best = 0) | |
3 | MAE | Mean Absolute Error | Smaller is better (Best = 0) | |
4 | MSE | Mean Squared Error | Smaller is better (Best = 0) | |
5 | RMSE | Root Mean Squared Error | Smaller is better (Best = 0) | |
6 | MSLE | Mean Squared Log Error | Smaller is better (Best = 0) | |
7 | MedAE | Median Absolute Error | Smaller is better (Best = 0) | |
8 | MRE | Mean Relative Error | Smaller is better (Best = 0) | |
9 | MAPE | Mean Absolute Percentage Error | Smaller is better (Best = 0) | |
10 | SMAPE | Symmetric Mean Absolute Percentage Error | Smaller is better (Best = 0) | |
11 | MAAPE | Mean Arctangent Absolute Percentage Error | Smaller is better (Best = 0) | |
12 | MASE | Mean Absolute Scaled Error | Smaller is better (Best = 0) | |
13 | NSE | Nash-Sutcliffe Efficiency Coefficient | Larger is better (Best = 1) | |
14 | WI | Willmott Index | Larger is better (Best = 1) | |
15 | R | Pearson’s Correlation Index | Larger is better (Best = 1) | |
16 | CI | Confidence Index | Larger is better (Best = 1) | |
17 | R2 | Coefficient of Determination | Larger is better (Best = 1) | |
18 | R2s | (Pearson’s Correlation Index) ^ 2 | Larger is better (Best = 1) | |
19 | DRV | Deviation of Runoff Volume | Smaller is better (Best = 0) | |
20 | KGE | Kling-Gupta Efficiency | Larger is better (Best = 1) | |
21 | ||||
Single Loss | 1 | RE | Relative error | Smaller is better (Best = 0) |
2 | AE | Absolute error | Smaller is better (Best = 0) | |
3 | SE | Squared error | Smaller is better (Best = 0) | |
4 | SLE | Squared log error | Smaller is better (Best = 0) | |
5 | LL | Log likelihood | Smaller is better (Best = 0) | |
6 | ||||
Classification | 1 | MLL | Mean Log Likelihood | Smaller is better (Best = 0) |
2 | ||||
Clustering | 1 | |||
2 |
Dependencies
- Python (>= 3.6)
- Numpy (>= 1.15.1)
User installation
Install the current PyPI release:
pip install permetrics
Or install the development version from GitHub:
pip install git+https://github.com/thieu1995/permetrics
Example
- All you need to do is: (Make sure your y_true and y_pred is a numpy array)
* Simple example:
## For example with RMSE:
from numpy import array
from permetrics.regression import Metrics
## For 1-D array
y_true = array([3, -0.5, 2, 7])
y_pred = array([2.5, 0.0, 2, 8])
obj1 = Metrics(y_true, y_pred)
print(obj1.rmse_func(clean=True, decimal=5))
## For > 1-D array
y_true = array([[0.5, 1], [-1, 1], [7, -6]])
y_pred = array([[0, 2], [-1, 2], [8, -5]])
multi_outputs = [None, "raw_values", [0.3, 1.2], array([0.5, 0.2]), (0.1, 0.9)]
obj2 = Metrics(y_true, y_pred)
for multi_output in multi_outputs:
print(obj2.rmse_func(clean=False, multi_output=multi_output, decimal=5))
* Or run the simple:
python examples/RMSE.py
* The more complicated tests in the folder: examples
The documentation includes more detailed installation instructions and explanations.
Changelog
- See the ChangeLog.md for a history of notable changes to permetrics.
Important links
-
Official source code repo: https://github.com/thieu1995/permetrics
-
Official documentation: https://permetrics.readthedocs.io/
-
Download releases: https://pypi.org/project/permetrics/
-
Issue tracker: https://github.com/thieu1995/permetrics/issues
-
This project also related to my another projects which are "meta-heuristics" and "neural-network", check it here
Contributions
Citation
- If you use permetrics in your project, please cite my works:
@software{thieu_nguyen_2020_3951205,
author = {Thieu Nguyen},
title = {A framework of PERformance METRICS (PerMetrics) for artificial intelligence models},
month = jul,
year = 2020,
publisher = {Zenodo},
doi = {10.5281/zenodo.3951205},
url = {https://doi.org/10.5281/zenodo.3951205}
}
@article{nguyen2019efficient,
title={Efficient Time-Series Forecasting Using Neural Network and Opposition-Based Coral Reefs Optimization},
author={Nguyen, Thieu and Nguyen, Tu and Nguyen, Binh Minh and Nguyen, Giang},
journal={International Journal of Computational Intelligence Systems},
volume={12},
number={2},
pages={1144--1161},
year={2019},
publisher={Atlantis Press}
}
Future works
Classification
- F1 score
- Multiclass log loss
- Lift
- Average Precision for binary classification
- precision / recall break-even point
- cross-entropy
- True Pos / False Pos / True Neg / False Neg rates
- precision / recall / sensitivity / specificity
- mutual information
HIGHER LEVEL TRANSFORMATIONS TO HANDLE
- GroupBy / Reduce
- Weight individual samples or groups
PROPERTIES METRICS CAN HAVE
- Min or Max (optimize through minimization or maximization)
- Binary Classification
- Scores predicted class labels
- Scores predicted ranking (most likely to least likely for being in one class)
- Scores predicted probabilities
- Multiclass Classification
- Scores predicted class labels
- Scores predicted probabilities
- Regression (More)
- Discrete Rater Comparison (confusion matrix)
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
Hashes for permetrics-1.0.4-py3-none-any.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 681c5ce6522d55a5cc374ae05b311054e677374a86a8c4391124dd7cb1fabd34 |
|
MD5 | d471255d76a2b37fc4c80d531e3b5b52 |
|
BLAKE2b-256 | b607bea17dc732fcd498e2368abfd0e6265e96701720693000acb4080d4fdddb |