Testing and Reporting framework for Time Series Analysis
Project description
KRISI
(/creesee/)
Testing and Reporting Framework for Time Series Analysis
View Demo ~
Check Examples ~
Explore the docs »
Krisi is a Scoring library for Time-Series Forecasting. It calculates, stores and vizualises the performance of your predictions!
Krisi is from the ground-up extensible and lightweight and comes with the fundamental metrics for regression and classification.
It can generate reports in:
- static PDF (with
plotly) - interactive HTML (with
plotly) - pretty formatted for console (with
richandplotext)
Krisi solves the following problems
- Most TS libraries attach reporting to modelling (eg.: Darts, Statsmodel).
→ Krisi is independent of any modelling method or library. - Extendability is tedious: only works by subclassing objects.
→ Krisi supports easy configuration of custom metrics along with an extensive library of predefined metrics. - Missing Rolling window based evaluation.
→ Krisi supports evaluating metrics over time. - Too many dependencies.
→ Krisi has few hard dependencies (only core libarries, eg.: sklearn and plotting libraries). - Visualisation results are too basic.
→ With Krisi you can decide to share and interactive HTML, a static PDF or quickly look at results pretty printed to the console.
Installation
The project was entirely built in python.
Prerequisites
python >= 3.7andpip
Then run:
pip install krisi
If you'd like to also use interactive plotting (html) and pdf generation then run:
pip install krisi "krisi[plotting]"
Quickstart
You can quickly evaluate your predictions by running:
import numpy as np
from krisi.evaluate import score
score(y=np.random.rand(1000), predictions=np.random.rand(1000)).print()
Krisi's main object is the ScoreCard that contains predefined Metrics and which you can add further Metrics to.
from krisi.evaluate import ScoreCard
import numpy as np
# Random targets and predictions for Demo
target, predictions = np.random.rand(1000), np.random.rand(1000)
sc = ScoreCard(target, predictions)
# Calculate predefined metrics
sc.evaluate(defaults=True)
# Add a new metric
sc["own_metric"] = (target - predictions).mean()
# Print the result
sc.print()
Outputs:
┏━━━━━━━━━━━━━━━━━━━━━━━━ Result of <your_model_name> on <your_dataset_name> tested on insample ━━━━━━━━━━━━━━━━━━━━━━━━━┓
┃ ┃
┃ Residual Diagnostics ┃
┃ ╭───────────────────────────┬─────────────────────────────────────────────────────────────┬──────────────────────────╮ ┃
┃ │ Metric Name │ Result │ Parameters │ ┃
┃ ├───────────────────────────┼─────────────────────────────────────────────────────────────┼──────────────────────────┤ ┃
┃ │ Mean of the Residuals │ 0.007 │ {} │ ┃
┃ │ (residuals_mean) │ │ │ ┃
┃ ├───────────────────────────┼─────────────────────────────────────────────────────────────┼──────────────────────────┤ ┃
┃ │ Standard Deviation of the │ 0.409 │ {} │ ┃
┃ │ Residuals (residuals_std) │ │ │ ┃
┃ ╰───────────────────────────┴─────────────────────────────────────────────────────────────┴──────────────────────────╯ ┃
┃ Forecast Errors - Regression ┃
┃ ╭───────────────────────────┬─────────────────────────────────────────────────────────────┬──────────────────────────╮ ┃
┃ │ Mean Absolute Error (mae) │ 0.332 │ {} │ ┃
┃ ├───────────────────────────┼─────────────────────────────────────────────────────────────┼──────────────────────────┤ ┃
┃ │ Mean Absolute Percentage │ 2.85 │ {} │ ┃
┃ │ Error (mape) │ │ │ ┃
┃ ├───────────────────────────┼─────────────────────────────────────────────────────────────┼──────────────────────────┤ ┃
┃ │ Mean Squared Error (mse) │ 0.168 │ {'squared': True} │ ┃
┃ ├───────────────────────────┼─────────────────────────────────────────────────────────────┼──────────────────────────┤ ┃
┃ │ Root Mean Squared Error │ 0.41 │ {'squared': False} │ ┃
┃ │ (rmse) │ │ │ ┃
┃ ├───────────────────────────┼─────────────────────────────────────────────────────────────┼──────────────────────────┤ ┃
┃ │ Root Mean Squared Log │ 0.281 │ {'squared': False} │ ┃
┃ │ Error (rmsle) │ │ │ ┃
┃ ├───────────────────────────┼─────────────────────────────────────────────────────────────┼──────────────────────────┤ ┃
┃ │ R-squared (r2) │ -0.923 │ {} │ ┃
┃ ╰───────────────────────────┴─────────────────────────────────────────────────────────────┴──────────────────────────╯ ┃
┃ Unknown ┃
┃ ╭───────────────────────────┬─────────────────────────────────────────────────────────────┬──────────────────────────╮ ┃
┃ │ own_metric (own_metric) │ 0.007 │ {} │ ┃
┃ ╰───────────────────────────┴─────────────────────────────────────────────────────────────┴──────────────────────────╯ ┃
┃ ┃
┗━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┛
Creating more sophisticated Metrics with metadata.
import numpy as np
from krisi.evaluate import Metric, MetricCategories, ScoreCard
# Random targets and predictions for Demo
target, predictions = np.random.rand(100), np.random.rand(100)
# Create ScoreCard
sc = ScoreCard(target, predictions)
# Calculate a random metric for Demo
calculated_metric_example = (target - predictions).mean()
# Adding a simple new metric (a float)
# As a Dictionary:
sc["metric_barebones"] = calculated_metric_example
# As an Object assignment:
sc.another_metric_barebones = calculated_metric_example * 2.0
sc["metric_with_metadata"] = Metric(
name="A new, own Metric",
category=MetricCategories.residual,
result=calculated_metric_example * 3.0,
parameters={"hyper_1": 5.0},
)
# Updating the metadata of an existing metric
sc.metric_barebones = dict(info="Giving description to a metric")
# Print a pretty summary to the console
sc.print(with_info=True)
Outputs:
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ Result of <your_model_name> on <your_dataset_name> tested on insample ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
┃ ┃
┃ Residual Diagnostics ┃
┃ ╭─────────────────────┬────────────────────────────────────────────────┬────────────────────┬─────────────────────────────────╮ ┃
┃ │ Metric Name │ Result │ Parameters │ Info │ ┃
┃ ├─────────────────────┼────────────────────────────────────────────────┼────────────────────┼─────────────────────────────────┤ ┃
┃ │ Mean of the │ 0.035 │ {} │ '' │ ┃
┃ │ Residuals │ │ │ │ ┃
┃ │ (residuals_mean) │ │ │ │ ┃
┃ ├─────────────────────┼────────────────────────────────────────────────┼────────────────────┼─────────────────────────────────┤ ┃
┃ │ Standard Deviation │ 0.42 │ {} │ '' │ ┃
┃ │ of the Residuals │ │ │ │ ┃
┃ │ (residuals_std) │ │ │ │ ┃
┃ ├─────────────────────┼────────────────────────────────────────────────┼────────────────────┼─────────────────────────────────┤ ┃
┃ │ A new, own Metric │ 0.105 │ {'hyper_1': 5.0} │ 'Giving description to a │ ┃
┃ │ (yet_another_metri… │ │ │ metric' │ ┃
┃ ╰─────────────────────┴────────────────────────────────────────────────┴────────────────────┴─────────────────────────────────╯ ┃
┃ Forecast Errors - Regression ┃
┃ ╭─────────────────────┬────────────────────────────────────────────────┬────────────────────┬─────────────────────────────────╮ ┃
┃ │ Mean Absolute Error │ 0.35 │ {} │ '(Mean absolute error) │ ┃
┃ │ (mae) │ │ │ represents the difference │ ┃
┃ │ │ │ │ between the original and │ ┃
┃ │ │ │ │ predicted values extracted by │ ┃
┃ │ │ │ │ averaged the absolute │ ┃
┃ │ │ │ │ difference over the data set.' │ ┃
┃ ├─────────────────────┼────────────────────────────────────────────────┼────────────────────┼─────────────────────────────────┤ ┃
┃ │ Mean Absolute │ 2.543 │ {} │ '' │ ┃
┃ │ Percentage Error │ │ │ │ ┃
┃ │ (mape) │ │ │ │ ┃
┃ ├─────────────────────┼────────────────────────────────────────────────┼────────────────────┼─────────────────────────────────┤ ┃
┃ │ Mean Squared Error │ 0.178 │ {'squared': True} │ '(Mean Squared Error) │ ┃
┃ │ (mse) │ │ │ represents the difference │ ┃
┃ │ │ │ │ between the original and │ ┃
┃ │ │ │ │ predicted values extracted by │ ┃
┃ │ │ │ │ squared the average difference │ ┃
┃ │ │ │ │ over the data set.' │ ┃
┃ ├─────────────────────┼────────────────────────────────────────────────┼────────────────────┼─────────────────────────────────┤ ┃
┃ │ Root Mean Squared │ 0.421 │ {'squared': False} │ '(Root Mean Squared Error) is │ ┃
┃ │ Error (rmse) │ │ │ the error rate by the square │ ┃
┃ │ │ │ │ root of Mean Squared Error.' │ ┃
┃ ├─────────────────────┼────────────────────────────────────────────────┼────────────────────┼─────────────────────────────────┤ ┃
┃ │ Root Mean Squared │ 0.29 │ {'squared': False} │ '' │ ┃
┃ │ Log Error (rmsle) │ │ │ │ ┃
┃ ├─────────────────────┼────────────────────────────────────────────────┼────────────────────┼─────────────────────────────────┤ ┃
┃ │ R-squared (r2) │ -1.28 │ {} │ '(Coefficient of determination) │ ┃
┃ │ │ │ │ represents the coefficient of │ ┃
┃ │ │ │ │ how well the values fit │ ┃
┃ │ │ │ │ compared to the original │ ┃
┃ │ │ │ │ values. The value from 0 to 1 │ ┃
┃ │ │ │ │ interpreted as percentages. The │ ┃
┃ │ │ │ │ higher the value is, the better │ ┃
┃ │ │ │ │ the model is.' │ ┃
┃ ╰─────────────────────┴────────────────────────────────────────────────┴────────────────────┴─────────────────────────────────╯ ┃
┃ Unknown ┃
┃ ╭─────────────────────┬────────────────────────────────────────────────┬────────────────────┬─────────────────────────────────╮ ┃
┃ │ own_metric │ 0.035 │ {} │ '' │ ┃
┃ │ (own_metric) │ │ │ │ ┃
┃ ├─────────────────────┼────────────────────────────────────────────────┼────────────────────┼─────────────────────────────────┤ ┃
┃ │ another_metric │ 0.07 │ {} │ '' │ ┃
┃ │ (another_metric) │ │ │ │ ┃
┃ ╰─────────────────────┴────────────────────────────────────────────────┴────────────────────┴─────────────────────────────────╯ ┃
┃ ┃
┗━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┛
Default Metrics
See evaluate/library/default_metrics.py for source.
Contributors are continously adding new default metrics, press watch to keep track of the project and see in issues planned default metrics.
Residual Diagnostics
- Mean of the Residuals
- Standard Deviation of the Residuals
- Ljung Box Statistics
- (wip) Autocorrelation of Residuals
Regression Errors
- Mean Absolute Error
- Mean Absolute Percentage Error
- Mean Squared Error
- Root Mean Squared Error
- Root Mean Squared Log Error
Our Open-core Time Series Toolkit
If you want to try them out, we'd love to hear about your use case and help, please book a free 30-min call with us!
Contribution
Join our Discord for live discussion!
Submit an issue or reach out to us on info at dream-faster.ai for any inquiries.
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
krisi-0.1.0.tar.gz
(44.1 kB
view hashes)
Built Distribution
krisi-0.1.0-py3-none-any.whl
(41.8 kB
view hashes)
