Skip to main content

Time Series analysis and evaluation tools

Project description

ts-eval Time Series analysis and evaluation tools

pypi Build Status codecov python3 Code style: black License: MIT Contributions welcome


A set of tools to help you analyse time series using Python.

🧩 Current features

  • N-step ahead time series evaluation – using a Jupyter widget.
  • Friedman / Nemenyi rank test (posthoc) – to see which model statistically performs better.
  • Relative Metrics – rMSE, rMAE + Forecasted Value analogues.
  • Prediction Interval Metrics – MIS, rMIS, FVrMIS
  • Fixed fourier series generation – fixed in time according to pandas index
  • Naive/Seasonal models for baseline predictions (with prediction intervals)
  • Statsmodels n-step evaluation – helper functions to evaluate n-step ahead forecasts using Statsmodels models or naive/seasonal naive models.

Here's how the Jupyter widget looks like: Demo Screenshot

Check out Demo Notebook.

Installation

  pip install ts-eval

📋 Release Planning:

  • Release 0.3
    • travis: add 3.8 default python when it's available
    • docs: supported metrics & API options
    • Maybe use api like Summary in statsmodels MLEModel class, it has extend methods and warn/info messages
    • pretty legend for lots like here https://studywolf.wordpress.com/2017/11/21/matplotlib-legends-for-mean-and-confidence-interval-plots/
    • Look for TODOs
    • changable colors
    • turn off colored display option
    • a nicer API for raw metrics container
    • codacy badge
    • coverage badge
    • travis/circleci badge
    • violin plots to compare predictions – areas can be colored, different metrics on left and right (like relative...)
  • Release 0.4
    • names of datasets passed in (now default ordinal numbers)
    • holiday/fourier features model
    • fix viz module to have less of important stuff
    • a gif with project visualization
    • check shapes of input arrays (target vs preds), now it doesn't raise an error
    • Baseline prediction using target dataset (without explicit calculation, but losing some time points)
    • Graph: plot confint

💡 Ideas

  • components
    • Graph: Visualize outliers from confidence interval
    • Multi-comparison component: scikit_posthocs lib or homecooked?
    • inspect true confidence interval coverage via sampling (was done in postings around bayesian dropout sampling)
    • xarrays: compare if compared datasets are actually equal (offets by dates, shapes, maybe even hashing)
    • bin together step performance, like steps 0-1, 2-5, 6-12, 13-24
    • highlight regions using a mask (holidays, etc.)
    • option to view interactively points using widget (plotly)?
    • diagnostics: bias to over / underestimate points
    • animated graphs for change in seasonality
  • features
    • example notebook for fourier?
    • tests for fourier
    • nint generation
  • utils:
    • model adaptor (for different models, generic) which generates 3d prediction dataset. For stastmodels using dyn forecast or kalman filter
    • future importance calculator, but only if I can manipulate input features
    • feature selection using PACF / prewhiten?
  • project
  • sMAPE & MASE can be added for the jupyter evaluation tables
  • ? Residual stats: since I have residuals => Ljung-Box, Heteroscedasticity test, Jarque-Bera – like in statsmodels results, but probably these stats were inspected already by the user... and on which step should they be computed then?

🤹🏼‍♂️ Development

Recommended development workflow:

pipenv install -e .[dev]
pipenv shell

The library doesn't use Flit/Poetry, so the suggested workflow is based on Pipenv (as per https://github.com/pypa/pipenv/issues/1911). Pipfile* are ignored in the .gitignore.

Changelog

0.2.0 (2019-10-16)

Features

  • Multiple prediction ranking with Friedman Nemenyi posthoc.
  • Visualization of prediction intervals
  • Indication of prediction ranking in a colorful table
  • Rewrite of the internal computation machinery

0.1.0 (2019-10-04)

Features

  • N-step ahead evaluation widget for Jupyter
  • Absolute & relative metrics for point forecasts and prediction intervals (MSE, MAE, rMSE, rMAE, MIS, rMIS)
  • Naive/Seasonal models for baseline (with prediction intervals)
  • Helper functions to evaluate n-step ahead forecasts using Statsmodels models or naive/seasonal naive models.
  • Holiday features generation and model evaluation on holiday datetimes.

0.0.1 (2019-09-18)

Features

  • Fixed fourier series generation (fixed in time according to pandas index)

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

ts-eval-0.2.0.tar.gz (1.2 MB view hashes)

Uploaded source

Built Distribution

ts_eval-0.2.0-py2.py3-none-any.whl (40.9 kB view hashes)

Uploaded py2 py3

Supported by

AWS AWS Cloud computing and Security Sponsor Datadog Datadog Monitoring Fastly Fastly CDN Google Google Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Sentry Sentry Error logging StatusPage StatusPage Status page