Evaluation Tool for Time Series Anomaly Detection Methods
Project description
See TimeEval Algorithms (use this link on Github) for algorithms that are compatible to this tool.
The algorithms in this repository are containerized and can be executed using the DockerAdapter
of TimeEval.
If you use TimeEval, please consider citing our paper.
📖 TimeEval's documentation is hosted at https://timeeval.readthedocs.io.
Features
- Large integrated benchmark dataset collection with more than 700 datasets
- Benchmark dataset interface to select datasets easily
- Adapter architecture for algorithm integration
- JarAdapter
- DistributedAdapter
- MultivarAdapter
- DockerAdapter
- ... (add your own adapter)
- Automatic algorithm detection quality scoring using AUC (Area under the ROC curve, also c-statistic) metric
- Automatic timing of the algorithm execution (differentiates pre-, main-, and post-processing)
- Distributed experiment execution
- Output and logfile tracking for subsequent inspection
Installation
TimeEval can be installed as a package or from source.
Installation using pip
Builds of TimeEval
are published to the internal package registry of the Gitlab instance running at gitlab.hpi.de and to PyPI.
Prerequisites
- python >= 3.7, <=3.9
- pip >= 20
- Docker
- (optional) A personal access token with the scope set to
api
(read) or another type of access token able to read the package registry of TimeEval hosted at gitlab.hpi.de. - (optional)
rsync
for distributed TimeEval
Steps
You can use pip
to install TimeEval from PyPI:
pip install TimeEval
Installation from source
tl;dr
git clone git@gitlab.hpi.de:akita/bp2020fn1/timeeval.git
cd timeeval/
conda env create --file environment.yml
conda activate timeeval
python setup.py install
Prerequisites
The following tools are required to install TimeEval from source:
- git
- conda (anaconda or miniconda)
Steps
- Clone this repository using git and change into its root directory.
- Create a conda-environment and install all required dependencies.
Use the file
environment.yml
for this:conda env create --file environment.yml
. - Activate the new environment and install TimeEval using setup.py:
python setup.py install
. - If you want to make changes to TimeEval or run the tests, you need to install the development dependencies from
requirements.dev
:pip install -r requirements.dev
.
Usage
tl;dr
from typing import Dict, Any
import numpy as np
from timeeval import TimeEval, DatasetManager, Algorithm, TrainingType, InputDimensionality
from timeeval.adapters import FunctionAdapter
from timeeval.constants import HPI_CLUSTER
from timeeval.params import FixedParameters
# Load dataset metadata
dm = DatasetManager(HPI_CLUSTER.akita_dataset_paths[HPI_CLUSTER.BENCHMARK], create_if_missing=False)
# Define algorithm
def my_algorithm(data: np.ndarray, args: Dict[str, Any]) -> np.ndarray:
score_value = args.get("score_value", 0)
return np.full_like(data, fill_value=score_value)
# Select datasets and algorithms
datasets = dm.select(collection="NAB")
datasets = datasets[-1:]
# Add algorithms to evaluate...
algorithms = [
Algorithm(
name="MyAlgorithm",
main=FunctionAdapter(my_algorithm),
data_as_file=False,
training_type=TrainingType.UNSUPERVISED,
input_dimensionality=InputDimensionality.UNIVARIATE,
param_config=FixedParameters({"score_value": 1.})
)
]
timeeval = TimeEval(dm, datasets, algorithms)
# execute evaluation
timeeval.run()
# retrieve results
print(timeeval.get_results())
Citation
If you use TimeEval in your project or research, please cite our demonstration paper:
Phillip Wenig, Sebastian Schmidl, and Thorsten Papenbrock. TimeEval: A Benchmarking Toolkit for Time Series Anomaly Detection Algorithms. PVLDB, 15(12): 3678 - 3681, 2022. doi:10.14778/3554821.3554873
@article{WenigEtAl2022TimeEval,
title = {TimeEval: {{A}} Benchmarking Toolkit for Time Series Anomaly Detection Algorithms},
author = {Wenig, Phillip and Schmidl, Sebastian and Papenbrock, Thorsten},
date = {2022},
journaltitle = {Proceedings of the {{VLDB Endowment}} ({{PVLDB}})},
volume = {15},
number = {12},
pages = {3678--3681},
doi = {10.14778/3554821.3554873}
}
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.