traval 0.5.1

Python package for applying automatic error detection algorithms to time series. Create custom error detection algorithms to support data validation workflows.

traval

Tools for applying automatic error detection algorithms to time series.

Introduction

This module is set up to provide tools for applying any error detection algorithm to any time series. The module consists of three main components:

• RuleSet: the RuleSet object is a highly flexible object for defining error detection algorithms based on (user-defined) functions.
• Detector: a data management object for storing time series and error detection results.
• SeriesComparison*: objects for comparing time series. These objects include plots for visualizing the comparisons.

The general workflow consists of the following steps:

1. Define error detection algorithm(s).
2. Load data, i.e. raw time series data and optionally time series representing the "truth" to see how well the algorithms perform.
3. Initialize Detector objects and apply algorithms to time series.
4. Store and analyze the results.

For more detailed information and examples, please refer to the notebooks in the examples directory.

Installation

To install the traval module, follow these steps:

1. Clone the repository from GitHub.
2. Open a terminal and navigate to the module root directory: <your path here>/traval
3. Type pip install -e .

Usage

The basic usage of the module is described below. To start using the module, import the package:

import traval

The first step is generally to define an error detection algorithm. This is done with the RuleSet object:

ruleset = traval.RuleSet("my_first_algorithm")

Add a detection rule (using a general rule from the library contained within the module). In this case the rule states any value above 10.0 is suspect:

ruleset.add_rule(
    "rule1",
    traval.rulelib.rule_ufunc_threshold,
    apply_to=0,
    kwargs={"ufunc": (np.greater,), "threshold": 10.0}
)

Take a look at the ruleset by just typing ruleset:

ruleset

RuleSet: 'my_first_algorithm'
  step: name            apply_to
     1: rule1                  0

Next define a Detector object. This object is designed to store a time series and the intermediate and final results after applying an error detection algorithm. Initialize the Detector object with some time series. In this example we assume there is a time series called raw_series:

>>> detect = traval.Detector(raw_series)

Apply our first algorithm to the time series.

>>> detect.apply_ruleset(ruleset)

By default, the result of each step in the algorithm is compared to the original series and stored in the detect.comparisons attribute. Take a look at the comparison between the raw data and the result of the error detection algorithm.

Since we only defined one step, step 1 represents the final result.

>>> cp = detect.comparisons[1]  # result of step 1 = final result

The SeriesComparison* objects contain methods to visualize the comparison, or summarize the number of observations in each category:

>>> cp.plots.plot_series_comparison()  # plot a comparison
>>> cp.summary  # series containing number of observations in each category

For more detailed explanation and more complex examples, see the notebook(s) in the examples directory.

Author

• D.A. Brakenhoff, Artesia, 2020