Skip to main content

Orion is a machine learning library built for data generated by satellites.

Project description

“DAI-Lab” An open source project from Data to AI Lab at MIT.

“Orion”

Development Status PyPi Shield CircleCI Travis CI Shield Downloads Binder

Orion

Orion is a machine learning library built for telemetry data generated by satellites.

Overview

Orion is a machine learning library built for telemetry data generated by Satellites.

With this data, our interest is to develop techniques to:

  • identify rare patterns and flag them for expert review.
  • predict outcomes ahead of time.

The library makes use of a number of automated machine learning tools developed under "The human data interaction project" within the Data to AI Lab at MIT.

With the ready availability of automated machine learning tools, the focus is on:

  • domain expert interaction with the machine learning system;
  • learning from minimal labels;
  • explainability of model outputs;
  • model audit;
  • scalability;

Leaderboard

In this repository we maintain this up-to-date leaderboard with the current scoring of the pipelines according to the benchmarking procedure explained in the benchmark documentation.

Results obtained during benchmarking as well as previous releases can be found within benchmark/results folder as CSV files. Results can also be browsed in the following Google Sheets document.

pipeline # wins # detected anomalies average f1
LSTM Dynamic Thresholding 6 1704 0.6462
ARIMA 1715 0.5999

Table of Contents

Data Format

Input

Orion Pipelines work on time Series that are provided as a single table of telemetry observations with two columns:

  • timestamp: an INTEGER or FLOAT column with the time of the observation in Unix Time Format
  • value: an INTEGER or FLOAT column with the observed value at the indicated timestamp

This is an example of such table:

timestamp value
1222819200 -0.366358
1222840800 -0.394107
1222862400 0.403624
1222884000 -0.362759
1222905600 -0.370746

Output

The output of the Orion Pipelines is another table that contains the detected anomalous intervals and that has at least two columns:

  • start: timestamp where the anomalous interval starts
  • end: timestamp where the anomalous interval ends

Optionally, a third column called score can be included with a value that represents the severity of the detected anomaly.

An example of such a table is:

start end score
1222970400 1222992000 0.572643
1223013600 1223035200 0.572643

Dataset we use in this library

For development, evaluation of pipelines, we include a dataset which includes several satellite telemetry signals already formatted as expected by the Orion Pipelines.

This formatted dataset can be browsed and downloaded directly from the d3-ai-orion AWS S3 Bucket.

This dataset is adapted from the one used for the experiments in the Detecting Spacecraft Anomalies Using LSTMs and Nonparametric Dynamic Thresholding paper. Original source data is available for download here. We thank NASA for making this data available for public use.

Orion Pipelines

The main component in the Orion project are the Orion Pipelines, which consist of MLBlocks Pipelines specialized in detecting anomalies in time series.

As MLPipeline instances, Orion Pipelines:

  • consist of a list of one or more MLPrimitives
  • can be fitted on some data and later on used to predict anomalies on more data
  • can be scored by comparing their predictions with some known anomalies
  • have hyperparameters that can be tuned to improve their anomaly detection performance
  • can be stored as a JSON file that includes all the primitives that compose them, as well as other required configuration options.

Current Available Pipelines

In the Orion Project, the pipelines are included as JSON files, which can be found in the subdirectories inside the orion/pipelines folder.

This is the list of pipelines available so far, which will grow over time:

name location description
ARIMA orion/pipelines/arima ARIMA based pipeline
LSTM Dynamic Threshold orion/pipelines/lstm_dynamic_threshold LSTM based pipeline inspired by the Detecting Spacecraft Anomalies Using LSTMs and Nonparametric Dynamic Thresholding paper
Dummy orion/pipelines/dummy Dummy pipeline to showcase the input and output format and the usage of sample primitives
TadGAN orion/pipelines/tadgan GAN based pipeline with reconstruction based errors
Azure orion/pipelines/azure Azure API for Anomaly Detector

Install

Requirements

Python

Orion has been developed and runs on Python 3.6.

Also, although it is not strictly required, the usage of a virtualenv is highly recommended in order to avoid interfering with other software installed in the system where you are trying to run Orion.

MongoDB

In order to be fully operational, Orion requires having access to a MongoDB database running version 3.6 or higher.

Install with pip

The easiest and recommended way to install Orion is using pip:

pip install orion-ml

This will pull and install the latest stable release from PyPi.

If you want to install from source or contribute to the project please read the Contributing Guide.

Docker

Even thought it's not mandatory to use it, Orion comes with the possibility to be distributed and run as a docker image, making its usage in offline systems easier.

For more details please read the Docker Usage Documentation.

Quickstart

In the following steps we will show a short guide about how to run one of the Orion Pipelines on one of the signals from the Demo Dataset.

1. Load the data

In the first step we will load the S-1 signal from the Demo Dataset.

We will do so in two parts, train and test, as we will use the first part to fit the pipeline and the second one to evaluate its performance.

To do so, we need to import the orion.data.load_signal function and call it twice passing the 'S-1-train' and 'S-1-test' names.

from orion.data import load_signal

train_data = load_signal('S-1-train')
test_data = load_signal('S-1-test')

The output will be a table in the format described above:

    timestamp     value
0  1222819200 -0.366359
1  1222840800 -0.394108
2  1222862400  0.403625
3  1222884000 -0.362759
4  1222905600 -0.370746

2. Detect anomalies using Orion

Once we have the data, let us try to use an Orion pipeline to analyze it and search for anomalies.

In order to do so, we will have to create an instance of the orion.Orion class.

from orion import Orion

orion = Orion()

Optionally, we might want to select a pipeline other than the default one or alter the hyperparameters by the underlying MLBlocks pipeline.

For example, let's select the lstm_dynamic_threshold pipeline and reduce the number of training epochs and increase the verbosity if the LSTM primitive that it uses.

hyperparameters = {
    'keras.Sequential.LSTMTimeSeriesRegressor#1': {
        'epochs': 5,
        'verbose': True
    }
}
orion = Orion(
    pipeline='lstm_dynamic_threshold',
    hyperparameters=hyperparameters
)

Once we the pipeline is ready, we can proceed to fit it to our data:

orion.fit(train_data)

Once it is fitted, we are ready to use it to detect anomalies in our data:

anomalies = orion.detect(test_data)

:warning: Depending on your system and the exact versions that you might have installed some WARNINGS may be printed. These can be safely ignored as they do not interfere with the proper behavior of the pipeline.

The output of the previous command will be a pandas.DataFrame containing a table in the Output format described above:

        start         end     score
0  1394323200  1399701600  0.673494

3. Evaluate the performance of your pipeline

In this next step we will load some already known anomalous intervals and evaluate how good our anomaly detection was by comparing those with our detected intervals.

For this, we will first load the known anomalies for the signal that we are using:

from orion.data import load_anomalies

ground_truth = load_anomalies('S-1')

The output will be a table in the same format as the anomalies one.

        start         end
0  1392768000  1402423200

Afterwards, we can call the Orion.evaluate method, passing both the test data and the ground truth:

scores = orion.evaluate(test_data, ground_truth)

The output will be a pandas.Series containing a collection of scores indicating how the predictions were:

accuracy     0.988131
f1           0.892193
recall       0.805369
precision    1.000000
dtype: float64

Database

Orion comes ready to use a MongoDB Database to easily register and explore:

  • Multiple Datasets based on signals from one or more satellites.
  • Multiple Pipelines, including historical Pipeline versions.
  • Pipeline executions on the registered Datasets, including any environment details required to later on reproduce the results.
  • Pipeline execution results and detected events.
  • Comments about the detected events.

This, among other things, allows:

  • Providing visibility about the system usage.
  • Keeping track of the evolution of the registered pipelines and their performance over multiple datasets.
  • Visualizing and browsing the detected events by the pipelines using a web application.
  • Collecting comments from multiple domain experts about the detected events to be able to later on curate the pipelines based on their knowledge.
  • Reproducing previous executions in identical environments to replicate the obtained results.
  • Detecting and keeping a history of system failures for later investigation.

The complete Database schema and usage instructions can be found in the database documentation

History

0.1.2 - 2020-07-03

New Evaluation sub-package and refactor TadGAN.

  • Two bugs when saving signalrun if there is no event detected - Issue #92 by @dyuliu 
  • File encoding/decoding issues about README.md and HISTORY.md - Issue #88 by @dyuliu
  • Fix bottle neck of score_anomaly in Cyclegan primitive - Issue #86 by @dyuliu
  • Adjust epoch meaning in Cyclegan primitive - Issue #85 by @sarahmish
  • Rename evaluation to benchmark and metrics to evaluation - Issue #83 by @sarahmish
  • Scoring function for intervals of size one - Issue #76 by @sarahmish

0.1.1 - 2020-05-11

New class and function based interfaces.

  • Implement the Orion Class - Issue #79 by @csala
  • Implement new functional interface - Issue #80 by @csala

0.1.0 - 2020-04-23

First Orion release to PyPI: https://pypi.org/project/orion-ml/

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

orion-ml-0.1.3.dev1.tar.gz (191.1 kB view details)

Uploaded Source

Built Distribution

orion_ml-0.1.3.dev1-py2.py3-none-any.whl (79.2 kB view details)

Uploaded Python 2 Python 3

File details

Details for the file orion-ml-0.1.3.dev1.tar.gz.

File metadata

  • Download URL: orion-ml-0.1.3.dev1.tar.gz
  • Upload date:
  • Size: 191.1 kB
  • Tags: Source
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/3.2.0 pkginfo/1.5.0.1 requests/2.24.0 setuptools/47.3.1.post20200622 requests-toolbelt/0.9.1 tqdm/4.47.0 CPython/3.6.10

File hashes

Hashes for orion-ml-0.1.3.dev1.tar.gz
Algorithm Hash digest
SHA256 6a29dc878cac0afdbadc6b4f117caf906a43d7f3f496f2141dbcbd201a474546
MD5 a4156f30dc94e0d3e2d8d65416f51f04
BLAKE2b-256 d87e82a996a8a3611c50f0d066f75153f487322e20275efcd7cae21fc718673e

See more details on using hashes here.

File details

Details for the file orion_ml-0.1.3.dev1-py2.py3-none-any.whl.

File metadata

  • Download URL: orion_ml-0.1.3.dev1-py2.py3-none-any.whl
  • Upload date:
  • Size: 79.2 kB
  • Tags: Python 2, Python 3
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/3.2.0 pkginfo/1.5.0.1 requests/2.24.0 setuptools/47.3.1.post20200622 requests-toolbelt/0.9.1 tqdm/4.47.0 CPython/3.6.10

File hashes

Hashes for orion_ml-0.1.3.dev1-py2.py3-none-any.whl
Algorithm Hash digest
SHA256 23ed67ec561883736dafccd6da7e13b0998c7e74f61c5615023bf050119dd0ab
MD5 dd695b79a68b4c80aece603549349263
BLAKE2b-256 1b2f64dd2e09f45ee4a07ac9793a09d852962b7f4ae47b557075f0bc97fb7293

See more details on using hashes here.

Supported by

AWS AWS Cloud computing and Security Sponsor Datadog Datadog Monitoring Fastly Fastly CDN Google Google Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Sentry Sentry Error logging StatusPage StatusPage Status page