suod 0.0.1

An Acceleration System for Large Scale Unsupervised Anomaly Detection

Deployment & Documentation & Stats

---

Build Status & Coverage & Maintainability & License

---

SUOD (Toward Scalable Unsupervised Outlier Detection) is an acceleration framework for large scale unsupervised outlier detector training and prediction. Notably, anomaly detection is often formulated as an unsupervised problem since the ground truth is expensive to acquire. As a result, analysts often build many diversified models and further combine them (sometimes with rule-based models)—this has become a standard process in many industries to offset the challenges of the data imbalance and unsupervised nature. However, building a large number of unsupervised models are very costly or even infeasible on high-dimensional, large datasets.

SUOD is therefore proposed to alleviate, if not fully fix, this problem. The focus of SUOD is to accelerate the training and prediction when a large number of anomaly detectors are presented.

API Demo:

from suod.models.base import SUOD

# initialize a set of base outlier detectors to train and predict on
base_estimators = [
    LOF(n_neighbors=5, contamination=contamination),
    LOF(n_neighbors=15, contamination=contamination),
    LOF(n_neighbors=25, contamination=contamination),
    LOF(n_neighbors=35, contamination=contamination),
    LOF(n_neighbors=45, contamination=contamination),
    HBOS(contamination=contamination),
    PCA(contamination=contamination),
    OCSVM(contamination=contamination),
    KNN(n_neighbors=5, contamination=contamination),
    KNN(n_neighbors=15, contamination=contamination),
    KNN(n_neighbors=25, contamination=contamination),
    KNN(n_neighbors=35, contamination=contamination)
]

# initialize a SUOD model with all features turned on
model = SUOD(base_estimators=base_estimators,
             n_jobs=6, bps_flag=True,
             contamination=contamination, approx_flag_global=False)

model.fit(X)  # fit all models with X
model.approximate(X)  # conduct model approximation if it is enabled
predicted_labels = model.predict(X)  # predict labels on X; for demo purpose only
predicted_scores = model.decision_function(X)  # predict scores on X; for demo purpose only

---

If you use SUOD in a scientific publication, we would appreciate citations to the following paper:

@inproceedings{zhao2020suod,
  author  = {Zhao, Yue and Ding, Xueying and Yang, Jianing and Haoping Bai},
  title   = {{SUOD}: Toward Scalable Unsupervised Outlier Detection},
  journal = {Workshops at the Thirty-Fourth AAAI Conference on Artificial Intelligence},
  year    = {2020}
}

Yue Zhao, Xueying Ding, Jianing Yang, Haoping Bai, "Toward Scalable Unsupervised Outlier Detection". Workshops at the Thirty-Fourth AAAI Conference on Artificial Intelligence, 2020.

A preliminary version of paper can be accessed here. The revised and extended version will be submitted to KDD 2020 (ADS track). [Preprint], [slides], [AICS]

Table of Contents:

• Reproduction Instructions

• Installation

---

Reproduction Instructions

All three modules can be executed separately and the demo codes are in /AAAI_Workshop/{M1_RP, M2_BPS, and M3_PSA}. For instance, you could navigate to /M1_RP/demo_random_projection.py. Demo codes all start with “demo_*.py”.

A full example may be found in demo_full.py under the root directory.

Production level code will be released soon—it will support PyPI installation with full documentation and example!

---

Installation

It is recommended to use pip for installation. Please make sure the latest version is installed, as suod is updated frequently:

pip install suod            # normal install
pip install --upgrade suod  # or update if needed
pip install --pre suod      # or include pre-release version for new features

Alternatively, you could clone and run setup.py file:

git clone https://github.com/yzhao062/suod.git
cd suod
pip install .

Required Dependencies:

• Python 3.5, 3.6, or 3.7

• joblib

• matplotlib (optional for running examples)

• numpy>=1.13

• numba>=0.35

• pandas (optional for building the cost forecast model)

• pyod

• scipy>=0.19.1

• scikit_learn>=0.19.1

Note on Python 2: The maintenance of Python 2.7 will be stopped by January 1, 2020 (see official announcement). To be consistent with the Python change and suod’s dependent libraries, e.g., scikit-learn, SUOD only supports Python 3.5+ and we encourage you to use Python 3.5 or newer for the latest functions and bug fixes. More information can be found at Moving to require Python 3.

---

More to come… Last updated on Jan 4th, 2019.

Feel free to star for the future update :)

---

References

[1]

Johnson, W.B. and Lindenstrauss, J., 1984. Extensions of Lipschitz mappings into a Hilbert space. Contemporary mathematics, 26(189-206), p.1.