Log anomaly detector for streaming logs
Log Anomaly Detector
Log anomaly detector is an open source project code named "Project Scorpio". LAD is also used for short. It can connect to streaming sources and produce predictions of abnormal log lines. Internally it uses unsupervised machine learning. We incorporate a number of machine learning models to achieve this result. In addition it includes a human in the loop feedback system.
The original goal for this project was to develop an automated means of notifying users when problems occur with their applications based on the information contained in their application logs. Unfortunately logs are full of messages that contain warnings or even errors that are safe to ignore, so simple “find-keyword” methods are insufficient . In addition, the number of logs are increasing constantly and no human will, or can, monitor them all. In short, our original aim was to employ natural language processing tools for text encoding and machine learning methods for automated anomaly detection, in an effort to construct a tool that could help developers perform root cause analysis more quickly on failing applications by highlighting the logs most likely to provide insight into the problem or to generate an alert if an application starts to produce a high frequency of anomalous logs.
It currently contains the following components:
- LAD-Core: Contains custom code to train model and predict if a log line is an anomaly. We are currently use W2V (word 2 vec) and SOM (self organizing map) with unsupervised machine learning. We are planning to add more models.
- Metrics: To monitor this system in production we utilize grafana and prometheus to visualize the health of this machine learning system.
- Fact-Store: In addition we have a metadata registry for tracking feedback from false_positives in the machine learning system and to providing a method for ML to self correcting false predictions called the “fact-store”.
Install Log Anomaly Detector (LAD):
pip install -i https://test.pypi.org/simple/ scorpio
LAD requires python 3.6 or greater
Official documentation for LAD can be found at
For help or questions about Log Anomaly Detector usage (e.g. "how do I do X?") then you can open an issue and mark it as question. One of our engineers would be glad to answer.
To report a bug, file a documentation issue, or submit a feature request, please open a GitHub issue.
For release announcements and other discussions, please subscribe to our mailing list (
Major updates will be presented at our AiOps special interest group meeting which is a part of openshift commons
OpenShift Commons AiOps Sig Calendar:
We happily welcome contributions to LAD. Please see our contribution guide for details.
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
|Filename, size||File type||Python version||Upload date||Hashes|
|Filename, size log_anomaly_detector-0.0.2-py3-none-any.whl (71.1 kB)||File type Wheel||Python version py3||Upload date||Hashes View|
|Filename, size log-anomaly-detector-0.0.2.tar.gz (41.9 kB)||File type Source||Python version None||Upload date||Hashes View|
Hashes for log_anomaly_detector-0.0.2-py3-none-any.whl
Hashes for log-anomaly-detector-0.0.2.tar.gz